mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
170 lines
7.5 KiB
TypeScript
170 lines
7.5 KiB
TypeScript
import express from "express";
|
|
import request from "supertest";
|
|
import { request as httpRequest } from "node:http";
|
|
import { createServer as createViteServer } from "vite";
|
|
import { afterEach, describe, expect, it } from "vitest";
|
|
import { drainHttpRequestWork, idleAdmissionMiddleware, trackIdleRequestHandlers } from "../middleware/idle-admission.js";
|
|
import { idleWorkSnapshot, startTaskDrain, stopTaskDrain, readTaskDrain, beginIdleTrackedWork } from "../services/task-admission.js";
|
|
|
|
function deferred() {
|
|
let resolve!: () => void;
|
|
const promise = new Promise<void>(done => { resolve = done; });
|
|
return { promise, resolve };
|
|
}
|
|
function app() { const app = express(); app.use(idleAdmissionMiddleware); return app; }
|
|
afterEach(() => stopTaskDrain());
|
|
|
|
describe("idle admission", () => {
|
|
it("preserves legacy drains, but blocks all new work during an owned idle hold", async () => {
|
|
const server = app();
|
|
server.all("/{*path}", (_req, res) => res.sendStatus(204));
|
|
trackIdleRequestHandlers(server);
|
|
startTaskDrain({ ttlMs: 60_000 });
|
|
await request(server).post("/api/issues").expect(204);
|
|
startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
|
for (const path of ["/api/issues", "/api/chat-webhooks/test", "/mcp", "/api/auth/callback"]) {
|
|
await request(server).get(path).expect(503).expect("Retry-After", "1");
|
|
await request(server).post(path).expect(503);
|
|
}
|
|
await request(server).get("/api/health").expect(204);
|
|
await request(server).get("/api/instance/task-drain/").expect(204);
|
|
await request(server).post("/api/instance/task-drain").expect(204);
|
|
stopTaskDrain();
|
|
await request(server).post("/api/issues").expect(204);
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
});
|
|
|
|
it("counts a nested async handler after it sends a response", async () => {
|
|
const server = app();
|
|
const router = express.Router();
|
|
const done = deferred();
|
|
router.post("/work", async (_req, res) => { res.sendStatus(202); await done.promise; });
|
|
server.use("/api", router);
|
|
trackIdleRequestHandlers(server);
|
|
await request(server).post("/api/work").expect(202);
|
|
expect(idleWorkSnapshot().active).toBe(1);
|
|
let drained = false;
|
|
const drain = drainHttpRequestWork(server, new AbortController().signal).then(() => { drained = true; });
|
|
await new Promise(resolve => setImmediate(resolve));
|
|
expect(drained).toBe(false);
|
|
startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
|
await request(server).post("/api/work").expect(503);
|
|
done.resolve();
|
|
await drain;
|
|
expect(drained).toBe(true);
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
});
|
|
|
|
it("tracks ordinary middleware with non-router stack metadata", async () => {
|
|
const server = app(), done = deferred();
|
|
const middleware = Object.assign(async (_req: express.Request, res: express.Response) => {
|
|
res.sendStatus(202); await done.promise;
|
|
}, { stack: { name: "middleware metadata" } });
|
|
server.use(middleware);
|
|
expect(() => trackIdleRequestHandlers(server)).not.toThrow();
|
|
await request(server).get("/work").expect(202);
|
|
expect(idleWorkSnapshot().active).toBe(1);
|
|
done.resolve();
|
|
await new Promise(resolve => setImmediate(resolve));
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
});
|
|
|
|
it("supports real Vite Connect route metadata without losing async work", async () => {
|
|
const vite = await createViteServer({ configFile: false, appType: "custom",
|
|
server: { middlewareMode: true, watch: null }, optimizeDeps: { noDiscovery: true, include: [] } });
|
|
const server = app(), done = deferred();
|
|
try {
|
|
server.use(vite.middlewares);
|
|
server.post("/work", async (_req, res) => { res.sendStatus(202); await done.promise; });
|
|
expect(() => trackIdleRequestHandlers(server)).not.toThrow();
|
|
await request(server).post("/work").expect(202);
|
|
expect(idleWorkSnapshot().active).toBe(1);
|
|
done.resolve();
|
|
await new Promise(resolve => setImmediate(resolve));
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
} finally { await vite.close(); }
|
|
});
|
|
|
|
it("tracks real Vite middleware without confusing Connect mount paths with Express routes", async () => {
|
|
const { createServer } = await import("vite");
|
|
const vite = await createServer({
|
|
configFile: false,
|
|
appType: "custom",
|
|
server: { middlewareMode: true, hmr: false, watch: null },
|
|
optimizeDeps: { noDiscovery: true, include: [] },
|
|
});
|
|
const server = app(), done = deferred();
|
|
vite.middlewares.use("/vite-work", async (_req, res) => {
|
|
res.statusCode = 202;
|
|
res.end();
|
|
await done.promise;
|
|
});
|
|
server.use(vite.middlewares);
|
|
try {
|
|
expect(() => trackIdleRequestHandlers(server)).not.toThrow();
|
|
await request(server).get("/vite-work").expect(202);
|
|
expect(idleWorkSnapshot().active).toBe(1);
|
|
startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
|
await request(server).get("/vite-work").expect(503);
|
|
done.resolve();
|
|
await new Promise(resolve => setImmediate(resolve));
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
} finally {
|
|
done.resolve();
|
|
await vite.close();
|
|
}
|
|
});
|
|
|
|
it("does not mistake a client disconnect for completed accepted work", async () => {
|
|
const server = app();
|
|
const entered = deferred(), done = deferred(), closed = deferred();
|
|
server.post("/work", async (_req, res) => { res.once("close", closed.resolve); entered.resolve(); await done.promise; /* client has left */ });
|
|
trackIdleRequestHandlers(server);
|
|
const listener = server.listen(0, "127.0.0.1");
|
|
await new Promise<void>(resolve => listener.once("listening", resolve));
|
|
const address = listener.address() as { port: number };
|
|
const client = httpRequest({ host: "127.0.0.1", port: address.port, path: "/work", method: "POST" });
|
|
client.on("error", () => undefined);
|
|
client.end();
|
|
try {
|
|
await entered.promise;
|
|
client.destroy();
|
|
await closed.promise;
|
|
startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
|
expect(idleWorkSnapshot().active).toBe(1);
|
|
done.resolve();
|
|
await new Promise(resolve => setImmediate(resolve));
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
} finally {
|
|
done.resolve(); client.destroy();
|
|
await new Promise<void>(resolve => listener.close(() => resolve()));
|
|
}
|
|
});
|
|
|
|
it("settles errors through async Express error middleware", async () => {
|
|
const server = app(), done = deferred();
|
|
server.get("/work", async () => { throw new Error("test"); });
|
|
server.use(async (_err: Error, _req: express.Request, res: express.Response, _next: express.NextFunction) => {
|
|
res.sendStatus(500); await done.promise;
|
|
});
|
|
trackIdleRequestHandlers(server);
|
|
await request(server).get("/work").expect(500);
|
|
expect(idleWorkSnapshot().active).toBe(1);
|
|
done.resolve(); await new Promise(resolve => setImmediate(resolve));
|
|
expect(idleWorkSnapshot().active).toBe(0);
|
|
});
|
|
|
|
it("uses unique owners even for simultaneous starts and expires the hold", () => {
|
|
const first = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
|
const second = startTaskDrain({ purpose: "idle", ttlMs: 60_000 });
|
|
expect(second.ownerId).not.toBe(first.ownerId);
|
|
expect(readTaskDrain(second.expiresAt!)).toBeNull();
|
|
});
|
|
|
|
it("records even work that starts and finishes between observations", () => {
|
|
const before = idleWorkSnapshot();
|
|
const done = beginIdleTrackedWork(); done(); done();
|
|
expect(idleWorkSnapshot()).toEqual({ active: before.active, generation: before.generation + 2 });
|
|
});
|
|
});
|