mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-04 02:32:07 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
78 lines
4.1 KiB
TypeScript
78 lines
4.1 KiB
TypeScript
import { createHash } from "node:crypto";
|
|
import { createReadStream } from "node:fs";
|
|
import fs from "node:fs/promises";
|
|
import path from "node:path";
|
|
import { pipeline } from "node:stream/promises";
|
|
import type { Readable } from "node:stream";
|
|
import type { RequestHandler } from "express";
|
|
import type { StorageProvider } from "./storage/types.js";
|
|
|
|
const prefix = "ui-assets/v1/";
|
|
function assetPath(value: string): string | null {
|
|
const relative = value.replace(/^\//, "");
|
|
return relative.length <= 512 && relative.split("/").every(part => /^[a-zA-Z0-9_.-]+$/.test(part) && part !== "." && part !== "..")
|
|
&& !relative.endsWith(".map") ? relative : null;
|
|
}
|
|
async function digest(stream: Readable) {
|
|
const hash = createHash("sha256");
|
|
for await (const chunk of stream) hash.update(chunk);
|
|
return hash.digest("hex");
|
|
}
|
|
|
|
/** Public, immutable build assets only. Publishing before readiness lets an
|
|
* older tab fetch its chunks after the last old container has disappeared.
|
|
* Objects have no automatic expiry: retain them at least 30 days after the
|
|
* last serving deployment, including rollback, before operator reclamation. */
|
|
export function retainedUiAssets(provider: StorageProvider): { publish(directory: string): Promise<void>; serve: RequestHandler } {
|
|
if (provider.id !== "s3" || !provider.putObjectConditional) throw new Error("Retained UI assets require conditional shared object storage");
|
|
async function publish(directory: string) {
|
|
async function visit(relativeDirectory: string) {
|
|
for (const entry of await fs.readdir(path.join(directory, relativeDirectory), { withFileTypes: true })) {
|
|
const relative = path.posix.join(relativeDirectory, entry.name);
|
|
if (entry.isDirectory()) { await visit(relative); continue; }
|
|
if (relative.endsWith(".map")) continue;
|
|
if (!entry.isFile() || !assetPath(relative)) throw new Error("Unsupported static build asset");
|
|
const filename = path.join(directory, relative);
|
|
const signal = AbortSignal.timeout(30_000);
|
|
const contentLength = (await fs.stat(filename)).size;
|
|
const expected = await digest(createReadStream(filename, { signal }));
|
|
const objectKey = `${prefix}${relative}`;
|
|
const body = createReadStream(filename, { signal });
|
|
try {
|
|
await provider.putObjectConditional!({ objectKey, body, contentLength, contentType: "application/octet-stream", ifNoneMatch: "*", signal });
|
|
} catch (error) {
|
|
// An existing key or uncertain upload may be accepted only if its
|
|
// complete bytes match. Never overwrite a colliding build asset.
|
|
const existing = await provider.getObject({ objectKey, signal }).catch(() => { throw error; });
|
|
if (await digest(existing.stream) !== expected) throw new Error("Static build asset name collides with different published bytes");
|
|
} finally { body.destroy(); }
|
|
}
|
|
}
|
|
await visit("");
|
|
}
|
|
const serve: RequestHandler = async (req, res, next) => {
|
|
if (req.method !== "GET" && req.method !== "HEAD") { next(); return; }
|
|
let relative: string | null;
|
|
try { relative = assetPath(decodeURIComponent(req.path)); }
|
|
catch { relative = null; }
|
|
if (!relative) { res.status(404).end(); return; }
|
|
const controller = new AbortController();
|
|
const signal = AbortSignal.any([controller.signal, AbortSignal.timeout(30_000)]);
|
|
const cancel = () => controller.abort();
|
|
res.once("close", cancel);
|
|
try {
|
|
const object = await provider.getObject({ objectKey: `${prefix}${relative}`, signal });
|
|
res.type(relative).set("Cache-Control", "public, max-age=31536000, immutable");
|
|
if (object.etag) res.set("ETag", object.etag);
|
|
if (object.contentLength !== undefined) res.set("Content-Length", String(object.contentLength));
|
|
if (req.method === "HEAD") { object.stream.destroy(); res.end(); return; }
|
|
await pipeline(object.stream, res, { signal });
|
|
} catch (error) {
|
|
if (controller.signal.aborted) return;
|
|
if ((error as { status?: number }).status === 404) { res.status(404).end(); return; }
|
|
next(error);
|
|
} finally { res.off("close", cancel); }
|
|
};
|
|
return { publish, serve };
|
|
}
|