mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-03 17:51:58 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
101 lines
4.8 KiB
TypeScript
101 lines
4.8 KiB
TypeScript
import { createDb, type Db } from "@paperclipai/db";
|
|
import { DATABASE_WORK_CHANNEL, enableDatabaseWorkNotifications, publishDatabaseWorkHint,
|
|
receiveDatabaseWorkNotification, reconcileDatabaseWorkNotifications, subscribeDatabaseWork } from "@paperclipai/db/work-signals";
|
|
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
|
|
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
|
import { activeNativeExecutionRunIds, subscribeActiveNativeExecutions } from "./native-runtime/native-active-runs.js";
|
|
import { isIdleTaskDrainActive, subscribeTaskDrainChanges } from "./task-admission.js";
|
|
import { subscribeAllCompanyLiveEvents } from "./live-events.js";
|
|
|
|
/** Temporary overlap, not steady-state horizontal scaling. Listen while startup
|
|
* and its 120-second handoff window are active, or while this process owns a
|
|
* native execution. Browser connections alone never keep PostgreSQL awake.
|
|
* Durable queues and their existing claims remain execution authority. */
|
|
export function deploymentNotifications(db: Db, coordinator: ReturnType<typeof createDeliveryWorkCoordinator>, input: {
|
|
directDatabaseUrl: string; isWarmStandby?: () => boolean; handoffMs?: number;
|
|
}) {
|
|
enableDatabaseWorkNotifications(db);
|
|
let stopped = false, handoffEndsAt: number | null = null;
|
|
let client: ReturnType<typeof createDb>["$client"] | null = null;
|
|
const wanted = () => !stopped && !isIdleTaskDrainActive()
|
|
&& (handoffEndsAt === null || handoffEndsAt > Date.now() || activeNativeExecutionRunIds.size > 0);
|
|
async function close() {
|
|
const previous = client; client = null;
|
|
await previous?.end({ timeout: 1 });
|
|
}
|
|
const listener = coordinator.register(DELIVERY_QUEUES.deploymentNotifications, {
|
|
retryMs: 1000, runDuringDrain: true,
|
|
async run(signal) {
|
|
if (!wanted()) { await close(); return; }
|
|
if (client) return;
|
|
const opened = createDb(input.directDatabaseUrl, { maxConnections: 1, connectTimeoutSeconds: 5,
|
|
applicationName: "paperclip-deployment-notifications" }).$client;
|
|
client = opened;
|
|
const abort = () => { void opened.end({ timeout: 0 }); };
|
|
signal.addEventListener("abort", abort, { once: true });
|
|
try {
|
|
await opened.listen(DATABASE_WORK_CHANNEL, payload => receiveDatabaseWorkNotification(db, payload),
|
|
() => reconcileDatabaseWorkNotifications(db));
|
|
} catch (error) { if (client === opened) await close(); throw error; }
|
|
finally { signal.removeEventListener("abort", abort); }
|
|
if (!wanted()) await close();
|
|
},
|
|
hasPending: async () => false,
|
|
nextRunAt: () => handoffEndsAt !== null && handoffEndsAt > Date.now() ? handoffEndsAt : null,
|
|
});
|
|
const stopActive = subscribeActiveNativeExecutions(listener.wake);
|
|
const stopDrain = subscribeTaskDrainChanges(listener.wake);
|
|
|
|
// Only company identity crosses the hint channel, never event contents or
|
|
// authorization decisions. A receiving tab refreshes through its normal API.
|
|
const companies = new Set<string>();
|
|
let flushAt: number | null = null;
|
|
const publisher = coordinator.register(DELIVERY_QUEUES.liveInvalidations, {
|
|
retryMs: 1000, runDuringDrain: true,
|
|
async run() {
|
|
if (flushAt === null || Date.now() < flushAt) return;
|
|
// Remove before awaiting: a same-company event arriving during the SQL
|
|
// write belongs to the next batch and must not be cleared by this one.
|
|
flushAt = null;
|
|
for (const companyId of [...companies]) {
|
|
companies.delete(companyId);
|
|
try { await publishDatabaseWorkHint(db, `live:${companyId}`); }
|
|
catch (error) {
|
|
companies.add(companyId);
|
|
flushAt ??= Date.now() + 1000;
|
|
throw error;
|
|
}
|
|
}
|
|
},
|
|
hasPending: async () => false,
|
|
nextRunAt: () => flushAt,
|
|
});
|
|
const stopEvents = subscribeAllCompanyLiveEvents(event => {
|
|
if (stopped || !/^[a-f0-9-]{36}$/i.test(event.companyId)) return;
|
|
companies.add(event.companyId);
|
|
flushAt ??= Date.now() + 1000;
|
|
publisher.wake();
|
|
});
|
|
return {
|
|
async ready() {
|
|
await listener.ready;
|
|
// Coordinator startup errors normally retry. A replacement must not
|
|
// advertise readiness until the cross-controller channel is connected.
|
|
if (!client && !input.isWarmStandby?.()) throw new Error("Deployment notification listener did not become ready");
|
|
},
|
|
markReady() {
|
|
handoffEndsAt = Date.now() + (input.handoffMs ?? 120_000);
|
|
listener.wake();
|
|
},
|
|
subscribeCompany(companyId: string, resync: () => void) {
|
|
return subscribeDatabaseWork(db, `live:${companyId}`, event => { if (event === "settled") resync(); });
|
|
},
|
|
async stop() {
|
|
stopped = true;
|
|
stopActive(); stopDrain(); stopEvents();
|
|
await close();
|
|
await Promise.all([listener.stop(), publisher.stop()]);
|
|
},
|
|
};
|
|
}
|