Files
PaperClipAI/server/src/services/ephemeral-server-contract.ts
T
DottaandPaperclip 006d5691a6 feat(server): support volume-free web server handoff (#18518)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its web server also owns controller admission and user commands.
> - A rolling deployment starts a replacement before the old server
exits.
> - Shared database state alone does not preserve local agent files,
browser chunks, or in-process commands.
> - This pull request adds an opt-in ephemeral server contract and a
local overlap test.
> - The benefit is a web server that can relinquish traffic and local
files without losing acknowledged work.

## Linked Issues or Issue Description

Refs #18403, #18327, #18335, #18341.

**Current behavior**

The control-plane web server still has local agent-file authority and
local command targets. A new container can miss saved bytes or send a
command to the wrong process. An existing browser tab also needs chunks
from its original build.

**Proposed behavior**

Eligible servers share only PostgreSQL, object storage and stable
secrets. A replacement waits for readiness. The old server drains
admitted requests, keeps owner commands available through handoff, and
releases controller leases. Explicit maintenance adopts existing files
before the volume is removed.

**Reason and benefit**

This is the application prerequisite for rolling app-only deployments.
Schema migrations and permanent multi-replica operation are outside this
change.

## What Changed

- Store agent-file manifests in PostgreSQL and immutable bytes in S3.
Treat local roots as disposable projections. Preserve existing
permission checks and edit conflicts.
- Add an explicit migration, eligibility and object-retention command to
the production runtime bundle.
- Add `/api/ready`, admitted-request drain, browser restart signaling
and retained immutable UI chunks.
- Carry transaction-aware work hints across controllers during handoff
and active execution. Keep listeners closed when idle.
- Deliver durable Stop and runtime responses to the execution owner.
Recheck request lifecycle and access before response dispatch.
- Reject local-only configuration in ephemeral mode. Keep the feature
disabled for existing installations by default.
- Add an isolated PostgreSQL/S3/proxy/browser deployment harness with
independent server homes and retained failure evidence.

## Verification

- All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`:
workspace typecheck, build, Runner checks, server/general tests and
browser shards. Greptile reviewed the runtime changes at 5/5 with all
findings resolved; the final commit updates maps and the plan.
- Local targeted checks passed: coordinator 27; Stop/recovery 33;
PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap
Stop 1; restart recovery 75. PostgreSQL command tests simulate the
provider edge.
- Local browser proof passed 91 reads and 19 saves; reconnect took
1,013ms. The tab opened Audit from S3 after deletion of A's whole
home/build. This uses compiled workspace code and local-trusted auth,
not released containers or live agents.
- Local workspace typecheck/build passed; current server and runtime
bundle passed. The earlier broad local suite was stopped after three
hours and is not a passing receipt. Its retained failures and isolated
reruns are documented in the plan.
- Stacked #18526 adds stricter graceful-exit proof, an unready
replacement control, live Runner qualification and a Stop handoff
correction. Hosted enablement remains gated on those results and Cloud
#1096. No hosted stack has been enabled.

## Risks

- Apply migration 0422 before enabling the feature. Existing data must
pass each owner's migration and eligibility checks. Do not discard a
volume based on a single migration page.
- Durable manifests remain authoritative after the opt-in is disabled.
Rolling back to a binary that ignores them is unsupported.
- Cold directory restoration and complete shutdown timing still need
qualification before enabling a hosted stack.
- Retained UI assets must cover the supported browser and rollback
window. Cleanup is explicit; no new idle maintenance polling is
introduced.
- This change does not claim production zero downtime or authorize a
fleet-wide enablement.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window limit are not exposed in this session. Used reasoning,
code editing, shell execution and browser automation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-01 01:58:54 -06:00

77 lines
5.0 KiB
TypeScript

import { and, eq, isNotNull, isNull, sql } from "drizzle-orm";
import { agentFileRoots, agents, executionWorkspaces, heartbeatRuns, providerTraceRecords, type Db } from "@paperclipai/db";
import type { Config } from "../config.js";
import { isValidLocalEncryptedMasterKey } from "../secrets/local-encrypted-provider.js";
import { agentInstructionsBundleMode } from "./agent-instructions.js";
import { skillInstallationMaintenance } from "./skill-installation-maintenance.js";
export const ephemeralServerEnabled = () => process.env.PAPERCLIP_EPHEMERAL_SERVER === "true";
/** Validate before consumers can generate a replacement local identity. Error
* messages name settings only, never credential values or connection URLs. */
export function assertEphemeralServerConfig(config: Pick<Config, "databaseUrl" | "storageProvider">, env = process.env): string | null {
if (env.PAPERCLIP_EPHEMERAL_SERVER !== "true") return null;
const refuse = (reason: string): never => { throw new Error(`Volume-free server configuration: ${reason}`); };
if (config.storageProvider !== "s3" || env.PAPERCLIP_DURABLE_AGENT_FILES !== "true" || env.PAPERCLIP_SHARED_UI_ASSETS !== "true") {
refuse("shared S3 storage, durable agent files and shared UI assets are required");
}
if (!env.PAPERCLIP_SECRETS_MASTER_KEY || !isValidLocalEncryptedMasterKey(env.PAPERCLIP_SECRETS_MASTER_KEY)) {
refuse("a stable PAPERCLIP_SECRETS_MASTER_KEY is required");
}
if ((env.PAPERCLIP_DECISION_SIGNING_SECRET?.trim().length ?? 0) < 32) {
refuse("a stable PAPERCLIP_DECISION_SIGNING_SECRET of at least 32 characters is required; preserve the existing decision key during migration");
}
let runtime: URL, listener: URL;
try { runtime = new URL(config.databaseUrl ?? ""); listener = new URL(env.PAPERCLIP_DATABASE_LISTEN_URL ?? ""); }
catch { return refuse("external DATABASE_URL and PAPERCLIP_DATABASE_LISTEN_URL are required"); }
if (![runtime, listener].every(url => ["postgres:", "postgresql:"].includes(url.protocol))
|| runtime.username !== listener.username || runtime.password !== listener.password || runtime.pathname !== listener.pathname
|| runtime.hostname.replace(/-pooler(?=\.)/, "") !== listener.hostname
|| /pooler|pgbouncer/i.test(listener.hostname)) {
refuse("PAPERCLIP_DATABASE_LISTEN_URL must be a direct connection to the same database using the runtime role");
}
return listener.href;
}
/** A readiness gate, not a periodic scan or an implicit migration. Existing
* volume data is adopted by the owning maintenance commands before removal. */
export async function assertEphemeralServerData(db: Db) {
if (!ephemeralServerEnabled()) return;
const { blockers } = await readEphemeralServerDataEligibility(db);
if (blockers.length) throw new Error(`Volume-free server data is not eligible: ${blockers.join(", ")}`);
}
export async function readEphemeralServerDataEligibility(db: Db) {
const blockers: string[] = [];
const [missingFiles] = await db.select({ id: agents.id }).from(agents).leftJoin(agentFileRoots,
and(eq(agentFileRoots.companyId, agents.companyId), eq(agentFileRoots.agentId, agents.id)))
.where(isNull(agentFileRoots.agentId)).limit(1);
if (missingFiles) blockers.push("agent_files_require_migration");
// Only inspect configuration here; filesystem reads must never adopt an
// empty directory on the replacement machine as the old server's content.
const configured = await db.select({ id: agents.id, companyId: agents.companyId, name: agents.name, adapterConfig: agents.adapterConfig }).from(agents);
if (configured.some(agent => agentInstructionsBundleMode(agent) === "external")) blockers.push("external_agent_instructions");
const [localWorkspace] = await db.select({ id: executionWorkspaces.id }).from(executionWorkspaces).where(and(
isNotNull(executionWorkspaces.cwd), sql`${executionWorkspaces.providerType} <> 'plugin'`,
sql`coalesce(${executionWorkspaces.metadata}->'remoteWorkspace'->>'schema', '') <> 'paperclip.remote-workspace.v1'`,
)).limit(1);
if (localWorkspace) blockers.push("workspaces_require_migration");
const [localLog] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.logStore, "local_file"), isNotNull(heartbeatRuns.logRef),
)).limit(1);
if (localLog) blockers.push("run_logs_require_migration");
const [localTrace] = await db.select({ id: providerTraceRecords.id }).from(providerTraceRecords).where(and(
isNull(providerTraceRecords.deletedAt), sql`${providerTraceRecords.traceRef} not like 's3-segments-v1:%'`,
)).limit(1);
if (localTrace) blockers.push("provider_traces_require_migration");
const skills = skillInstallationMaintenance(db);
let after: string | undefined;
for (;;) {
const page = await skills.eligibility({ after, limit: 50 });
if (page.results.some(skill => skill.status !== "version_backed")) { blockers.push("skills_require_migration"); break; }
if (page.inspected < 50 || !page.after) break;
after = page.after;
}
return { eligible: blockers.length === 0, blockers };
}