mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 20:29:18 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted Paperclip servers can move from a server with a volume to a volume-free server, which allows deploys without downtime > - A server with history can move online only after its durable data is off the local disk > - Today only an offline maintenance process moves that data, so each existing server takes an outage of many minutes > - Most data owners already work safely while the server keeps serving, but nothing drives them online > - This pull request adds a signed, bounded pass that drives those owners online and reports progress, and it names configuration blockers > - The benefit is that existing servers can move to volume-free with no maintenance window ## Linked Issues or Issue Description **What existing behavior does this improve?** An existing volume-backed server can move to volume-free only through offline maintenance, because only the offline process migrates its local data. **Subsystem affected** Server: ephemeral enrollment, durable data owners (agent files, skill packages and installations, runtime skill assets, native harness backups and controllers), Cloud control endpoints. **Current behavior** - Online preparation refuses while local data remains, and no online path moves that data. - When the volume-free configuration is incomplete, preparation fails with a server error that does not say why. **Proposed behavior** - `POST /api/instance/ephemeral-enrollment/migrate` (Cloud control action `ephemeral-enrollment:migrate`) runs one pass with a 60-second budget. The pass drives the online-safe owners, defers items that are busy, and returns counts, deferral reasons and fresh eligibility. - Preparation and the eligibility read report an incomplete configuration as a named blocker: `volume_free_storage_flags_required`, `volume_free_master_key_required`, `volume_free_decision_key_required`, `volume_free_database_listener_required` or `volume_free_configuration_required`. Preparation returns 409 with these blockers. **Reason and benefit** Cloud can call the pass until a server is eligible, and then move it online with no outage. **Breaking changes** None. The offline maintenance behavior is unchanged. ## What Changed - New `services/online-durable-data-migration.ts`. The pass: - requires S3 storage and `PAPERCLIP_DURABLE_AGENT_FILES=true`, so the server does not keep creating local data; - runs these owners in order, with live writers: `migrateAgentFiles` (`writersStopped: false`), the local skill object copy, skill snapshots and versions, skill installations, runtime skill assets, and the native harness and controller prestage; - stops between pages when the budget is spent; - runs one pass at a time in each process. - `skill-package-storage-migration.ts`: a new `online` mode. In this mode a row that another owner holds is skipped and counted as `busy`. The row claim and the publication already use compare-and-swap. The offline mode is unchanged and still fails closed on a busy row. - `services/ephemeral-enrollment.ts`: a configuration check maps the volume-free contract errors to blocker codes. Preparation and the eligibility read both use this check. - `routes/ephemeral-enrollment.ts`, `middleware/cloud-control.ts`, `services/cloud-runtime-identity.ts`: the new route and action. The route accepts only an empty body. ## Verification - `npx vitest run src/routes/ephemeral-enrollment.test.ts src/services/ephemeral-enrollment.test.ts src/services/online-durable-data-migration.test.ts src/services/skill-package-storage-migration.test.ts src/__tests__/openapi-routes.test.ts src/services/cloud-runtime-identity.test.ts`: 102/102 pass. - New tests: - the pass drives every owner with live writers, aggregates counts and deferrals, and reports eligibility; - the pass does nothing without S3 or durable agent files; - the pass stops at the budget; - one pass runs at a time; - the skill object copy refuses outside maintenance unless the mode is online, skips a busy row online, and still fails closed offline; - each missing setting becomes a named blocker in preparation and in the eligibility read; - the migrate route accepts only a signed migrate assertion and an empty body. - `tsc --noEmit` reports no errors in the touched files. A full local `pnpm verify` needs the Runner build, so CI covers it. ## Risks - The pass runs real migrations while the server serves. Each owner keeps its own locks and compare-and-swap rules. Busy items are deferred, not forced. The budget limits each call. - The skill object copy can now run online. It copies immutable objects, verifies the hashes, and publishes only if the row did not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, browser). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
81 lines
5.6 KiB
TypeScript
81 lines
5.6 KiB
TypeScript
import { createReadStream, createWriteStream } from "node:fs";
|
|
import { promises as fs } from "node:fs";
|
|
import { randomUUID } from "node:crypto";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { pipeline } from "node:stream/promises";
|
|
import { Writable } from "node:stream";
|
|
import { and, asc, eq, gt, sql } from "drizzle-orm";
|
|
import { companySkillRepositorySnapshots as snapshots, type Db, type SkillPackageStorageReceipt } from "@paperclipai/db";
|
|
import type { StorageService } from "../storage/index.js";
|
|
import { boundedHashStream } from "./skill-package-files.js";
|
|
import { skillPackageStore } from "./skill-package-store.js";
|
|
|
|
export async function migrateLocalSkillPackageObjects(db: Db, input: {
|
|
source: StorageService; destination: StorageService; writersStopped: boolean; online?: boolean; signal: AbortSignal; after?: string; limit?: number;
|
|
}) {
|
|
const offline = input.writersStopped && process.env.PAPERCLIP_EPHEMERAL_MIGRATION === "true";
|
|
if ((!offline && input.online !== true)
|
|
|| input.source.provider !== "local_disk" || input.destination.provider !== "s3") throw new Error("offline_skill_object_migration_required");
|
|
const rows = await db.select({ id: snapshots.id, companyId: snapshots.companyId, storage: snapshots.storage })
|
|
.from(snapshots).where(and(sql`${snapshots.storage}->>'provider' = 'local_disk'`,
|
|
input.after ? gt(snapshots.id, input.after) : undefined)).orderBy(asc(snapshots.id)).limit(Math.max(1, Math.min(50, input.limit ?? 50)));
|
|
const source = skillPackageStore(db, input.source);
|
|
let busy = 0;
|
|
for (const row of rows) {
|
|
input.signal.throwIfAborted();
|
|
const original = row.storage!;
|
|
if (!await source.verifyDurable(row.companyId, row.id, input.signal)) throw new Error("local_skill_object_verification_failed");
|
|
const token = randomUUID(), prefix = `${row.companyId}/skill-packages/${token}`;
|
|
const upload = { token, legacy: true, provider: "s3" as const, archiveKey: `${prefix}/files.tar.gz`,
|
|
manifestKey: `${prefix}/manifest.json`, expiresAt: new Date(Date.now() + 10 * 60_000).toISOString() };
|
|
const unchanged = sql`${snapshots.storage} = ${JSON.stringify(original)}::jsonb`;
|
|
const [claimed] = await db.update(snapshots).set({ upload }).where(and(eq(snapshots.companyId, row.companyId), eq(snapshots.id, row.id), unchanged,
|
|
sql`(${snapshots.upload} is null or (${snapshots.upload}->>'expiresAt')::timestamptz <= now())`)).returning({ id: snapshots.id });
|
|
if (!claimed) {
|
|
if (offline) throw new Error("skill_object_migration_owner_busy");
|
|
busy++;
|
|
continue;
|
|
}
|
|
const directory = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-skill-object-migration-"));
|
|
try {
|
|
for (const object of [
|
|
{ source: original.archiveKey, target: upload.archiveKey, bytes: original.archiveBytes, sha256: original.archiveSha256,
|
|
filename: "files.tar.gz", contentType: "application/gzip" },
|
|
{ source: original.manifestKey, target: upload.manifestKey, bytes: original.manifestBytes, sha256: original.manifestSha256,
|
|
filename: "manifest.json", contentType: "application/json" },
|
|
]) {
|
|
const filename = path.join(directory, object.filename);
|
|
const read = await input.source.getObject(row.companyId, object.source, { signal: input.signal });
|
|
const originalDigest = boundedHashStream(object.bytes);
|
|
await pipeline(read.stream, originalDigest.stream, createWriteStream(filename, { flags: "wx", mode: 0o600 }), { signal: input.signal });
|
|
const actual = originalDigest.result();
|
|
if (actual.bytes !== object.bytes || actual.sha256 !== object.sha256) throw new Error("local_skill_object_changed");
|
|
await input.destination.putFile({ companyId: row.companyId, namespace: "skill-packages", objectKey: object.target,
|
|
originalFilename: object.filename, contentType: object.contentType, body: createReadStream(filename),
|
|
byteSize: object.bytes, sha256: object.sha256, signal: input.signal });
|
|
const saved = await input.destination.getObject(row.companyId, object.target, { signal: input.signal });
|
|
const remoteDigest = boundedHashStream(object.bytes);
|
|
await pipeline(saved.stream, remoteDigest.stream, new Writable({ write(_chunk, _encoding, done) { done(); } }), { signal: input.signal });
|
|
const verified = remoteDigest.result();
|
|
if (verified.bytes !== object.bytes || verified.sha256 !== object.sha256) throw new Error("remote_skill_object_verification_failed");
|
|
}
|
|
input.signal.throwIfAborted();
|
|
const receipt: SkillPackageStorageReceipt = { ...original, provider: "s3", archiveKey: upload.archiveKey, manifestKey: upload.manifestKey };
|
|
const [published] = await db.update(snapshots).set({ storage: receipt, upload: null, migrationAt: null })
|
|
.where(and(eq(snapshots.companyId, row.companyId), eq(snapshots.id, row.id), unchanged,
|
|
sql`${snapshots.upload}->>'token' = ${token}`)).returning({ id: snapshots.id });
|
|
if (!published) {
|
|
if (offline) throw new Error("skill_object_migration_owner_changed");
|
|
busy++;
|
|
await db.update(snapshots).set({ upload: null }).where(and(eq(snapshots.companyId, row.companyId), eq(snapshots.id, row.id),
|
|
sql`${snapshots.upload}->>'token' = ${token}`));
|
|
for (const key of [upload.archiveKey, upload.manifestKey]) {
|
|
await input.destination.deleteObject(row.companyId, key, { signal: input.signal }).catch(() => undefined);
|
|
}
|
|
}
|
|
} finally { await fs.rm(directory, { recursive: true, force: true }); }
|
|
}
|
|
return { inspected: rows.length, migrated: rows.length - busy, busy, after: rows.at(-1)?.id ?? null };
|
|
}
|