mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 18:26:40 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted Paperclip servers can move from a server with a volume to a volume-free server, which allows deploys without downtime > - A server with history can move online only after its durable data is off the local disk > - Today only an offline maintenance process moves that data, so each existing server takes an outage of many minutes > - Most data owners already work safely while the server keeps serving, but nothing drives them online > - This pull request adds a signed, bounded pass that drives those owners online and reports progress, and it names configuration blockers > - The benefit is that existing servers can move to volume-free with no maintenance window ## Linked Issues or Issue Description **What existing behavior does this improve?** An existing volume-backed server can move to volume-free only through offline maintenance, because only the offline process migrates its local data. **Subsystem affected** Server: ephemeral enrollment, durable data owners (agent files, skill packages and installations, runtime skill assets, native harness backups and controllers), Cloud control endpoints. **Current behavior** - Online preparation refuses while local data remains, and no online path moves that data. - When the volume-free configuration is incomplete, preparation fails with a server error that does not say why. **Proposed behavior** - `POST /api/instance/ephemeral-enrollment/migrate` (Cloud control action `ephemeral-enrollment:migrate`) runs one pass with a 60-second budget. The pass drives the online-safe owners, defers items that are busy, and returns counts, deferral reasons and fresh eligibility. - Preparation and the eligibility read report an incomplete configuration as a named blocker: `volume_free_storage_flags_required`, `volume_free_master_key_required`, `volume_free_decision_key_required`, `volume_free_database_listener_required` or `volume_free_configuration_required`. Preparation returns 409 with these blockers. **Reason and benefit** Cloud can call the pass until a server is eligible, and then move it online with no outage. **Breaking changes** None. The offline maintenance behavior is unchanged. ## What Changed - New `services/online-durable-data-migration.ts`. The pass: - requires S3 storage and `PAPERCLIP_DURABLE_AGENT_FILES=true`, so the server does not keep creating local data; - runs these owners in order, with live writers: `migrateAgentFiles` (`writersStopped: false`), the local skill object copy, skill snapshots and versions, skill installations, runtime skill assets, and the native harness and controller prestage; - stops between pages when the budget is spent; - runs one pass at a time in each process. - `skill-package-storage-migration.ts`: a new `online` mode. In this mode a row that another owner holds is skipped and counted as `busy`. The row claim and the publication already use compare-and-swap. The offline mode is unchanged and still fails closed on a busy row. - `services/ephemeral-enrollment.ts`: a configuration check maps the volume-free contract errors to blocker codes. Preparation and the eligibility read both use this check. - `routes/ephemeral-enrollment.ts`, `middleware/cloud-control.ts`, `services/cloud-runtime-identity.ts`: the new route and action. The route accepts only an empty body. ## Verification - `npx vitest run src/routes/ephemeral-enrollment.test.ts src/services/ephemeral-enrollment.test.ts src/services/online-durable-data-migration.test.ts src/services/skill-package-storage-migration.test.ts src/__tests__/openapi-routes.test.ts src/services/cloud-runtime-identity.test.ts`: 102/102 pass. - New tests: - the pass drives every owner with live writers, aggregates counts and deferrals, and reports eligibility; - the pass does nothing without S3 or durable agent files; - the pass stops at the budget; - one pass runs at a time; - the skill object copy refuses outside maintenance unless the mode is online, skips a busy row online, and still fails closed offline; - each missing setting becomes a named blocker in preparation and in the eligibility read; - the migrate route accepts only a signed migrate assertion and an empty body. - `tsc --noEmit` reports no errors in the touched files. A full local `pnpm verify` needs the Runner build, so CI covers it. ## Risks - The pass runs real migrations while the server serves. Each owner keeps its own locks and compare-and-swap rules. Busy items are deferred, not forced. The budget limits each call. - The skill object copy can now run online. It copies immutable objects, verifies the hashes, and publishes only if the row did not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, browser). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
107 lines
5.3 KiB
TypeScript
107 lines
5.3 KiB
TypeScript
import { createHash, randomUUID } from "node:crypto";
|
|
import { Readable } from "node:stream";
|
|
import { eq } from "drizzle-orm";
|
|
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
|
|
import { companies, companySkillRepositorySnapshots as snapshots, createDb } from "@paperclipai/db";
|
|
import type { StorageService } from "../storage/index.js";
|
|
import { migrateLocalSkillPackageObjects } from "./skill-package-storage-migration.js";
|
|
import {
|
|
EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase,
|
|
} from "../__tests__/helpers/embedded-postgres.js";
|
|
|
|
vi.mock("./skill-package-store.js", () => ({ skillPackageStore: () => ({ verifyDurable: async () => true }) }));
|
|
|
|
const support = await getEmbeddedPostgresTestSupport();
|
|
const archive = Buffer.from("archive-bytes"), manifest = Buffer.from("{\"files\":[]}");
|
|
const sha = (value: Buffer) => createHash("sha256").update(value).digest("hex");
|
|
const localReceipt = () => ({ provider: "local_disk", archiveKey: `local/${randomUUID()}/files.tar.gz`, manifestKey: `local/${randomUUID()}/manifest.json`,
|
|
archiveBytes: archive.length, archiveSha256: sha(archive), manifestBytes: manifest.length, manifestSha256: sha(manifest) });
|
|
|
|
const source = {
|
|
provider: "local_disk",
|
|
getObject: async (_companyId: string, key: string) => ({ stream: Readable.from([key.endsWith("manifest.json") ? manifest : archive]) }),
|
|
} as unknown as StorageService;
|
|
|
|
function destination(onPut?: () => Promise<void>) {
|
|
const objects = new Map<string, Buffer>();
|
|
let puts = 0;
|
|
return {
|
|
objects,
|
|
get puts() { return puts; },
|
|
storage: {
|
|
provider: "s3",
|
|
putFile: async (input: { objectKey: string; body: NodeJS.ReadableStream }) => {
|
|
puts++;
|
|
const chunks: Buffer[] = [];
|
|
for await (const chunk of input.body) chunks.push(Buffer.from(chunk as Buffer));
|
|
objects.set(input.objectKey, Buffer.concat(chunks));
|
|
await onPut?.();
|
|
},
|
|
getObject: async (_companyId: string, key: string) => ({ stream: Readable.from([objects.get(key)!]) }),
|
|
deleteObject: async (_companyId: string, key: string) => { objects.delete(key); },
|
|
} as unknown as StorageService,
|
|
};
|
|
}
|
|
|
|
describe.skipIf(!support.supported)("online skill object copy against PostgreSQL", () => {
|
|
let db!: ReturnType<typeof createDb>;
|
|
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
|
beforeAll(async () => {
|
|
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-skill-object-online-");
|
|
db = createDb(tempDb.connectionString);
|
|
}, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
|
|
afterAll(async () => {
|
|
await db?.$client.end();
|
|
await tempDb?.cleanup();
|
|
});
|
|
|
|
async function snapshot(upload: Record<string, unknown> | null = null) {
|
|
const [company] = await db.insert(companies).values({ name: "Skills", issuePrefix: `S${randomUUID().slice(0, 4).toUpperCase()}` }).returning();
|
|
const receipt = localReceipt();
|
|
const [row] = await db.insert(snapshots).values({ companyId: company!.id, contentHash: randomUUID(), storage: receipt as never,
|
|
upload: upload as never }).returning();
|
|
return { row: row!, receipt };
|
|
}
|
|
const read = async (id: string) => (await db.select().from(snapshots).where(eq(snapshots.id, id)))[0]!;
|
|
const run = (storage: StorageService, after?: string) => migrateLocalSkillPackageObjects(db as never, { source, destination: storage,
|
|
writersStopped: false, online: true, signal: AbortSignal.timeout(30_000), ...(after ? { after } : {}) });
|
|
|
|
it("publishes a verified copy while serving and clears the claim", async () => {
|
|
const { row } = await snapshot();
|
|
const target = destination();
|
|
const result = await run(target.storage);
|
|
expect(result.busy).toBe(0);
|
|
const saved = await read(row.id);
|
|
expect(saved.storage).toMatchObject({ provider: "s3", archiveSha256: sha(archive), manifestSha256: sha(manifest) });
|
|
expect(saved.upload).toBeNull();
|
|
});
|
|
|
|
it("leaves a row another owner is copying untouched and counts it busy", async () => {
|
|
const held = { token: randomUUID(), provider: "s3", archiveKey: "other/files.tar.gz", manifestKey: "other/manifest.json",
|
|
expiresAt: new Date(Date.now() + 5 * 60_000).toISOString() };
|
|
const { row, receipt } = await snapshot(held);
|
|
const target = destination();
|
|
const result = await run(target.storage, undefined);
|
|
expect(result.busy).toBeGreaterThanOrEqual(1);
|
|
const saved = await read(row.id);
|
|
expect(saved.storage).toEqual(receipt);
|
|
expect(saved.upload).toEqual(held);
|
|
});
|
|
|
|
it("never publishes over a receipt that changed during the copy, and the newer receipt stays readable", async () => {
|
|
const { row } = await snapshot();
|
|
const newer = { ...localReceipt(), provider: "s3", archiveKey: "newer/files.tar.gz", manifestKey: "newer/manifest.json" };
|
|
const target = destination(async () => {
|
|
await db.update(snapshots).set({ storage: newer as never }).where(eq(snapshots.id, row.id));
|
|
});
|
|
await db.update(snapshots).set({ upload: null }).where(eq(snapshots.id, row.id));
|
|
const result = await run(target.storage);
|
|
expect(result.busy).toBeGreaterThanOrEqual(1);
|
|
const saved = await read(row.id);
|
|
expect(saved.storage).toEqual(newer);
|
|
expect(saved.upload).toBeNull();
|
|
expect(target.puts).toBe(2);
|
|
expect(target.objects.size).toBe(0);
|
|
});
|
|
});
|