Files
PaperClipAI/server/src/services/online-durable-data-migration.ts
T
DottaandClaude Opus 5.5 440f96247b fix(cloud): resolve or name the remaining online move blockers (#22294)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A hosted server can move its durable data to shared storage while it
serves work, and then run without a volume
> - A bounded online pass moves the data, and a readiness check names
what still blocks the move
> - Some servers stay blocked after many passes, and the pass reports
only `skills:blocked` or `native_harness:source_not_successful`, or
reports nothing for run logs
> - Operators cannot see why, so they cannot tell recoverable data from
data that needs a maintenance step
> - This pull request recovers the cases that are safe to recover online
and gives every other case a precise code
> - The benefit is fewer blocked moves and a clear next step for each
one that stays blocked

## Linked Issues or Issue Description

Refs #19719

**What happened?**
A serving server that runs the online durable-data pass many times stays
ineligible. The pass reports `skills:blocked` without a reason,
`native_harness:source_not_successful` for several different causes, and
no reason at all for run logs that stay on the local disk.

**Expected behavior**
The pass moves every item that it can move safely while the server
serves work. For each item that stays, the pass reports a code that
names the cause.

**Steps to reproduce**
1. Install a catalog skill on a volume-backed server. Remove its
installed files and do not keep a saved version.
2. Keep a settled native run whose local log file has no final size or
digest and whose events do not prove full coverage.
3. Run the online durable-data pass.
4. See that the skill is reported only as `skills:blocked` and the run
log stays local with no reason in the pass result.

**Additional context**
The settled native log archive path from #19719 had no online caller.

## What Changed

- Skill installation conversion returns a `code` for each blocked
installation (for example `source_files_unavailable`,
`saved_version_incomplete`, `catalog_origin_unavailable`,
`legacy_source_unsafe`, `source_changed`). The pass reports
`skills:<code>`. An unknown cause stays `skills:blocked`.
- A spent budget during conversion now stops the page. Before, it was
reported as a blocked skill and the page moved past it.
- A catalog skill whose files are gone converts from the bundled catalog
package only when the catalog content hash equals the recorded origin
hash and the last audit recorded the installed bytes as equal to that
origin with no local changes.
- A modified catalog skill whose reset origin is gone gets its origin
version from the bundled catalog package only when the catalog content
hash equals the recorded origin hash.
- Native harness pre-copy replaces `source_not_successful` with
`source_run_active`, `source_run_unfinished`,
`source_coordinator_missing`, `source_coordinator_unsettled`,
`source_session_not_selected` and `source_lease_missing`. The
settled-source rules do not change.
- The run evidence worker archives a settled native log by run id when
the canonical import cannot prove event coverage. The existing
settled-source guard still decides. The worker records the reasons from
its last completed sweep, and the pass reports them as `run_logs:<code>`
and `provider_traces:<code>`.
- `migrateRunEvidence` accepts a `runId` filter.

## Verification

- `cd server && npx vitest run
src/__tests__/managed-skill-content.test.ts` (embedded PostgreSQL): new
catalog recovery, hash mismatch, unproven install and budget abort tests
fail before the change and pass after it.
- `npx vitest run
src/services/native-runtime/native-harness-online-preservation.integration.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
the new reason tests fail before the change and pass after it.
- `npx vitest run src/__tests__/run-evidence-migration-work.test.ts
src/services/native-log-archive.integration.test.ts`: the worker archive
test fails before the change and passes after it. The integration test
shows a real settled source that is blocked as
`native_log_source_coverage_unproven` and then archived by run id.
- `npx vitest run src/services/online-durable-data-migration.test.ts
src/services/online-durable-data-migration-postgres.test.ts`
- `pnpm --filter @paperclipai/server typecheck`

## Risks

- A settled native log with unproven coverage now moves to the read-only
historical archive instead of staying local. The archive keeps the exact
bytes. It does not claim canonical event coverage. This is the same
result that the offline path gives.
- The catalog fallback reads bundled or pinned catalog bytes. Each file
is checked against its pinned SHA-256, and the package must match the
recorded origin hash.
- Pass deferral codes change. Readers that match
`native_harness:source_not_successful` must use the new codes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:54:10 -06:00

280 lines
15 KiB
TypeScript

import type { Db } from "@paperclipai/db";
import { loadConfig } from "../config.js";
import { createStorageService } from "../storage/service.js";
import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js";
import { getStorageService, type StorageService } from "../storage/index.js";
import { persistActivity, publishActivity } from "./activity-log.js";
import { durableAgentFilesEnabled } from "./agent-file-durability.js";
import { migrateAgentFiles } from "./agent-file-maintenance.js";
import { cloudTenantPrimaryCompanyId, getCloudStackContext } from "./cloud-instance.js";
import { readOnlineEphemeralEnrollmentEligibility } from "./ephemeral-enrollment.js";
import { migrateLocalSkillPackageObjects } from "./skill-package-storage-migration.js";
import { skillPackageMaintenance } from "./skill-package-maintenance.js";
import { skillInstallationMaintenance } from "./skill-installation-maintenance.js";
import { migrateRuntimeSkillAssets } from "./native-runtime/runtime-skill-preservation.js";
import { prestageNativeHarnessBackups } from "./native-runtime/native-harness-backup-migration.js";
import { prestageNativeControllers } from "./native-runtime/native-controller-migration.js";
import { listLocalWorkspaceCompanies, listLocalWorkspaces } from "./local-workspace-dependencies.js";
import { reconcileOnlineWorkspaceOperations } from "./offline-workspace-operation-reconciliation.js";
import { inventoryRemoteWorkspaceMigration, migrateRemoteWorkspace } from "./remote-workspace-migration.js";
import { readRunEvidenceMigrationDeferrals } from "./run-evidence-migration-work.js";
export type OnlineDurableDataMigrationResult = {
state: "completed" | "budget_exhausted" | "unsupported";
reason?: string;
resumePhase?: string;
counts: Record<string, number>;
deferred: Record<string, number>;
eligible: boolean;
blockers: string[];
};
type PassRecord = OnlineDurableDataMigrationResult | { state: "failed"; counts: Record<string, number> };
type Owners = {
storage: () => StorageService;
localSkillStorage: () => StorageService;
durableAgentFiles: () => boolean;
agentFiles: typeof migrateAgentFiles;
skillObjects: typeof migrateLocalSkillPackageObjects;
skillPackages: (db: Db, storage: StorageService) => Pick<ReturnType<typeof skillPackageMaintenance>, "migrateSnapshots" | "migrateVersions">;
skillInstallations: (db: Db, storage: StorageService) => Pick<ReturnType<typeof skillInstallationMaintenance>, "installations">;
runtimeSkillAssets: typeof migrateRuntimeSkillAssets;
workspaceCompanies: typeof listLocalWorkspaceCompanies;
workspaceOperations: (db: Db, input: { companyId: string; signal: AbortSignal }) => Promise<{ reconciled: number; held: Array<{ reason: string }> }>;
localWorkspaces: typeof listLocalWorkspaces;
inventoryWorkspace: (db: Db, input: { companyId: string; workspaceId: string }) => Promise<{ reasons: string[] }>;
migrateWorkspace: (input: { db: Db; storage: StorageService; companyId: string; workspaceId: string; apply: true;
signal: AbortSignal; sourceLockWaitMs: number }) => Promise<{ alreadyMigrated: boolean; reasons: string[] }>;
nativeHarnessBackups: typeof prestageNativeHarnessBackups;
nativeControllers: typeof prestageNativeControllers;
runEvidenceDeferrals: () => Record<string, number>;
eligibility: (db: Db) => Promise<{ eligible: boolean; blockers: string[] }>;
record: (db: Db, result: PassRecord) => Promise<void>;
};
async function recordPass(db: Db, result: PassRecord) {
const stackId = getCloudStackContext()?.stackId;
if (!stackId) return;
const { publication } = await persistActivity(db, { companyId: cloudTenantPrimaryCompanyId(stackId), actorType: "system",
actorId: "paperclip-cloud", action: "instance.durable_data_migration.pass", entityType: "instance", entityId: stackId, details: result });
publishActivity(publication);
}
const defaultOwners: Owners = {
storage: getStorageService,
localSkillStorage: () => createStorageService(createLocalDiskStorageProvider(loadConfig().storageLocalDiskBaseDir)),
durableAgentFiles: durableAgentFilesEnabled,
agentFiles: migrateAgentFiles,
skillObjects: migrateLocalSkillPackageObjects,
skillPackages: skillPackageMaintenance,
skillInstallations: skillInstallationMaintenance,
runtimeSkillAssets: migrateRuntimeSkillAssets,
workspaceCompanies: listLocalWorkspaceCompanies,
workspaceOperations: reconcileOnlineWorkspaceOperations,
localWorkspaces: listLocalWorkspaces,
inventoryWorkspace: inventoryRemoteWorkspaceMigration,
migrateWorkspace: migrateRemoteWorkspace,
nativeHarnessBackups: prestageNativeHarnessBackups,
nativeControllers: prestageNativeControllers,
runEvidenceDeferrals: readRunEvidenceMigrationDeferrals,
eligibility: readOnlineEphemeralEnrollmentEligibility,
record: recordPass,
};
const PHASES = ["agent_files", "skill_objects", "skill_snapshots", "skill_versions", "skill_installations",
"runtime_skill_assets", "workspace_operations", "workspaces", "native_harness_backups", "native_controllers"] as const;
type Phase = typeof PHASES[number];
let running: Promise<OnlineDurableDataMigrationResult> | null = null;
let workspaceOwner: Promise<void> | null = null;
type Progress = { phase: number; after?: string; limit?: number; timeouts?: number };
let progress: Progress = { phase: 0 };
const DEFAULT_LIMITS: Partial<Record<Phase, number>> = { agent_files: 5, skill_objects: 50, skill_snapshots: 50, skill_versions: 50,
skill_installations: 50, runtime_skill_assets: 50, workspace_operations: 10, workspaces: 4 };
const WORKSPACE_SOURCE_LOCK_WAIT_MS = 1_000;
const WORKSPACE_BLOCKED = "remote_workspace_migration_blocked:";
function workspaceDeferrals(error: unknown): string[] {
const message = error instanceof Error ? error.message : "";
if (message.startsWith(WORKSPACE_BLOCKED)) {
const reasons = message.slice(WORKSPACE_BLOCKED.length).split(",").filter(reason => /^[a-z_]{1,80}$/.test(reason));
if (reasons.length) return reasons;
}
return [/^[a-z_]{1,80}$/.test(message) ? message : "owner_failed"];
}
export function resetOnlineDurableDataMigrationProgressForTests() {
progress = { phase: 0 };
workspaceOwner = null;
}
function untilAborted<T>(work: Promise<T>, signal: AbortSignal): Promise<T> {
return new Promise<T>((resolve, reject) => {
const abort = () => reject(signal.reason);
if (signal.aborted) return abort();
signal.addEventListener("abort", abort, { once: true });
work.then(resolve, reject).finally(() => signal.removeEventListener("abort", abort));
});
}
export function runOnlineDurableDataMigration(db: Db, input: { budgetMs: number; now?: () => number }, owners: Owners = defaultOwners) {
if (!running) running = migrate(db, input, owners).finally(() => { running = null; });
return running;
}
class BudgetSpent extends Error {}
async function migrate(db: Db, input: { budgetMs: number; now?: () => number }, owners: Owners): Promise<OnlineDurableDataMigrationResult> {
const now = input.now ?? Date.now;
const deadline = now() + input.budgetMs;
const budget = new AbortController();
const timer = setTimeout(() => budget.abort(new BudgetSpent()), Math.max(1, input.budgetMs));
const signal = budget.signal;
const counts: Record<string, number> = {};
const deferred: Record<string, number> = {};
const add = (bucket: Record<string, number>, key: string, by = 1) => { if (by) bucket[key] = (bucket[key] ?? 0) + by; };
const finish = async (state: OnlineDurableDataMigrationResult["state"], reason?: string): Promise<OnlineDurableDataMigrationResult> => {
if (state !== "unsupported") for (const [key, count] of Object.entries(owners.runEvidenceDeferrals())) add(deferred, key, count);
const { eligible, blockers } = await owners.eligibility(db);
const result: OnlineDurableDataMigrationResult = { state, ...(reason ? { reason } : {}),
...(state === "budget_exhausted" ? { resumePhase: PHASES[progress.phase] } : {}), counts, deferred, eligible, blockers };
await owners.record(db, result);
return result;
};
const spent = () => now() >= deadline || signal.aborted;
try {
const storage = owners.storage();
if (storage.provider !== "s3") return await finish("unsupported", "shared_object_storage_required");
if (!owners.durableAgentFiles()) return await finish("unsupported", "durable_agent_files_disabled");
const localSkills = owners.localSkillStorage();
const packages = owners.skillPackages(db, storage);
const skills = owners.skillInstallations(db, storage);
const page = async (phase: Phase, after: string | undefined, limit: number): Promise<string | null> => {
switch (phase) {
case "agent_files": {
const result = await owners.agentFiles(db, { apply: true, writersStopped: false, after, limit, signal });
for (const row of result.results) {
if (row.state === "durable") add(counts, "agentFiles");
else add(deferred, `agent_files:${row.state}`);
}
return result.next ?? null;
}
case "skill_objects": {
const result = await owners.skillObjects(db, { source: localSkills, destination: storage, writersStopped: false, online: true, signal, after, limit });
add(counts, "skillObjects", result.migrated);
add(deferred, "skill_objects:owner_busy", result.busy);
return result.inspected < limit || !result.after ? null : result.after;
}
case "skill_snapshots":
case "skill_versions": {
const step = phase === "skill_snapshots" ? packages.migrateSnapshots : packages.migrateVersions;
const result: Awaited<ReturnType<typeof step>> = await step({ apply: true, after, limit, signal });
add(counts, phase === "skill_snapshots" ? "skillSnapshots" : "skillVersions", result.migrated);
return result.inspected < limit || !result.after ? null : result.after;
}
case "skill_installations": {
const result = await skills.installations({ apply: true, writersStopped: false, after, limit, signal });
for (const row of result.results) {
if (row.status === "converted") add(counts, "skillInstallations");
else if (row.status !== "unchanged") add(deferred, `skills:${row.status === "blocked" ? row.code ?? "blocked" : row.status}`);
}
return result.inspected < limit || !result.after ? null : result.after;
}
case "runtime_skill_assets": {
const result = await owners.runtimeSkillAssets(db, { apply: true, after, limit, signal }, storage);
add(counts, "runtimeSkillRuns", result.results.filter(row => row.status === "preserved").length);
return result.inspected < limit || !result.after ? null : result.after;
}
case "workspace_operations": {
const companies = await owners.workspaceCompanies(db, { after, limit });
for (const companyId of companies) {
const result = await owners.workspaceOperations(db, { companyId, signal });
add(counts, "workspaceOperations", result.reconciled);
for (const held of result.held) add(deferred, `workspace_operations:${held.reason}`);
}
return companies.length < limit ? null : companies.at(-1)!;
}
case "workspaces": {
if (workspaceOwner) {
add(deferred, "workspaces:owner_busy");
return null;
}
const rows = await owners.localWorkspaces(db, { after, limit });
for (const { companyId, workspaceId } of rows) {
let reasons: string[];
try {
reasons = (await owners.inventoryWorkspace(db, { companyId, workspaceId })).reasons;
if (!reasons.length) {
const work = owners.migrateWorkspace({ db, storage, companyId, workspaceId, apply: true, signal,
sourceLockWaitMs: WORKSPACE_SOURCE_LOCK_WAIT_MS });
const tracked: Promise<void> = work.then(() => undefined, () => undefined)
.finally(() => { if (workspaceOwner === tracked) workspaceOwner = null; });
workspaceOwner = tracked;
const result = await untilAborted(work, signal);
reasons = result.alreadyMigrated ? [] : result.reasons.length ? result.reasons : ["not_migrated"];
}
} catch (error) {
if (signal.aborted) throw error;
reasons = workspaceDeferrals(error);
}
if (!reasons.length) add(counts, "workspaces");
for (const reason of reasons) add(deferred, `workspaces:${reason}`);
}
return rows.length < limit ? null : rows.at(-1)!.cursor;
}
case "native_harness_backups": {
await owners.nativeHarnessBackups(db, { storage, signal, onOutcome: outcome => {
if (outcome === "published") add(counts, "nativeHarnessBackups");
else if (outcome !== "current") add(deferred, `native_harness:${outcome}`);
} });
return null;
}
case "native_controllers": {
await owners.nativeControllers(db, { storage, signal, onOutcome: outcome => {
if (outcome === "imported") add(counts, "nativeControllers");
else if (!["unchanged", "archived_failed", "source_already_absent"].includes(outcome)) add(deferred, `native_controllers:${outcome}`);
} });
return null;
}
}
};
while (progress.phase < PHASES.length) {
if (spent()) return await finish("budget_exhausted");
const phase = PHASES[progress.phase]!;
let next: string | null;
try {
next = await page(phase, progress.after, progress.limit ?? DEFAULT_LIMITS[phase] ?? 1);
} catch (error) {
if (signal.reason instanceof BudgetSpent) throw error;
const code = error instanceof Error && /^[a-z_]{1,60}(?::[a-z_]{1,40})?$/.test(error.message) ? error.message : "owner_failed";
add(deferred, `${phase}:${code}`);
next = null;
}
progress = next ? { phase: progress.phase, after: next, limit: progress.limit } : { phase: progress.phase + 1 };
}
progress = { phase: 0 };
return await finish("completed");
} catch (error) {
if (signal.reason instanceof BudgetSpent) {
const phase = PHASES[progress.phase];
if (phase) {
const limit = progress.limit ?? DEFAULT_LIMITS[phase] ?? 1;
if (limit > 1) progress = { ...progress, limit: Math.max(1, Math.floor(limit / 2)), timeouts: 0 };
else if ((progress.timeouts ?? 0) + 1 >= 2) {
add(deferred, `${phase}:stalled`);
progress = { phase: progress.phase + 1 };
} else progress = { ...progress, timeouts: (progress.timeouts ?? 0) + 1 };
}
return await finish("budget_exhausted");
}
await owners.record(db, { state: "failed", counts }).catch(() => undefined);
throw error;
} finally {
clearTimeout(timer);
}
}