fix(cloud): resolve or name the remaining online move blockers (#22294)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A hosted server can move its durable data to shared storage while it
serves work, and then run without a volume
> - A bounded online pass moves the data, and a readiness check names
what still blocks the move
> - Some servers stay blocked after many passes, and the pass reports
only `skills:blocked` or `native_harness:source_not_successful`, or
reports nothing for run logs
> - Operators cannot see why, so they cannot tell recoverable data from
data that needs a maintenance step
> - This pull request recovers the cases that are safe to recover online
and gives every other case a precise code
> - The benefit is fewer blocked moves and a clear next step for each
one that stays blocked

## Linked Issues or Issue Description

Refs #19719

**What happened?**
A serving server that runs the online durable-data pass many times stays
ineligible. The pass reports `skills:blocked` without a reason,
`native_harness:source_not_successful` for several different causes, and
no reason at all for run logs that stay on the local disk.

**Expected behavior**
The pass moves every item that it can move safely while the server
serves work. For each item that stays, the pass reports a code that
names the cause.

**Steps to reproduce**
1. Install a catalog skill on a volume-backed server. Remove its
installed files and do not keep a saved version.
2. Keep a settled native run whose local log file has no final size or
digest and whose events do not prove full coverage.
3. Run the online durable-data pass.
4. See that the skill is reported only as `skills:blocked` and the run
log stays local with no reason in the pass result.

**Additional context**
The settled native log archive path from #19719 had no online caller.

## What Changed

- Skill installation conversion returns a `code` for each blocked
installation (for example `source_files_unavailable`,
`saved_version_incomplete`, `catalog_origin_unavailable`,
`legacy_source_unsafe`, `source_changed`). The pass reports
`skills:<code>`. An unknown cause stays `skills:blocked`.
- A spent budget during conversion now stops the page. Before, it was
reported as a blocked skill and the page moved past it.
- A catalog skill whose files are gone converts from the bundled catalog
package only when the catalog content hash equals the recorded origin
hash and the last audit recorded the installed bytes as equal to that
origin with no local changes.
- A modified catalog skill whose reset origin is gone gets its origin
version from the bundled catalog package only when the catalog content
hash equals the recorded origin hash.
- Native harness pre-copy replaces `source_not_successful` with
`source_run_active`, `source_run_unfinished`,
`source_coordinator_missing`, `source_coordinator_unsettled`,
`source_session_not_selected` and `source_lease_missing`. The
settled-source rules do not change.
- The run evidence worker archives a settled native log by run id when
the canonical import cannot prove event coverage. The existing
settled-source guard still decides. The worker records the reasons from
its last completed sweep, and the pass reports them as `run_logs:<code>`
and `provider_traces:<code>`.
- `migrateRunEvidence` accepts a `runId` filter.

## Verification

- `cd server && npx vitest run
src/__tests__/managed-skill-content.test.ts` (embedded PostgreSQL): new
catalog recovery, hash mismatch, unproven install and budget abort tests
fail before the change and pass after it.
- `npx vitest run
src/services/native-runtime/native-harness-online-preservation.integration.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
the new reason tests fail before the change and pass after it.
- `npx vitest run src/__tests__/run-evidence-migration-work.test.ts
src/services/native-log-archive.integration.test.ts`: the worker archive
test fails before the change and passes after it. The integration test
shows a real settled source that is blocked as
`native_log_source_coverage_unproven` and then archived by run id.
- `npx vitest run src/services/online-durable-data-migration.test.ts
src/services/online-durable-data-migration-postgres.test.ts`
- `pnpm --filter @paperclipai/server typecheck`

## Risks

- A settled native log with unproven coverage now moves to the read-only
historical archive instead of staying local. The archive keeps the exact
bytes. It does not claim canonical event coverage. This is the same
result that the offline path gives.
- The catalog fallback reads bundled or pinned catalog bytes. Each file
is checked against its pinned SHA-256, and the package must match the
recorded origin hash.
- Pass deferral codes change. Readers that match
`native_harness:source_not_successful` must use the new codes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-18 23:54:10 -06:00
1 parent 170e1b0c07
commit 440f96247b
14 files changed
+287 -42

No files matched your search

@@ -18,6 +18,7 @@ import path from "node:path";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import { applyPendingMigrations, agents, heartbeatRuns, companySkills, companySkillVersions, companies, createDb, instanceSettings } from "@paperclipai/db";
import { companySkillService } from "../services/company-skills.js";
import { listCatalogSkills, readCatalogSkillFile } from "../services/skills-catalog.js";
import * as storageModule from "../storage/index.js";
import { createStorageService } from "../storage/service.js";
import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js";
@@ -618,7 +619,7 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
markdown: "# Linked", sourceType: "local_path", sourceLocator: path.join(root, "linked") }).returning();
const report = await skillInstallationMaintenance(db).installations({ companyId, apply: true, writersStopped: true });
expect(report.results).toEqual(expect.arrayContaining([
expect.objectContaining({ skillId: legacy!.id, status: "blocked" }),
expect.objectContaining({ skillId: legacy!.id, status: "blocked", code: "source_files_unavailable" }),
expect.objectContaining({ skillId: linked!.id, status: "linked_local" }),
]));
expect(await companySkillService(db).getById(companyId, legacy!.id)).toMatchObject({ sourceType: "local_path", currentVersionId: null });
@@ -680,11 +681,106 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
const [legacy] = await db.insert(companySkills).values({ companyId, key: "unsafe", slug: "unsafe", name: "Unsafe", markdown: "# legacy",
sourceType: "local_path", sourceLocator: directory, metadata: { sourceKind: "managed_local" } }).returning();
expect((await skillInstallationMaintenance(db).installations({ id: legacy!.id, apply: true, writersStopped: true })).results)
.toEqual([expect.objectContaining({ status: "blocked", reason: "Legacy skill contains a symbolic link" })]);
.toEqual([expect.objectContaining({ status: "blocked", reason: "Legacy skill contains a symbolic link", code: "legacy_source_unsafe" })]);
expect(await companySkillService(db).getById(companyId, legacy!.id)).toMatchObject({ sourceType: "local_path", currentVersionId: null });
} finally { await fs.unlink(path.join(directory, "support.txt")); }
});
it("converts an unmodified catalog installation whose files are gone from its pinned catalog package", async () => {
const companyId = await company(), catalog = listCatalogSkills().find(entry => !entry.source)!;
const [skill] = await db.insert(companySkills).values({ companyId, key: `catalog-${randomUUID()}`, slug: `catalog-${randomUUID().slice(0, 8)}`,
name: "Catalog", markdown: "# stale row", sourceType: "catalog", sourceLocator: path.join(root, `catalog-missing-${randomUUID()}`),
sourceRef: catalog.contentHash, fileInventory: catalog.files.map(({ path, kind }) => ({ path, kind })),
metadata: { sourceKind: "catalog", catalogId: catalog.id, originHash: catalog.contentHash, installedHash: catalog.contentHash, userModifiedAt: null },
}).returning();
const maintenance = skillInstallationMaintenance(db);
expect((await maintenance.installations({ id: skill.id, apply: true })).results).toEqual([expect.objectContaining({ status: "converted" })]);
const [row] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id));
expect(row.metadata?.originVersionId).toBe(row.currentVersionId);
expect((await maintenance.eligibility({ id: skill.id })).results[0]).toMatchObject({ status: "version_backed" });
const store = skillPackageStore(db), saved = await store.version(companyId, skill.id, row.currentVersionId!);
expect((await store.versionFiles(saved!)).find(file => file.path === "SKILL.md")?.content)
.toBe((await readCatalogSkillFile(catalog.id, "SKILL.md")).content);
});
it("keeps an unmodified catalog installation blocked when it was not proven identical to its catalog origin", async () => {
const companyId = await company(), catalog = listCatalogSkills().find(entry => !entry.source)!;
const [skill] = await db.insert(companySkills).values({ companyId, key: `catalog-${randomUUID()}`, slug: `catalog-${randomUUID().slice(0, 8)}`,
name: "Catalog", markdown: "# stale row", sourceType: "catalog", sourceLocator: path.join(root, `catalog-missing-${randomUUID()}`),
sourceRef: catalog.contentHash, fileInventory: catalog.files.map(({ path, kind }) => ({ path, kind })),
metadata: { sourceKind: "catalog", catalogId: catalog.id, originHash: catalog.contentHash, userModifiedAt: null },
}).returning();
expect((await skillInstallationMaintenance(db).installations({ id: skill.id, apply: true })).results)
.toEqual([expect.objectContaining({ status: "blocked", code: "source_files_unavailable" })]);
expect((await db.select().from(companySkills).where(eq(companySkills.id, skill.id)))[0].currentVersionId).toBeNull();
});
it("recovers a modified catalog installation's missing reset origin only from the exact pinned catalog package", async () => {
const companyId = await company(), service = companySkillService(db), catalog = listCatalogSkills().find(entry => !entry.source)!;
const skill = await service.createLocalSkill(companyId, { name: "Modified catalog", markdown: "# Locally modified catalog skill" });
const metadata = { sourceKind: "catalog", catalogId: catalog.id, originHash: `sha256:${"0".repeat(64)}`,
installedHash: `sha256:${"1".repeat(64)}`, userModifiedAt: new Date().toISOString(),
originSnapshotLocator: path.join(root, `catalog-origin-missing-${randomUUID()}`) };
await db.update(companySkills).set({ sourceType: "catalog", sourceLocator: path.join(root, `catalog-installed-missing-${randomUUID()}`),
fileInventory: [{ path: "SKILL.md", kind: "skill" }], metadata }).where(eq(companySkills.id, skill.id));
const maintenance = skillInstallationMaintenance(db);
expect((await maintenance.installations({ id: skill.id, apply: true })).results)
.toEqual([expect.objectContaining({ status: "blocked", code: "catalog_origin_unavailable" })]);
await db.update(companySkills).set({ metadata: { ...metadata, originHash: catalog.contentHash } }).where(eq(companySkills.id, skill.id));
expect((await maintenance.installations({ id: skill.id, apply: true })).results)
.toEqual([expect.objectContaining({ status: "converted", versionId: skill.currentVersionId })]);
const [row] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id));
const originVersionId = row.metadata?.originVersionId as string;
expect(originVersionId).toBeTruthy();
expect(originVersionId).not.toBe(row.currentVersionId);
const store = skillPackageStore(db);
const origin = await store.versionFiles((await store.version(companyId, skill.id, originVersionId))!);
expect(origin.find(file => file.path === "SKILL.md")?.content).toBe((await readCatalogSkillFile(catalog.id, "SKILL.md")).content);
const current = await store.versionFiles((await store.version(companyId, skill.id, row.currentVersionId!))!);
expect(current.find(file => file.path === "SKILL.md")?.content).toBe("# Locally modified catalog skill");
expect((await maintenance.eligibility({ id: skill.id })).results[0]).toMatchObject({ status: "version_backed" });
});
it("cancels a stalled pinned catalog download when the budget expires", async () => {
const companyId = await company(), catalog = listCatalogSkills().find(entry => entry.source)!;
const [skill] = await db.insert(companySkills).values({ companyId, key: `catalog-${randomUUID()}`, slug: `catalog-${randomUUID().slice(0, 8)}`,
name: "Remote catalog", markdown: "# stale row", sourceType: "catalog", sourceLocator: path.join(root, `catalog-missing-${randomUUID()}`),
sourceRef: catalog.contentHash, fileInventory: catalog.files.map(({ path, kind }) => ({ path, kind })),
metadata: { sourceKind: "catalog", catalogId: catalog.id, originHash: catalog.contentHash, installedHash: catalog.contentHash, userModifiedAt: null },
}).returning();
const budget = new AbortController();
const download = vi.spyOn(globalThis, "fetch").mockImplementation((_url, init) => new Promise((_resolve, reject) => {
const signal = init?.signal;
if (!signal) return;
signal.addEventListener("abort", () => reject(signal.reason), { once: true });
budget.abort(new Error("budget spent"));
}));
try {
await expect(skillInstallationMaintenance(db).installations({ id: skill.id, apply: true, signal: budget.signal }))
.rejects.toThrow("budget spent");
} finally { download.mockRestore(); }
expect((await db.select().from(companySkills).where(eq(companySkills.id, skill.id)))[0].currentVersionId).toBeNull();
});
it("stops at an aborted budget instead of reporting the installation as blocked", async () => {
const companyId = await company(), directory = path.join(root, `aborted-conversion-${randomUUID()}`);
await fs.mkdir(directory);
await fs.writeFile(path.join(directory, "SKILL.md"), "# Aborted conversion");
const [skill] = await db.insert(companySkills).values({ companyId, key: "aborted-conversion", slug: "aborted-conversion", name: "Aborted",
markdown: "# Aborted conversion", sourceType: "local_path", sourceLocator: directory,
fileInventory: [{ path: "SKILL.md", kind: "skill" }], metadata: { sourceKind: "managed_local" } }).returning();
const storage = storageModule.getStorageService(), budget = new AbortController();
const upload = vi.spyOn(storage, "putFile").mockImplementation(async () => {
budget.abort(new Error("budget spent"));
throw budget.signal.reason;
});
try {
await expect(skillInstallationMaintenance(db, storage).installations({ id: skill.id, apply: true, signal: budget.signal }))
.rejects.toThrow("budget spent");
} finally { upload.mockRestore(); }
expect((await db.select().from(companySkills).where(eq(companySkills.id, skill.id)))[0].currentVersionId).toBeNull();
});
it("keeps the compatibility override durable in Postgres and reports that storage accurately", async () => {
const companyId = await company();
process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = "false";
@@ -2,7 +2,7 @@ import type { Db } from "@paperclipai/db";
import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js";
import { afterEach, expect, it, vi } from "vitest";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { registerRunEvidenceMigrationWork } from "../services/run-evidence-migration-work.js";
import { readRunEvidenceMigrationDeferrals, registerRunEvidenceMigrationWork } from "../services/run-evidence-migration-work.js";
const state = vi.hoisted(() => ({
provider: "s3",
@@ -125,6 +125,30 @@ it("keeps blocked history visible and retries it without stranding later pages",
expect(state.probe!()).toBe("none");
expect(coordinator!.nextWakeAt()).toBeNull();
});
it("archives a settled native log whose canonical coverage is unproven and reports what stays blocked", async () => {
state.migrate
.mockResolvedValueOnce({
items: [
{ kind: "log", id: "settled", state: "blocked", reason: "native_log_source_coverage_unproven" },
{ kind: "log", id: "unsettled", state: "blocked", reason: "native_log_source_coverage_unproven" },
{ kind: "trace", id: "trace", state: "blocked", reason: "Trace file /data/secret is missing" },
],
})
.mockResolvedValueOnce({ items: [{ kind: "log", id: "settled", state: "migrated", historicalArchive: true }] })
.mockResolvedValueOnce({ items: [{ kind: "log", id: "unsettled", state: "blocked", reason: "native_log_archive_source_unsettled" }] })
.mockResolvedValue({ items: [] });
const { worker } = setup();
await worker!.ready;
expect(state.migrate).toHaveBeenNthCalledWith(2, expect.anything(), expect.objectContaining({
apply: true, runId: "settled", kinds: ["log"], limit: 1, preserveSettledNativeFiles: true }));
expect(state.migrate).toHaveBeenNthCalledWith(3, expect.anything(), expect.objectContaining({
runId: "unsettled", preserveSettledNativeFiles: true }));
expect(readRunEvidenceMigrationDeferrals()).toEqual({
"run_logs:native_log_archive_source_unsettled": 1,
"provider_traces:migration_failed": 1,
});
expect(state.probe!()).toBe("present");
});
it("retains startup work during drain and retries a failed sweep before becoming quiet", async () => {
let admitted = false;
state.migrate
@@ -75,6 +75,18 @@ describeEmbeddedPostgres("online native historical log preservation", () => {
});
});
it("archives by run id a settled source whose canonical online import cannot prove coverage", async () => {
await withSource(async f => {
const { preserveSettledNativeFiles: _archive, ...canonical } = f.options;
expect((await migrateRunEvidence(f.db, canonical)).items)
.toMatchObject([{ id: f.runId, state: "blocked", reason: "native_log_source_coverage_unproven" }]);
const result = await migrateRunEvidence(f.db, { apply: true, runId: f.runId, kinds: ["log"], limit: 1, preserveSettledNativeFiles: true });
expect(result.items).toMatchObject([{ id: f.runId, state: "migrated", historicalArchive: true, bytes: f.bytes.length }]);
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0])
.toMatchObject({ logStore: "s3_archive", logRef: `${f.companyId}/${f.agentId}/${f.runId}.archive` });
});
});
it("preserves a committed successful run without changing its result or completeness metadata", async () => {
await withSource(async f => {
const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
@@ -211,14 +211,16 @@ export async function prestageNativeHarnessBackups(db: Db, input: { storage: Sto
return "current";
}
if (run.status !== "succeeded" || !run.finishedAt) {
if (!run.finishedAt || !["failed", "cancelled", "interrupted", "timed_out"].includes(run.status)) return "source_not_successful";
if (!run.finishedAt || !["failed", "cancelled", "interrupted", "timed_out"].includes(run.status)) {
return ["queued", "scheduled_retry", "running"].includes(run.status) ? "source_run_active" : "source_run_unfinished";
}
const providerLeaseId = await verifiedSourceLease(db, source, run, signal);
if (!providerLeaseId) return "source_unverified";
try {
const owner = await settledNativeHarnessBackupOwner(db, { companyId: run.companyId, issueId: run.issueId,
const { owner, reason } = await settledNativeHarnessBackupOwner(db, { companyId: run.companyId, issueId: run.issueId,
runId: source.runId, sessionId: run.sessionId, runnerInstanceId: run.runnerId, scopeDigest: source.scopeDigest,
root: source.root, manifestSha256: source.manifestSha256, providerLeaseId, signal });
if (!owner) return "source_not_successful";
if (!owner) return `source_${reason}`;
await persistNativeHarnessBackup({ db, storage: input.storage, owner, root: source.root,
scopeDigest: source.scopeDigest, signal, maxArchiveBytes: source.archiveLimit });
return "published";
@@ -612,7 +612,7 @@ describeEmbeddedPostgres("durable native provider backups", () => {
it("defers failed history without treating pre-copy as offline authority", async () => {
const [f] = await offlineFixtures(1);
expect(await prestageNativeHarnessBackups(f.db, { storage: f.storage, signal: AbortSignal.timeout(15_000) }))
.toEqual({ inspected: 1, published: 0, current: 0, deferred: { source_not_successful: 1 } });
.toEqual({ inspected: 1, published: 0, current: 0, deferred: { source_session_not_selected: 1 } });
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toEqual([]);
});
it("defers a source being replaced online while offline migration still refuses it", async () => {
@@ -90,7 +90,7 @@ describeEmbeddedPostgres("online preservation of settled legacy harness backups"
await f.db.update(nativeRunFinalizations).set({ leaseOwner: "original-owner", leaseExpiresAt: new Date(0) })
.where(eq(nativeRunFinalizations.runId, f.runId));
expect(await prestageNativeHarnessBackups(f.db, { storage: f.storage, signal: AbortSignal.timeout(15_000) }))
.toMatchObject({ published: 0, deferred: { source_not_successful: 1 } });
.toMatchObject({ published: 0, deferred: { source_coordinator_unsettled: 1 } });
await f.db.update(nativeRunFinalizations).set({ leaseOwner: null }).where(eq(nativeRunFinalizations.runId, f.runId));
const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
await f.db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, nativeWorkspaceFinalizationOwner: { token: "physical-owner" } } })
@@ -99,6 +99,22 @@ describeEmbeddedPostgres("online preservation of settled legacy harness backups"
.toMatchObject({ published: 0, deferred: { source_changed: 1 } });
expect(put).not.toHaveBeenCalled();
});
it.each([
["running", "source_run_active"],
["scheduled_retry", "source_run_active"],
["failed_unfinished", "source_run_unfinished"],
["coordinator_missing", "source_coordinator_missing"],
["session_not_selected", "source_session_not_selected"],
])("names why a %s source stays local", async (kind, outcome) => {
const f = await fixture(), put = vi.spyOn(f.storage, "putFile");
if (kind === "running" || kind === "scheduled_retry") await f.db.update(heartbeatRuns).set({ status: kind, finishedAt: null }).where(eq(heartbeatRuns.id, f.runId));
if (kind === "failed_unfinished") await f.db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, f.runId));
if (kind === "coordinator_missing") await f.db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
if (kind === "session_not_selected") await f.db.delete(agentTaskSessions).where(eq(agentTaskSessions.lastRunId, f.runId));
expect(await prestageNativeHarnessBackups(f.db, { storage: f.storage, signal: AbortSignal.timeout(15_000) }))
.toMatchObject({ published: 0, deferred: { [outcome]: 1 } });
expect(put).not.toHaveBeenCalled();
});
it("respects the physical advisory lock while unrelated work stays active", async () => {
const f = await fixture(), put = vi.spyOn(f.storage, "putFile");
let release!: () => void, entered!: () => void;
@@ -139,7 +155,7 @@ describeEmbeddedPostgres("online preservation of settled legacy harness backups"
});
it("binds the archive payload and scope to the checked preservation owner", async () => {
const f = await fixture(), signal = AbortSignal.timeout(15_000);
const owner = await settledNativeHarnessBackupOwner(f.db, { ...f, sessionId: f.normalizedSessionId,
const { owner } = await settledNativeHarnessBackupOwner(f.db, { ...f, sessionId: f.normalizedSessionId,
manifestSha256: f.stamp.manifestSha256, signal });
expect(owner).not.toBeNull();
const put = vi.spyOn(f.storage, "putFile");
@@ -37,22 +37,23 @@ const fail = (): never => { throw new Error("native_harness_preservation_source_
export async function settledNativeHarnessBackupOwner(db: Db, input: {
companyId: string; issueId: string; runId: string; sessionId: string; runnerInstanceId: string;
scopeDigest: string; root: string; manifestSha256: string; providerLeaseId: string; signal: AbortSignal;
}): Promise<SettledNativeHarnessBackupOwner | null> {
}): Promise<{ owner: SettledNativeHarnessBackupOwner; reason?: never }
| { owner: null; reason: "coordinator_missing" | "coordinator_unsettled" | "session_not_selected" | "lease_missing" }> {
const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId),
eq(nativeRunFinalizations.issueId, input.issueId), eq(nativeRunFinalizations.runId, input.runId)));
if (!run || !coordinator || !settled(coordinator)) return null;
if (!run || !coordinator || !settled(coordinator)) return { owner: null, reason: run && coordinator ? "coordinator_unsettled" : "coordinator_missing" };
const selected = await db.select({ id: agentTaskSessions.id, lastRunId: agentTaskSessions.lastRunId }).from(agentTaskSessions)
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, agentTaskSessions.companyId), eq(heartbeatRuns.agentId, agentTaskSessions.agentId),
eq(heartbeatRuns.id, agentTaskSessions.lastRunId))).where(and(eq(agentTaskSessions.companyId, input.companyId),
eq(agentTaskSessions.agentId, run.agentId), eq(agentTaskSessions.adapterType, "paperclip_runner"),
eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.nativeSessionId, input.sessionId))).limit(2);
if (selected.length !== 1 || !selected[0]!.lastRunId) return null;
if (selected.length !== 1 || !selected[0]!.lastRunId) return { owner: null, reason: "session_not_selected" };
const leases = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
eq(environmentLeases.issueId, input.issueId), eq(environmentLeases.providerLeaseId, input.providerLeaseId),
sql`${environmentLeases.metadata}->'nativeHarnessBackup'->>'manifestSha256' = ${input.manifestSha256}`));
const lease = leases.find(row => readNativeHarnessBackupStamp(row.metadata?.nativeHarnessBackup, input.providerLeaseId)?.sessionScopeSha256 === input.scopeDigest);
if (!lease) return null;
if (!lease) return { owner: null, reason: "lease_missing" };
const owner: SettledNativeHarnessBackupOwner = { kind: "settled_native_harness_backup", companyId: input.companyId,
issueId: input.issueId, agentId: run.agentId, runId: input.runId, sessionId: input.sessionId, runnerInstanceId: input.runnerInstanceId,
scopeDigest: input.scopeDigest, root: input.root, manifestSha256: input.manifestSha256,
@@ -60,7 +61,7 @@ export async function settledNativeHarnessBackupOwner(db: Db, input: {
sourceLeaseId: lease.id, providerLeaseId: input.providerLeaseId,
runDigest: runDigest(run), coordinatorDigest: coordinatorDigest(coordinator), leaseDigest: leaseDigest(lease) };
await db.transaction(tx => assertSettledNativeHarnessBackupSource(tx as unknown as Db, owner, input.signal));
return owner;
return { owner };
}
export async function assertSettledNativeHarnessBackupSource(db: Db, owner: SettledNativeHarnessBackupOwner,
@@ -77,6 +77,7 @@ describe.skipIf(!support.supported)("online durable-data migration of local work
migrateWorkspace: migrateRemoteWorkspace,
nativeHarnessBackups: async () => undefined,
nativeControllers: async () => undefined,
runEvidenceDeferrals: () => ({}),
eligibility: async () => ({ eligible: false, blockers: [] }),
record: async () => undefined,
}) as never;
@@ -39,6 +39,7 @@ function owners(overrides: Record<string, unknown> = {}) {
for (const outcome of ["imported", "unchanged"]) input.onOutcome?.(outcome);
return { inspected: 2, imported: 1, unchanged: 1, archivedFailed: 0, sourceAlreadyAbsent: 0, deferred: {} };
},
runEvidenceDeferrals: () => ({}),
eligibility: vi.fn(async () => ({ eligible: false, blockers: ["skills_require_migration", "volume_free_storage_flags_required"] })),
record: vi.fn(async () => undefined),
...overrides,
@@ -65,6 +66,19 @@ describe("online durable-data migration pass", () => {
expect(fixture.record).toHaveBeenCalledWith(db, result);
});
it("names why skill installations and retained run logs stay behind", async () => {
const fixture = owners({
skillInstallations: () => ({ installations: async () => ({ inspected: 3, after: null, results: [
{ skillId: "s1", companyId: "c", status: "blocked", reason: "Saved version does not cover the installed inventory", code: "saved_version_incomplete" },
{ skillId: "s2", companyId: "c", status: "blocked", reason: "network down" },
{ skillId: "s3", companyId: "c", status: "linked_local" }] }) }),
runEvidenceDeferrals: () => ({ "run_logs:native_log_archive_source_unsettled": 2 }),
});
const result = await runOnlineDurableDataMigration(db, { budgetMs: 60_000 }, fixture.owners);
expect(result.deferred).toMatchObject({ "skills:saved_version_incomplete": 1, "skills:blocked": 1, "skills:linked_local": 1,
"run_logs:native_log_archive_source_unsettled": 2 });
});
it("migrates agent files with live writers, never the offline mode", async () => {
const agentFiles = vi.fn(async () => ({ results: [], next: null }));
await runOnlineDurableDataMigration(db, { budgetMs: 60_000 }, owners({ agentFiles }).owners);
@@ -17,6 +17,7 @@ import { prestageNativeControllers } from "./native-runtime/native-controller-mi
import { listLocalWorkspaceCompanies, listLocalWorkspaces } from "./local-workspace-dependencies.js";
import { reconcileOnlineWorkspaceOperations } from "./offline-workspace-operation-reconciliation.js";
import { inventoryRemoteWorkspaceMigration, migrateRemoteWorkspace } from "./remote-workspace-migration.js";
import { readRunEvidenceMigrationDeferrals } from "./run-evidence-migration-work.js";
export type OnlineDurableDataMigrationResult = {
state: "completed" | "budget_exhausted" | "unsupported";
@@ -47,6 +48,7 @@ type Owners = {
signal: AbortSignal; sourceLockWaitMs: number }) => Promise<{ alreadyMigrated: boolean; reasons: string[] }>;
nativeHarnessBackups: typeof prestageNativeHarnessBackups;
nativeControllers: typeof prestageNativeControllers;
runEvidenceDeferrals: () => Record<string, number>;
eligibility: (db: Db) => Promise<{ eligible: boolean; blockers: string[] }>;
record: (db: Db, result: PassRecord) => Promise<void>;
};
@@ -75,6 +77,7 @@ const defaultOwners: Owners = {
migrateWorkspace: migrateRemoteWorkspace,
nativeHarnessBackups: prestageNativeHarnessBackups,
nativeControllers: prestageNativeControllers,
runEvidenceDeferrals: readRunEvidenceMigrationDeferrals,
eligibility: readOnlineEphemeralEnrollmentEligibility,
record: recordPass,
};
@@ -132,6 +135,7 @@ async function migrate(db: Db, input: { budgetMs: number; now?: () => number },
const deferred: Record<string, number> = {};
const add = (bucket: Record<string, number>, key: string, by = 1) => { if (by) bucket[key] = (bucket[key] ?? 0) + by; };
const finish = async (state: OnlineDurableDataMigrationResult["state"], reason?: string): Promise<OnlineDurableDataMigrationResult> => {
if (state !== "unsupported") for (const [key, count] of Object.entries(owners.runEvidenceDeferrals())) add(deferred, key, count);
const { eligible, blockers } = await owners.eligibility(db);
const result: OnlineDurableDataMigrationResult = { state, ...(reason ? { reason } : {}),
...(state === "budget_exhausted" ? { resumePhase: PHASES[progress.phase] } : {}), counts, deferred, eligible, blockers };
@@ -175,7 +179,7 @@ async function migrate(db: Db, input: { budgetMs: number; now?: () => number },
const result = await skills.installations({ apply: true, writersStopped: false, after, limit, signal });
for (const row of result.results) {
if (row.status === "converted") add(counts, "skillInstallations");
else if (row.status !== "unchanged") add(deferred, `skills:${row.status}`);
else if (row.status !== "unchanged") add(deferred, `skills:${row.status === "blocked" ? row.code ?? "blocked" : row.status}`);
}
return result.inspected < limit || !result.after ? null : result.after;
}
@@ -5,7 +5,26 @@ import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import { registerDeadlineWork } from "./deadline-work.js";
import { registerIdleLocalWorkProbe, type IdleLocalWork } from "./idle-local-work.js";
import { migrateRunEvidence } from "./run-evidence-migration.js";
import { migrateRunEvidence, type RunEvidenceMigrationItem } from "./run-evidence-migration.js";
const UNPROVEN_NATIVE_COVERAGE = new Set(["native_log_source_coverage_unproven", "native_run_log_source_integrity_error"]);
let completedSweepDeferrals: Record<string, number> = {};
export function readRunEvidenceMigrationDeferrals(): Record<string, number> {
return { ...completedSweepDeferrals };
}
function deferralKey(item: RunEvidenceMigrationItem) {
const code = item.reason && /^[a-z_]{1,60}$/.test(item.reason) ? item.reason : "migration_failed";
return `${item.kind === "log" ? "run_logs" : "provider_traces"}:${code}`;
}
async function archiveSettledNativeLog(db: Db, item: RunEvidenceMigrationItem, signal: AbortSignal) {
if (item.kind !== "log" || item.state !== "blocked" || !UNPROVEN_NATIVE_COVERAGE.has(item.reason ?? "")) return item;
const archived = await migrateRunEvidence(db, { apply: true, runId: item.id, kinds: ["log"], limit: 1,
preserveSettledNativeFiles: true, signal });
return archived.items.find(candidate => candidate.id === item.id) ?? null;
}
export function registerRunEvidenceMigrationWork(
db: Db,
@@ -19,6 +38,7 @@ export function registerRunEvidenceMigrationWork(
tracesDone = false,
blocked = 0,
migrated = 0;
let deferrals: Record<string, number> = {};
let retryMs = 5 * 60_000;
let next: number | null = Date.now();
let pending: IdleLocalWork = "unknown";
@@ -38,9 +58,13 @@ export function registerRunEvidenceMigrationWork(
signal,
});
signal.throwIfAborted();
for (const item of result.items) {
for (const found of result.items) {
const item = await archiveSettledNativeLog(db, found, signal);
signal.throwIfAborted();
if (!item) continue;
if (item.state === "blocked") {
blocked++;
deferrals[deferralKey(item)] = (deferrals[deferralKey(item)] ?? 0) + 1;
logger.warn(
{ kind: item.kind, id: item.id, reason: item.reason },
"Run evidence migration deferred; source storage is still required",
@@ -70,6 +94,8 @@ export function registerRunEvidenceMigrationWork(
runCursor = traceCursor = undefined;
logsDone = tracesDone = false;
blocked = migrated = 0;
completedSweepDeferrals = deferrals;
deferrals = {};
}
},
next: async () => next,
@@ -43,6 +43,7 @@ export async function migrateRunEvidence(
options: {
apply: boolean;
companyId?: string;
runId?: string;
afterRunId?: string;
afterTraceId?: string;
limit?: number;
@@ -124,6 +125,7 @@ export async function migrateRunEvidence(
and(
eq(heartbeatRuns.logStore, "local_file"),
options.companyId ? eq(heartbeatRuns.companyId, options.companyId) : undefined,
options.runId ? eq(heartbeatRuns.id, options.runId) : undefined,
options.afterRunId ? gt(heartbeatRuns.id, options.afterRunId) : undefined,
),
)
@@ -3,7 +3,7 @@ import path from "node:path";
import { and, asc, eq, gt, sql } from "drizzle-orm";
import { companySkills, companySkillVersions, type Db } from "@paperclipai/db";
import { parseFrontmatterMarkdown, type CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared";
import { conflict } from "../errors.js";
import { HttpError } from "../errors.js";
import { classifyInventoryKind, companySkillService } from "./company-skills.js";
import { managedSkillContent } from "./managed-skill-content.js";
import { skillPackageStore, skillVersionMetadataColumns } from "./skill-package-store.js";
@@ -11,6 +11,7 @@ import { skillPackageMaintenance } from "./skill-package-maintenance.js";
import { skillPackageManifest } from "./skill-package-files.js";
import { assertSkillSnapshotPath, MAX_SKILL_SNAPSHOT_BYTES, MAX_SKILL_SNAPSHOT_FILES, skillSnapshotHash, snapshotFile } from "./skill-snapshot.js";
import { protectedVersions, withSkillVersionCacheLock } from "./skill-version-cache.js";
import { readPinnedCatalogSkillFiles } from "./skills-catalog.js";
import { getStorageService, type StorageService } from "../storage/index.js";
type Page = { companyId?: string; after?: string; id?: string; limit?: number; signal?: AbortSignal };
@@ -19,6 +20,11 @@ const limit = (options: Page) => Math.min(50, Math.max(1, options.limit ?? 10));
const legacyManaged = (skill: typeof companySkills.$inferSelect) => skill.sourceType === "local_path" && skill.metadata?.sourceKind === "managed_local";
const linkedLocal = (skill: typeof companySkills.$inferSelect) => skill.sourceType === "local_path"
&& !legacyManaged(skill) && skill.metadata?.sourceKind !== "paperclip_bundled";
class SkillConversionBlocked extends HttpError {
constructor(readonly code: string, message: string, status = 422) { super(status, message); }
}
const blocked = (code: string, message: string, status?: number) => new SkillConversionBlocked(code, message, status);
function coversInstalledInventory(skill: typeof companySkills.$inferSelect, files: Array<{ path: string }>) {
const paths = new Set(files.map(file => file.path));
return paths.has("SKILL.md") && skill.fileInventory.every(file => typeof file.path === "string" && paths.has(file.path));
@@ -32,7 +38,7 @@ async function legacySourceIdentity(skill: typeof companySkills.$inferSelect) {
throw error;
});
if (!stat) return null;
if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Legacy skill root is not a safe directory");
if (!stat.isDirectory() || stat.isSymbolicLink()) throw blocked("legacy_source_unsafe", "Legacy skill root is not a safe directory");
return { root: await fs.realpath(root), device: stat.dev, inode: stat.ino };
}
@@ -45,7 +51,7 @@ async function legacyFiles(skill: typeof companySkills.$inferSelect, signal?: Ab
throw error;
});
if (!stat) return null;
if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Legacy skill root is not a safe directory");
if (!stat.isDirectory() || stat.isSymbolicLink()) throw blocked("legacy_source_unsafe", "Legacy skill root is not a safe directory");
const files: CompanySkillVersionFileInventoryEntry[] = [];
let bytes = 0;
async function visit(directory: string) {
@@ -53,35 +59,47 @@ async function legacyFiles(skill: typeof companySkills.$inferSelect, signal?: Ab
signal?.throwIfAborted();
if (entry.name === ".git" || entry.name === "node_modules") continue;
const absolute = path.join(directory, entry.name);
if (entry.isSymbolicLink()) throw new Error("Legacy skill contains a symbolic link");
if (entry.isSymbolicLink()) throw blocked("legacy_source_unsafe", "Legacy skill contains a symbolic link");
if (entry.isDirectory()) { await visit(absolute); continue; }
if (!entry.isFile()) throw new Error("Legacy skill contains an unsupported file");
if (!entry.isFile()) throw blocked("legacy_source_unsafe", "Legacy skill contains an unsupported file");
const relative = assertSkillSnapshotPath(path.relative(root, absolute).split(path.sep).join("/"));
const handle = await fs.open(absolute, constants.O_RDONLY | constants.O_NOFOLLOW);
try {
const stat = await handle.stat();
if (!stat.isFile() || stat.nlink !== 1) throw new Error("Legacy skill contains an unsupported file");
if (!stat.isFile() || stat.nlink !== 1) throw blocked("legacy_source_unsafe", "Legacy skill contains an unsupported file");
bytes += stat.size;
if (files.length >= MAX_SKILL_SNAPSHOT_FILES || bytes > MAX_SKILL_SNAPSHOT_BYTES) throw new Error("Legacy skill exceeds package limits");
if (files.length >= MAX_SKILL_SNAPSHOT_FILES || bytes > MAX_SKILL_SNAPSHOT_BYTES) throw blocked("legacy_source_too_large", "Legacy skill exceeds package limits");
const body = Buffer.alloc(stat.size);
let offset = 0;
while (offset < body.length) {
signal?.throwIfAborted();
const { bytesRead } = await handle.read(body, offset, body.length - offset, offset);
if (!bytesRead) throw new Error("Legacy skill changed during conversion");
if (!bytesRead) throw blocked("source_changed", "Legacy skill changed during conversion");
offset += bytesRead;
}
const after = await handle.stat();
if (after.size !== stat.size || after.mtimeMs !== stat.mtimeMs) throw new Error("Legacy skill changed during conversion");
if (after.size !== stat.size || after.mtimeMs !== stat.mtimeMs) throw blocked("source_changed", "Legacy skill changed during conversion");
files.push(snapshotFile(relative, classifyInventoryKind(relative), body, Boolean(stat.mode & 0o111)));
} finally { await handle.close(); }
}
}
await visit(root);
if (!files.some(file => file.path === "SKILL.md" && file.encoding !== "base64")) throw new Error("Legacy skill is missing UTF-8 SKILL.md");
if (!files.some(file => file.path === "SKILL.md" && file.encoding !== "base64")) throw blocked("legacy_skill_markdown_missing", "Legacy skill is missing UTF-8 SKILL.md");
return files;
}
async function pinnedCatalogFiles(skill: typeof companySkills.$inferSelect, signal?: AbortSignal) {
const catalogId = skill.metadata?.catalogId, originHash = skill.metadata?.originHash;
if (skill.sourceType !== "catalog" || typeof catalogId !== "string" || typeof originHash !== "string") return null;
let files: Awaited<ReturnType<typeof readPinnedCatalogSkillFiles>>;
try { files = await readPinnedCatalogSkillFiles(catalogId, originHash, signal); }
catch (error) {
if (signal?.aborted) throw error;
throw blocked("catalog_package_unavailable", error instanceof Error ? error.message : "Catalog package is unavailable");
}
return files?.map(file => snapshotFile(assertSkillSnapshotPath(file.path), classifyInventoryKind(file.path), file.bytes)) ?? null;
}
export function skillInstallationMaintenance(db: Db, storage?: StorageService) {
const store = skillPackageStore(db, storage);
const content = managedSkillContent(db, db, storage);
@@ -93,7 +111,7 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) {
}
async function installations(options: Conversion = {}) {
const skills = await db.select().from(companySkills).where(scope(options)).orderBy(asc(companySkills.id)).limit(limit(options));
const results: Array<{ skillId: string; companyId: string; status: string; versionId?: string | null; reason?: string }> = [];
const results: Array<{ skillId: string; companyId: string; status: string; versionId?: string | null; reason?: string; code?: string }> = [];
for (const skill of skills) {
options.signal?.throwIfAborted();
const catalogOrigin = skill.sourceType === "catalog" && !skill.metadata?.originVersionId;
@@ -115,18 +133,25 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) {
const sourceIdentity = await legacySourceIdentity(skill);
const local = await legacyFiles(skill, options.signal);
const completeSaved = saved && coversInstalledInventory(skill, saved) ? saved : null;
const unmodifiedCatalog = skill.sourceType === "catalog" && !skill.metadata?.userModifiedAt
&& typeof skill.metadata?.originHash === "string" && skill.metadata.installedHash === skill.metadata.originHash;
const files = local ?? completeSaved ?? ((skill.sourceType === "github" || skill.sourceType === "skills_sh" || skill.sourceType === "url")
? await companySkillService(db).captureLegacyInstallation(skill.companyId, skill.id) : null);
? await companySkillService(db).captureLegacyInstallation(skill.companyId, skill.id).catch(error => {
if (options.signal?.aborted) throw error;
throw blocked("remote_source_unavailable", error instanceof Error ? error.message : "Remote source is unavailable");
})
: unmodifiedCatalog ? await pinnedCatalogFiles(skill, options.signal) : null);
const originSource = catalogOrigin && typeof skill.metadata?.originSnapshotLocator === "string"
? { ...skill, sourceLocator: skill.metadata.originSnapshotLocator } : null;
const originIdentity = originSource ? await legacySourceIdentity(originSource) : null;
const origin = originSource ? await legacyFiles(originSource, options.signal) : null;
const localOrigin = originSource ? await legacyFiles(originSource, options.signal) : null;
const origin = localOrigin ?? (catalogOrigin && skill.metadata?.userModifiedAt ? await pinnedCatalogFiles(skill, options.signal) : null);
if (catalogOrigin && !origin && skill.metadata?.userModifiedAt) {
throw new Error("Catalog reset snapshot is unavailable; preserve the original installed package before conversion");
throw blocked("catalog_origin_unavailable", "Catalog reset snapshot is unavailable; preserve the original installed package before conversion");
}
if (!files?.length || !files.some(file => file.path === "SKILL.md")) throw new Error("Neither complete legacy files nor a saved version is available");
if (!files?.length || !files.some(file => file.path === "SKILL.md")) throw blocked("source_files_unavailable", "Neither complete legacy files nor a saved version is available");
if (!local && !coversInstalledInventory(skill, files)) {
throw new Error("Saved version does not cover the installed inventory");
throw blocked("saved_version_incomplete", "Saved version does not cover the installed inventory");
}
if (!options.apply) {
return { skillId: skill.id, companyId: skill.companyId, status: "conversion_required", versionId: skill.currentVersionId };
@@ -137,12 +162,12 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) {
}
const candidate = await content.prepare(skill.companyId, files, options.signal);
if (!candidate.packageSnapshotId || !await store.verifyDurable(skill.companyId, candidate.packageSnapshotId, options.signal)) {
throw new Error("Final snapshot is not independently recoverable from object storage");
throw blocked("snapshot_not_durable", "Final snapshot is not independently recoverable from object storage");
}
let originVersionId = typeof skill.metadata?.originVersionId === "string" ? skill.metadata.originVersionId : null;
const originCandidate = catalogOrigin && origin && skillSnapshotHash(origin) !== skillSnapshotHash(files)
? await content.prepare(skill.companyId, origin, options.signal) : null;
if (originCandidate && !originCandidate.packageSnapshotId) throw new Error("Catalog origin is not durably available");
if (originCandidate && !originCandidate.packageSnapshotId) throw blocked("catalog_origin_not_durable", "Catalog origin is not durably available");
const markdown = files.find(file => file.path === "SKILL.md")!.content;
const parsed = parseFrontmatterMarkdown(markdown);
const { missingSource: _missing, ...metadata } = skill.metadata ?? {};
@@ -167,14 +192,14 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) {
if (!current || current.slug !== skill.slug || current.sourceLocator !== skill.sourceLocator
|| current.sourceRef !== skill.sourceRef || JSON.stringify(current.fileInventory) !== JSON.stringify(skill.fileInventory)
|| JSON.stringify(await legacySourceIdentity(current)) !== JSON.stringify(sourceIdentity))
throw conflict("Legacy skill source changed during conversion. Retry the operation.");
throw blocked("source_changed", "Legacy skill source changed during conversion. Retry the operation.", 409);
const latest = await legacyFiles(current, options.signal);
if ((latest ? skillSnapshotHash(latest) : null) !== (local ? skillSnapshotHash(local) : null))
throw conflict("Legacy skill files changed during conversion. Retry the operation.");
throw blocked("source_changed", "Legacy skill files changed during conversion. Retry the operation.", 409);
if (originSource && (JSON.stringify(await legacySourceIdentity(originSource)) !== JSON.stringify(originIdentity)
|| (await legacyFiles(originSource, options.signal).then(value => value ? skillSnapshotHash(value) : null))
!== (origin ? skillSnapshotHash(origin) : null)))
throw conflict("Catalog origin changed during conversion. Retry the operation.");
!== (localOrigin ? skillSnapshotHash(localOrigin) : null)))
throw blocked("source_changed", "Catalog origin changed during conversion. Retry the operation.", 409);
if (origin && originCandidate?.packageSnapshotId) {
originVersionId = (await companySkillService(db).createVersion(skill.companyId, skill.id,
{ label: "Original catalog package" }, { type: "system" }, {
@@ -192,7 +217,10 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) {
};
results.push(await convert());
} catch (error) {
results.push({ skillId: skill.id, companyId: skill.companyId, status: "blocked", reason: error instanceof Error ? error.message : "Conversion failed" });
if (options.signal?.aborted) throw options.signal.reason;
const reason = error instanceof Error ? error.message : "Conversion failed";
results.push({ skillId: skill.id, companyId: skill.companyId, status: "blocked", reason,
code: error instanceof SkillConversionBlocked ? error.code : "blocked" });
}
}
return { inspected: skills.length, after: skills.at(-1)?.id ?? null, results };
+21 -2
View File
@@ -202,6 +202,7 @@ function resolveCatalogSourcePath(source: CatalogSkillSource, relativePath: stri
async function fetchCatalogSourceFile(
skill: CatalogSkill,
relativePath: string,
signal?: AbortSignal,
): Promise<Buffer> {
const source = skill.source;
if (!source) {
@@ -220,7 +221,7 @@ async function fetchCatalogSourceFile(
const sourcePath = resolveCatalogSourcePath(source, relativePath);
const url = resolveRawGitHubUrl(source.hostname, source.owner, source.repo, source.commit, sourcePath);
const response = await ghFetch(url);
const response = await ghFetch(url, signal ? { signal } : undefined);
if (!response.ok) {
throw unprocessable(`Failed to fetch pinned catalog file ${sourcePath} from ${source.owner}/${source.repo}@${source.commit}: HTTP ${response.status}`);
}
@@ -230,13 +231,14 @@ async function fetchCatalogSourceFile(
async function readCatalogFileBytes(
skill: CatalogSkill,
relativePath: string,
signal?: AbortSignal,
): Promise<Buffer> {
const fileEntry = skill.files.find((entry) => entry.path === relativePath);
if (!fileEntry) {
throw notFound("Catalog skill file not found");
}
const bytes = await fetchCatalogSourceFile(skill, relativePath);
const bytes = await fetchCatalogSourceFile(skill, relativePath, signal);
const actualSha = createHash("sha256").update(bytes).digest("hex");
if (actualSha !== fileEntry.sha256) {
throw unprocessable(`Pinned catalog file hash mismatch for ${skill.id}:${relativePath}.`);
@@ -346,6 +348,23 @@ export async function copyCatalogSkillFile(reference: string, relativePath: stri
await fs.writeFile(targetPath, await readCatalogFileBytes(skill, normalizedPath));
}
export async function readPinnedCatalogSkillFiles(reference: string, contentHash: string, signal?: AbortSignal) {
let skill: CatalogSkill | null;
try { skill = resolveCatalogSkillReference(reference).skill; }
catch (error) {
if (isCatalogManifestUnavailableError(error)) return null;
throw error;
}
if (!skill || skill.contentHash !== contentHash) return null;
const pinned = skill;
const files: Array<{ path: string; bytes: Buffer }> = [];
for (const entry of pinned.files) {
signal?.throwIfAborted();
files.push({ path: entry.path, bytes: await readCatalogFileBytes(pinned, entry.path, signal) });
}
return files;
}
export function getCatalogPackageMetadata() {
const catalogManifest = getCatalogManifest();
return {