From 440f96247b77fdca9aaef963e71fb5a139033b9c Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 18 Nov 2026 23:54:10 -0600 Subject: [PATCH] fix(cloud): resolve or name the remaining online move blockers (#22294) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A hosted server can move its durable data to shared storage while it serves work, and then run without a volume > - A bounded online pass moves the data, and a readiness check names what still blocks the move > - Some servers stay blocked after many passes, and the pass reports only `skills:blocked` or `native_harness:source_not_successful`, or reports nothing for run logs > - Operators cannot see why, so they cannot tell recoverable data from data that needs a maintenance step > - This pull request recovers the cases that are safe to recover online and gives every other case a precise code > - The benefit is fewer blocked moves and a clear next step for each one that stays blocked ## Linked Issues or Issue Description Refs #19719 **What happened?** A serving server that runs the online durable-data pass many times stays ineligible. The pass reports `skills:blocked` without a reason, `native_harness:source_not_successful` for several different causes, and no reason at all for run logs that stay on the local disk. **Expected behavior** The pass moves every item that it can move safely while the server serves work. For each item that stays, the pass reports a code that names the cause. **Steps to reproduce** 1. Install a catalog skill on a volume-backed server. Remove its installed files and do not keep a saved version. 2. Keep a settled native run whose local log file has no final size or digest and whose events do not prove full coverage. 3. Run the online durable-data pass. 4. See that the skill is reported only as `skills:blocked` and the run log stays local with no reason in the pass result. **Additional context** The settled native log archive path from #19719 had no online caller. ## What Changed - Skill installation conversion returns a `code` for each blocked installation (for example `source_files_unavailable`, `saved_version_incomplete`, `catalog_origin_unavailable`, `legacy_source_unsafe`, `source_changed`). The pass reports `skills:`. An unknown cause stays `skills:blocked`. - A spent budget during conversion now stops the page. Before, it was reported as a blocked skill and the page moved past it. - A catalog skill whose files are gone converts from the bundled catalog package only when the catalog content hash equals the recorded origin hash and the last audit recorded the installed bytes as equal to that origin with no local changes. - A modified catalog skill whose reset origin is gone gets its origin version from the bundled catalog package only when the catalog content hash equals the recorded origin hash. - Native harness pre-copy replaces `source_not_successful` with `source_run_active`, `source_run_unfinished`, `source_coordinator_missing`, `source_coordinator_unsettled`, `source_session_not_selected` and `source_lease_missing`. The settled-source rules do not change. - The run evidence worker archives a settled native log by run id when the canonical import cannot prove event coverage. The existing settled-source guard still decides. The worker records the reasons from its last completed sweep, and the pass reports them as `run_logs:` and `provider_traces:`. - `migrateRunEvidence` accepts a `runId` filter. ## Verification - `cd server && npx vitest run src/__tests__/managed-skill-content.test.ts` (embedded PostgreSQL): new catalog recovery, hash mismatch, unproven install and budget abort tests fail before the change and pass after it. - `npx vitest run src/services/native-runtime/native-harness-online-preservation.integration.test.ts src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: the new reason tests fail before the change and pass after it. - `npx vitest run src/__tests__/run-evidence-migration-work.test.ts src/services/native-log-archive.integration.test.ts`: the worker archive test fails before the change and passes after it. The integration test shows a real settled source that is blocked as `native_log_source_coverage_unproven` and then archived by run id. - `npx vitest run src/services/online-durable-data-migration.test.ts src/services/online-durable-data-migration-postgres.test.ts` - `pnpm --filter @paperclipai/server typecheck` ## Risks - A settled native log with unproven coverage now moves to the read-only historical archive instead of staying local. The archive keeps the exact bytes. It does not claim canonical event coverage. This is the same result that the offline path gives. - The catalog fallback reads bundled or pinned catalog bytes. Each file is checked against its pinned SHA-256, and the package must match the recorded origin hash. - Pass deferral codes change. Readers that match `native_harness:source_not_successful` must use the new codes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 --- .../__tests__/managed-skill-content.test.ts | 100 +++++++++++++++++- .../run-evidence-migration-work.test.ts | 26 ++++- .../native-log-archive.integration.test.ts | 12 +++ .../native-harness-backup-migration.ts | 8 +- ...harness-backup-storage.integration.test.ts | 2 +- ...ss-online-preservation.integration.test.ts | 20 +++- .../native-harness-preservation-source.ts | 11 +- ...ne-durable-data-migration-postgres.test.ts | 1 + .../online-durable-data-migration.test.ts | 14 +++ .../services/online-durable-data-migration.ts | 6 +- .../services/run-evidence-migration-work.ts | 30 +++++- server/src/services/run-evidence-migration.ts | 2 + .../skill-installation-maintenance.ts | 74 +++++++++---- server/src/services/skills-catalog.ts | 23 +++- 14 files changed, 287 insertions(+), 42 deletions(-) diff --git a/server/src/__tests__/managed-skill-content.test.ts b/server/src/__tests__/managed-skill-content.test.ts index 1430bd763c3..92a993769a4 100644 --- a/server/src/__tests__/managed-skill-content.test.ts +++ b/server/src/__tests__/managed-skill-content.test.ts @@ -18,6 +18,7 @@ import path from "node:path"; import { afterAll, beforeAll, describe, expect, it, vi } from "vitest"; import { applyPendingMigrations, agents, heartbeatRuns, companySkills, companySkillVersions, companies, createDb, instanceSettings } from "@paperclipai/db"; import { companySkillService } from "../services/company-skills.js"; +import { listCatalogSkills, readCatalogSkillFile } from "../services/skills-catalog.js"; import * as storageModule from "../storage/index.js"; import { createStorageService } from "../storage/service.js"; import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; @@ -618,7 +619,7 @@ describe.skipIf(!support.supported)("managed skills use committed versions", () markdown: "# Linked", sourceType: "local_path", sourceLocator: path.join(root, "linked") }).returning(); const report = await skillInstallationMaintenance(db).installations({ companyId, apply: true, writersStopped: true }); expect(report.results).toEqual(expect.arrayContaining([ - expect.objectContaining({ skillId: legacy!.id, status: "blocked" }), + expect.objectContaining({ skillId: legacy!.id, status: "blocked", code: "source_files_unavailable" }), expect.objectContaining({ skillId: linked!.id, status: "linked_local" }), ])); expect(await companySkillService(db).getById(companyId, legacy!.id)).toMatchObject({ sourceType: "local_path", currentVersionId: null }); @@ -680,11 +681,106 @@ describe.skipIf(!support.supported)("managed skills use committed versions", () const [legacy] = await db.insert(companySkills).values({ companyId, key: "unsafe", slug: "unsafe", name: "Unsafe", markdown: "# legacy", sourceType: "local_path", sourceLocator: directory, metadata: { sourceKind: "managed_local" } }).returning(); expect((await skillInstallationMaintenance(db).installations({ id: legacy!.id, apply: true, writersStopped: true })).results) - .toEqual([expect.objectContaining({ status: "blocked", reason: "Legacy skill contains a symbolic link" })]); + .toEqual([expect.objectContaining({ status: "blocked", reason: "Legacy skill contains a symbolic link", code: "legacy_source_unsafe" })]); expect(await companySkillService(db).getById(companyId, legacy!.id)).toMatchObject({ sourceType: "local_path", currentVersionId: null }); } finally { await fs.unlink(path.join(directory, "support.txt")); } }); + it("converts an unmodified catalog installation whose files are gone from its pinned catalog package", async () => { + const companyId = await company(), catalog = listCatalogSkills().find(entry => !entry.source)!; + const [skill] = await db.insert(companySkills).values({ companyId, key: `catalog-${randomUUID()}`, slug: `catalog-${randomUUID().slice(0, 8)}`, + name: "Catalog", markdown: "# stale row", sourceType: "catalog", sourceLocator: path.join(root, `catalog-missing-${randomUUID()}`), + sourceRef: catalog.contentHash, fileInventory: catalog.files.map(({ path, kind }) => ({ path, kind })), + metadata: { sourceKind: "catalog", catalogId: catalog.id, originHash: catalog.contentHash, installedHash: catalog.contentHash, userModifiedAt: null }, + }).returning(); + const maintenance = skillInstallationMaintenance(db); + expect((await maintenance.installations({ id: skill.id, apply: true })).results).toEqual([expect.objectContaining({ status: "converted" })]); + const [row] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id)); + expect(row.metadata?.originVersionId).toBe(row.currentVersionId); + expect((await maintenance.eligibility({ id: skill.id })).results[0]).toMatchObject({ status: "version_backed" }); + const store = skillPackageStore(db), saved = await store.version(companyId, skill.id, row.currentVersionId!); + expect((await store.versionFiles(saved!)).find(file => file.path === "SKILL.md")?.content) + .toBe((await readCatalogSkillFile(catalog.id, "SKILL.md")).content); + }); + + it("keeps an unmodified catalog installation blocked when it was not proven identical to its catalog origin", async () => { + const companyId = await company(), catalog = listCatalogSkills().find(entry => !entry.source)!; + const [skill] = await db.insert(companySkills).values({ companyId, key: `catalog-${randomUUID()}`, slug: `catalog-${randomUUID().slice(0, 8)}`, + name: "Catalog", markdown: "# stale row", sourceType: "catalog", sourceLocator: path.join(root, `catalog-missing-${randomUUID()}`), + sourceRef: catalog.contentHash, fileInventory: catalog.files.map(({ path, kind }) => ({ path, kind })), + metadata: { sourceKind: "catalog", catalogId: catalog.id, originHash: catalog.contentHash, userModifiedAt: null }, + }).returning(); + expect((await skillInstallationMaintenance(db).installations({ id: skill.id, apply: true })).results) + .toEqual([expect.objectContaining({ status: "blocked", code: "source_files_unavailable" })]); + expect((await db.select().from(companySkills).where(eq(companySkills.id, skill.id)))[0].currentVersionId).toBeNull(); + }); + + it("recovers a modified catalog installation's missing reset origin only from the exact pinned catalog package", async () => { + const companyId = await company(), service = companySkillService(db), catalog = listCatalogSkills().find(entry => !entry.source)!; + const skill = await service.createLocalSkill(companyId, { name: "Modified catalog", markdown: "# Locally modified catalog skill" }); + const metadata = { sourceKind: "catalog", catalogId: catalog.id, originHash: `sha256:${"0".repeat(64)}`, + installedHash: `sha256:${"1".repeat(64)}`, userModifiedAt: new Date().toISOString(), + originSnapshotLocator: path.join(root, `catalog-origin-missing-${randomUUID()}`) }; + await db.update(companySkills).set({ sourceType: "catalog", sourceLocator: path.join(root, `catalog-installed-missing-${randomUUID()}`), + fileInventory: [{ path: "SKILL.md", kind: "skill" }], metadata }).where(eq(companySkills.id, skill.id)); + const maintenance = skillInstallationMaintenance(db); + expect((await maintenance.installations({ id: skill.id, apply: true })).results) + .toEqual([expect.objectContaining({ status: "blocked", code: "catalog_origin_unavailable" })]); + await db.update(companySkills).set({ metadata: { ...metadata, originHash: catalog.contentHash } }).where(eq(companySkills.id, skill.id)); + expect((await maintenance.installations({ id: skill.id, apply: true })).results) + .toEqual([expect.objectContaining({ status: "converted", versionId: skill.currentVersionId })]); + const [row] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id)); + const originVersionId = row.metadata?.originVersionId as string; + expect(originVersionId).toBeTruthy(); + expect(originVersionId).not.toBe(row.currentVersionId); + const store = skillPackageStore(db); + const origin = await store.versionFiles((await store.version(companyId, skill.id, originVersionId))!); + expect(origin.find(file => file.path === "SKILL.md")?.content).toBe((await readCatalogSkillFile(catalog.id, "SKILL.md")).content); + const current = await store.versionFiles((await store.version(companyId, skill.id, row.currentVersionId!))!); + expect(current.find(file => file.path === "SKILL.md")?.content).toBe("# Locally modified catalog skill"); + expect((await maintenance.eligibility({ id: skill.id })).results[0]).toMatchObject({ status: "version_backed" }); + }); + + it("cancels a stalled pinned catalog download when the budget expires", async () => { + const companyId = await company(), catalog = listCatalogSkills().find(entry => entry.source)!; + const [skill] = await db.insert(companySkills).values({ companyId, key: `catalog-${randomUUID()}`, slug: `catalog-${randomUUID().slice(0, 8)}`, + name: "Remote catalog", markdown: "# stale row", sourceType: "catalog", sourceLocator: path.join(root, `catalog-missing-${randomUUID()}`), + sourceRef: catalog.contentHash, fileInventory: catalog.files.map(({ path, kind }) => ({ path, kind })), + metadata: { sourceKind: "catalog", catalogId: catalog.id, originHash: catalog.contentHash, installedHash: catalog.contentHash, userModifiedAt: null }, + }).returning(); + const budget = new AbortController(); + const download = vi.spyOn(globalThis, "fetch").mockImplementation((_url, init) => new Promise((_resolve, reject) => { + const signal = init?.signal; + if (!signal) return; + signal.addEventListener("abort", () => reject(signal.reason), { once: true }); + budget.abort(new Error("budget spent")); + })); + try { + await expect(skillInstallationMaintenance(db).installations({ id: skill.id, apply: true, signal: budget.signal })) + .rejects.toThrow("budget spent"); + } finally { download.mockRestore(); } + expect((await db.select().from(companySkills).where(eq(companySkills.id, skill.id)))[0].currentVersionId).toBeNull(); + }); + + it("stops at an aborted budget instead of reporting the installation as blocked", async () => { + const companyId = await company(), directory = path.join(root, `aborted-conversion-${randomUUID()}`); + await fs.mkdir(directory); + await fs.writeFile(path.join(directory, "SKILL.md"), "# Aborted conversion"); + const [skill] = await db.insert(companySkills).values({ companyId, key: "aborted-conversion", slug: "aborted-conversion", name: "Aborted", + markdown: "# Aborted conversion", sourceType: "local_path", sourceLocator: directory, + fileInventory: [{ path: "SKILL.md", kind: "skill" }], metadata: { sourceKind: "managed_local" } }).returning(); + const storage = storageModule.getStorageService(), budget = new AbortController(); + const upload = vi.spyOn(storage, "putFile").mockImplementation(async () => { + budget.abort(new Error("budget spent")); + throw budget.signal.reason; + }); + try { + await expect(skillInstallationMaintenance(db, storage).installations({ id: skill.id, apply: true, signal: budget.signal })) + .rejects.toThrow("budget spent"); + } finally { upload.mockRestore(); } + expect((await db.select().from(companySkills).where(eq(companySkills.id, skill.id)))[0].currentVersionId).toBeNull(); + }); + it("keeps the compatibility override durable in Postgres and reports that storage accurately", async () => { const companyId = await company(); process.env.PAPERCLIP_SKILL_PACKAGE_STORAGE_ENABLED = "false"; diff --git a/server/src/__tests__/run-evidence-migration-work.test.ts b/server/src/__tests__/run-evidence-migration-work.test.ts index 79023be9aa8..4ffad25da3e 100644 --- a/server/src/__tests__/run-evidence-migration-work.test.ts +++ b/server/src/__tests__/run-evidence-migration-work.test.ts @@ -2,7 +2,7 @@ import type { Db } from "@paperclipai/db"; import { installDatabaseWorkSignals } from "../../../packages/db/src/work-signals.js"; import { afterEach, expect, it, vi } from "vitest"; import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js"; -import { registerRunEvidenceMigrationWork } from "../services/run-evidence-migration-work.js"; +import { readRunEvidenceMigrationDeferrals, registerRunEvidenceMigrationWork } from "../services/run-evidence-migration-work.js"; const state = vi.hoisted(() => ({ provider: "s3", @@ -125,6 +125,30 @@ it("keeps blocked history visible and retries it without stranding later pages", expect(state.probe!()).toBe("none"); expect(coordinator!.nextWakeAt()).toBeNull(); }); +it("archives a settled native log whose canonical coverage is unproven and reports what stays blocked", async () => { + state.migrate + .mockResolvedValueOnce({ + items: [ + { kind: "log", id: "settled", state: "blocked", reason: "native_log_source_coverage_unproven" }, + { kind: "log", id: "unsettled", state: "blocked", reason: "native_log_source_coverage_unproven" }, + { kind: "trace", id: "trace", state: "blocked", reason: "Trace file /data/secret is missing" }, + ], + }) + .mockResolvedValueOnce({ items: [{ kind: "log", id: "settled", state: "migrated", historicalArchive: true }] }) + .mockResolvedValueOnce({ items: [{ kind: "log", id: "unsettled", state: "blocked", reason: "native_log_archive_source_unsettled" }] }) + .mockResolvedValue({ items: [] }); + const { worker } = setup(); + await worker!.ready; + expect(state.migrate).toHaveBeenNthCalledWith(2, expect.anything(), expect.objectContaining({ + apply: true, runId: "settled", kinds: ["log"], limit: 1, preserveSettledNativeFiles: true })); + expect(state.migrate).toHaveBeenNthCalledWith(3, expect.anything(), expect.objectContaining({ + runId: "unsettled", preserveSettledNativeFiles: true })); + expect(readRunEvidenceMigrationDeferrals()).toEqual({ + "run_logs:native_log_archive_source_unsettled": 1, + "provider_traces:migration_failed": 1, + }); + expect(state.probe!()).toBe("present"); +}); it("retains startup work during drain and retries a failed sweep before becoming quiet", async () => { let admitted = false; state.migrate diff --git a/server/src/services/native-log-archive.integration.test.ts b/server/src/services/native-log-archive.integration.test.ts index 8dcd6c70d4f..910284928f0 100644 --- a/server/src/services/native-log-archive.integration.test.ts +++ b/server/src/services/native-log-archive.integration.test.ts @@ -75,6 +75,18 @@ describeEmbeddedPostgres("online native historical log preservation", () => { }); }); + it("archives by run id a settled source whose canonical online import cannot prove coverage", async () => { + await withSource(async f => { + const { preserveSettledNativeFiles: _archive, ...canonical } = f.options; + expect((await migrateRunEvidence(f.db, canonical)).items) + .toMatchObject([{ id: f.runId, state: "blocked", reason: "native_log_source_coverage_unproven" }]); + const result = await migrateRunEvidence(f.db, { apply: true, runId: f.runId, kinds: ["log"], limit: 1, preserveSettledNativeFiles: true }); + expect(result.items).toMatchObject([{ id: f.runId, state: "migrated", historicalArchive: true, bytes: f.bytes.length }]); + expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]) + .toMatchObject({ logStore: "s3_archive", logRef: `${f.companyId}/${f.agentId}/${f.runId}.archive` }); + }); + }); + it("preserves a committed successful run without changing its result or completeness metadata", async () => { await withSource(async f => { const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); diff --git a/server/src/services/native-runtime/native-harness-backup-migration.ts b/server/src/services/native-runtime/native-harness-backup-migration.ts index 0811b360a8b..1afac2805a8 100644 --- a/server/src/services/native-runtime/native-harness-backup-migration.ts +++ b/server/src/services/native-runtime/native-harness-backup-migration.ts @@ -211,14 +211,16 @@ export async function prestageNativeHarnessBackups(db: Db, input: { storage: Sto return "current"; } if (run.status !== "succeeded" || !run.finishedAt) { - if (!run.finishedAt || !["failed", "cancelled", "interrupted", "timed_out"].includes(run.status)) return "source_not_successful"; + if (!run.finishedAt || !["failed", "cancelled", "interrupted", "timed_out"].includes(run.status)) { + return ["queued", "scheduled_retry", "running"].includes(run.status) ? "source_run_active" : "source_run_unfinished"; + } const providerLeaseId = await verifiedSourceLease(db, source, run, signal); if (!providerLeaseId) return "source_unverified"; try { - const owner = await settledNativeHarnessBackupOwner(db, { companyId: run.companyId, issueId: run.issueId, + const { owner, reason } = await settledNativeHarnessBackupOwner(db, { companyId: run.companyId, issueId: run.issueId, runId: source.runId, sessionId: run.sessionId, runnerInstanceId: run.runnerId, scopeDigest: source.scopeDigest, root: source.root, manifestSha256: source.manifestSha256, providerLeaseId, signal }); - if (!owner) return "source_not_successful"; + if (!owner) return `source_${reason}`; await persistNativeHarnessBackup({ db, storage: input.storage, owner, root: source.root, scopeDigest: source.scopeDigest, signal, maxArchiveBytes: source.archiveLimit }); return "published"; diff --git a/server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts b/server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts index caf22cf696d..65b648d3b84 100644 --- a/server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts +++ b/server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts @@ -612,7 +612,7 @@ describeEmbeddedPostgres("durable native provider backups", () => { it("defers failed history without treating pre-copy as offline authority", async () => { const [f] = await offlineFixtures(1); expect(await prestageNativeHarnessBackups(f.db, { storage: f.storage, signal: AbortSignal.timeout(15_000) })) - .toEqual({ inspected: 1, published: 0, current: 0, deferred: { source_not_successful: 1 } }); + .toEqual({ inspected: 1, published: 0, current: 0, deferred: { source_session_not_selected: 1 } }); expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toEqual([]); }); it("defers a source being replaced online while offline migration still refuses it", async () => { diff --git a/server/src/services/native-runtime/native-harness-online-preservation.integration.test.ts b/server/src/services/native-runtime/native-harness-online-preservation.integration.test.ts index b334c96f5a6..2a22ca49cbf 100644 --- a/server/src/services/native-runtime/native-harness-online-preservation.integration.test.ts +++ b/server/src/services/native-runtime/native-harness-online-preservation.integration.test.ts @@ -90,7 +90,7 @@ describeEmbeddedPostgres("online preservation of settled legacy harness backups" await f.db.update(nativeRunFinalizations).set({ leaseOwner: "original-owner", leaseExpiresAt: new Date(0) }) .where(eq(nativeRunFinalizations.runId, f.runId)); expect(await prestageNativeHarnessBackups(f.db, { storage: f.storage, signal: AbortSignal.timeout(15_000) })) - .toMatchObject({ published: 0, deferred: { source_not_successful: 1 } }); + .toMatchObject({ published: 0, deferred: { source_coordinator_unsettled: 1 } }); await f.db.update(nativeRunFinalizations).set({ leaseOwner: null }).where(eq(nativeRunFinalizations.runId, f.runId)); const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)); await f.db.update(heartbeatRuns).set({ runnerProfileJson: { ...run.runnerProfileJson, nativeWorkspaceFinalizationOwner: { token: "physical-owner" } } }) @@ -99,6 +99,22 @@ describeEmbeddedPostgres("online preservation of settled legacy harness backups" .toMatchObject({ published: 0, deferred: { source_changed: 1 } }); expect(put).not.toHaveBeenCalled(); }); + it.each([ + ["running", "source_run_active"], + ["scheduled_retry", "source_run_active"], + ["failed_unfinished", "source_run_unfinished"], + ["coordinator_missing", "source_coordinator_missing"], + ["session_not_selected", "source_session_not_selected"], + ])("names why a %s source stays local", async (kind, outcome) => { + const f = await fixture(), put = vi.spyOn(f.storage, "putFile"); + if (kind === "running" || kind === "scheduled_retry") await f.db.update(heartbeatRuns).set({ status: kind, finishedAt: null }).where(eq(heartbeatRuns.id, f.runId)); + if (kind === "failed_unfinished") await f.db.update(heartbeatRuns).set({ finishedAt: null }).where(eq(heartbeatRuns.id, f.runId)); + if (kind === "coordinator_missing") await f.db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)); + if (kind === "session_not_selected") await f.db.delete(agentTaskSessions).where(eq(agentTaskSessions.lastRunId, f.runId)); + expect(await prestageNativeHarnessBackups(f.db, { storage: f.storage, signal: AbortSignal.timeout(15_000) })) + .toMatchObject({ published: 0, deferred: { [outcome]: 1 } }); + expect(put).not.toHaveBeenCalled(); + }); it("respects the physical advisory lock while unrelated work stays active", async () => { const f = await fixture(), put = vi.spyOn(f.storage, "putFile"); let release!: () => void, entered!: () => void; @@ -139,7 +155,7 @@ describeEmbeddedPostgres("online preservation of settled legacy harness backups" }); it("binds the archive payload and scope to the checked preservation owner", async () => { const f = await fixture(), signal = AbortSignal.timeout(15_000); - const owner = await settledNativeHarnessBackupOwner(f.db, { ...f, sessionId: f.normalizedSessionId, + const { owner } = await settledNativeHarnessBackupOwner(f.db, { ...f, sessionId: f.normalizedSessionId, manifestSha256: f.stamp.manifestSha256, signal }); expect(owner).not.toBeNull(); const put = vi.spyOn(f.storage, "putFile"); diff --git a/server/src/services/native-runtime/native-harness-preservation-source.ts b/server/src/services/native-runtime/native-harness-preservation-source.ts index de6c36d5dd8..8564ba6e101 100644 --- a/server/src/services/native-runtime/native-harness-preservation-source.ts +++ b/server/src/services/native-runtime/native-harness-preservation-source.ts @@ -37,22 +37,23 @@ const fail = (): never => { throw new Error("native_harness_preservation_source_ export async function settledNativeHarnessBackupOwner(db: Db, input: { companyId: string; issueId: string; runId: string; sessionId: string; runnerInstanceId: string; scopeDigest: string; root: string; manifestSha256: string; providerLeaseId: string; signal: AbortSignal; -}): Promise { +}): Promise<{ owner: SettledNativeHarnessBackupOwner; reason?: never } + | { owner: null; reason: "coordinator_missing" | "coordinator_unsettled" | "session_not_selected" | "lease_missing" }> { const [run] = await db.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId))); const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId), eq(nativeRunFinalizations.runId, input.runId))); - if (!run || !coordinator || !settled(coordinator)) return null; + if (!run || !coordinator || !settled(coordinator)) return { owner: null, reason: run && coordinator ? "coordinator_unsettled" : "coordinator_missing" }; const selected = await db.select({ id: agentTaskSessions.id, lastRunId: agentTaskSessions.lastRunId }).from(agentTaskSessions) .innerJoin(heartbeatRuns, and(eq(heartbeatRuns.companyId, agentTaskSessions.companyId), eq(heartbeatRuns.agentId, agentTaskSessions.agentId), eq(heartbeatRuns.id, agentTaskSessions.lastRunId))).where(and(eq(agentTaskSessions.companyId, input.companyId), eq(agentTaskSessions.agentId, run.agentId), eq(agentTaskSessions.adapterType, "paperclip_runner"), eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.nativeSessionId, input.sessionId))).limit(2); - if (selected.length !== 1 || !selected[0]!.lastRunId) return null; + if (selected.length !== 1 || !selected[0]!.lastRunId) return { owner: null, reason: "session_not_selected" }; const leases = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.issueId, input.issueId), eq(environmentLeases.providerLeaseId, input.providerLeaseId), sql`${environmentLeases.metadata}->'nativeHarnessBackup'->>'manifestSha256' = ${input.manifestSha256}`)); const lease = leases.find(row => readNativeHarnessBackupStamp(row.metadata?.nativeHarnessBackup, input.providerLeaseId)?.sessionScopeSha256 === input.scopeDigest); - if (!lease) return null; + if (!lease) return { owner: null, reason: "lease_missing" }; const owner: SettledNativeHarnessBackupOwner = { kind: "settled_native_harness_backup", companyId: input.companyId, issueId: input.issueId, agentId: run.agentId, runId: input.runId, sessionId: input.sessionId, runnerInstanceId: input.runnerInstanceId, scopeDigest: input.scopeDigest, root: input.root, manifestSha256: input.manifestSha256, @@ -60,7 +61,7 @@ export async function settledNativeHarnessBackupOwner(db: Db, input: { sourceLeaseId: lease.id, providerLeaseId: input.providerLeaseId, runDigest: runDigest(run), coordinatorDigest: coordinatorDigest(coordinator), leaseDigest: leaseDigest(lease) }; await db.transaction(tx => assertSettledNativeHarnessBackupSource(tx as unknown as Db, owner, input.signal)); - return owner; + return { owner }; } export async function assertSettledNativeHarnessBackupSource(db: Db, owner: SettledNativeHarnessBackupOwner, diff --git a/server/src/services/online-durable-data-migration-postgres.test.ts b/server/src/services/online-durable-data-migration-postgres.test.ts index 5b22b9dce2d..44ea1d26d1b 100644 --- a/server/src/services/online-durable-data-migration-postgres.test.ts +++ b/server/src/services/online-durable-data-migration-postgres.test.ts @@ -77,6 +77,7 @@ describe.skipIf(!support.supported)("online durable-data migration of local work migrateWorkspace: migrateRemoteWorkspace, nativeHarnessBackups: async () => undefined, nativeControllers: async () => undefined, + runEvidenceDeferrals: () => ({}), eligibility: async () => ({ eligible: false, blockers: [] }), record: async () => undefined, }) as never; diff --git a/server/src/services/online-durable-data-migration.test.ts b/server/src/services/online-durable-data-migration.test.ts index a157952367c..a1d9e33de31 100644 --- a/server/src/services/online-durable-data-migration.test.ts +++ b/server/src/services/online-durable-data-migration.test.ts @@ -39,6 +39,7 @@ function owners(overrides: Record = {}) { for (const outcome of ["imported", "unchanged"]) input.onOutcome?.(outcome); return { inspected: 2, imported: 1, unchanged: 1, archivedFailed: 0, sourceAlreadyAbsent: 0, deferred: {} }; }, + runEvidenceDeferrals: () => ({}), eligibility: vi.fn(async () => ({ eligible: false, blockers: ["skills_require_migration", "volume_free_storage_flags_required"] })), record: vi.fn(async () => undefined), ...overrides, @@ -65,6 +66,19 @@ describe("online durable-data migration pass", () => { expect(fixture.record).toHaveBeenCalledWith(db, result); }); + it("names why skill installations and retained run logs stay behind", async () => { + const fixture = owners({ + skillInstallations: () => ({ installations: async () => ({ inspected: 3, after: null, results: [ + { skillId: "s1", companyId: "c", status: "blocked", reason: "Saved version does not cover the installed inventory", code: "saved_version_incomplete" }, + { skillId: "s2", companyId: "c", status: "blocked", reason: "network down" }, + { skillId: "s3", companyId: "c", status: "linked_local" }] }) }), + runEvidenceDeferrals: () => ({ "run_logs:native_log_archive_source_unsettled": 2 }), + }); + const result = await runOnlineDurableDataMigration(db, { budgetMs: 60_000 }, fixture.owners); + expect(result.deferred).toMatchObject({ "skills:saved_version_incomplete": 1, "skills:blocked": 1, "skills:linked_local": 1, + "run_logs:native_log_archive_source_unsettled": 2 }); + }); + it("migrates agent files with live writers, never the offline mode", async () => { const agentFiles = vi.fn(async () => ({ results: [], next: null })); await runOnlineDurableDataMigration(db, { budgetMs: 60_000 }, owners({ agentFiles }).owners); diff --git a/server/src/services/online-durable-data-migration.ts b/server/src/services/online-durable-data-migration.ts index 635c3c16326..e38b0de1b63 100644 --- a/server/src/services/online-durable-data-migration.ts +++ b/server/src/services/online-durable-data-migration.ts @@ -17,6 +17,7 @@ import { prestageNativeControllers } from "./native-runtime/native-controller-mi import { listLocalWorkspaceCompanies, listLocalWorkspaces } from "./local-workspace-dependencies.js"; import { reconcileOnlineWorkspaceOperations } from "./offline-workspace-operation-reconciliation.js"; import { inventoryRemoteWorkspaceMigration, migrateRemoteWorkspace } from "./remote-workspace-migration.js"; +import { readRunEvidenceMigrationDeferrals } from "./run-evidence-migration-work.js"; export type OnlineDurableDataMigrationResult = { state: "completed" | "budget_exhausted" | "unsupported"; @@ -47,6 +48,7 @@ type Owners = { signal: AbortSignal; sourceLockWaitMs: number }) => Promise<{ alreadyMigrated: boolean; reasons: string[] }>; nativeHarnessBackups: typeof prestageNativeHarnessBackups; nativeControllers: typeof prestageNativeControllers; + runEvidenceDeferrals: () => Record; eligibility: (db: Db) => Promise<{ eligible: boolean; blockers: string[] }>; record: (db: Db, result: PassRecord) => Promise; }; @@ -75,6 +77,7 @@ const defaultOwners: Owners = { migrateWorkspace: migrateRemoteWorkspace, nativeHarnessBackups: prestageNativeHarnessBackups, nativeControllers: prestageNativeControllers, + runEvidenceDeferrals: readRunEvidenceMigrationDeferrals, eligibility: readOnlineEphemeralEnrollmentEligibility, record: recordPass, }; @@ -132,6 +135,7 @@ async function migrate(db: Db, input: { budgetMs: number; now?: () => number }, const deferred: Record = {}; const add = (bucket: Record, key: string, by = 1) => { if (by) bucket[key] = (bucket[key] ?? 0) + by; }; const finish = async (state: OnlineDurableDataMigrationResult["state"], reason?: string): Promise => { + if (state !== "unsupported") for (const [key, count] of Object.entries(owners.runEvidenceDeferrals())) add(deferred, key, count); const { eligible, blockers } = await owners.eligibility(db); const result: OnlineDurableDataMigrationResult = { state, ...(reason ? { reason } : {}), ...(state === "budget_exhausted" ? { resumePhase: PHASES[progress.phase] } : {}), counts, deferred, eligible, blockers }; @@ -175,7 +179,7 @@ async function migrate(db: Db, input: { budgetMs: number; now?: () => number }, const result = await skills.installations({ apply: true, writersStopped: false, after, limit, signal }); for (const row of result.results) { if (row.status === "converted") add(counts, "skillInstallations"); - else if (row.status !== "unchanged") add(deferred, `skills:${row.status}`); + else if (row.status !== "unchanged") add(deferred, `skills:${row.status === "blocked" ? row.code ?? "blocked" : row.status}`); } return result.inspected < limit || !result.after ? null : result.after; } diff --git a/server/src/services/run-evidence-migration-work.ts b/server/src/services/run-evidence-migration-work.ts index 2ff3b1c8843..21dcbebc66e 100644 --- a/server/src/services/run-evidence-migration-work.ts +++ b/server/src/services/run-evidence-migration-work.ts @@ -5,7 +5,26 @@ import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator. import { DELIVERY_QUEUES } from "./delivery-work-notifications.js"; import { registerDeadlineWork } from "./deadline-work.js"; import { registerIdleLocalWorkProbe, type IdleLocalWork } from "./idle-local-work.js"; -import { migrateRunEvidence } from "./run-evidence-migration.js"; +import { migrateRunEvidence, type RunEvidenceMigrationItem } from "./run-evidence-migration.js"; + +const UNPROVEN_NATIVE_COVERAGE = new Set(["native_log_source_coverage_unproven", "native_run_log_source_integrity_error"]); +let completedSweepDeferrals: Record = {}; + +export function readRunEvidenceMigrationDeferrals(): Record { + return { ...completedSweepDeferrals }; +} + +function deferralKey(item: RunEvidenceMigrationItem) { + const code = item.reason && /^[a-z_]{1,60}$/.test(item.reason) ? item.reason : "migration_failed"; + return `${item.kind === "log" ? "run_logs" : "provider_traces"}:${code}`; +} + +async function archiveSettledNativeLog(db: Db, item: RunEvidenceMigrationItem, signal: AbortSignal) { + if (item.kind !== "log" || item.state !== "blocked" || !UNPROVEN_NATIVE_COVERAGE.has(item.reason ?? "")) return item; + const archived = await migrateRunEvidence(db, { apply: true, runId: item.id, kinds: ["log"], limit: 1, + preserveSettledNativeFiles: true, signal }); + return archived.items.find(candidate => candidate.id === item.id) ?? null; +} export function registerRunEvidenceMigrationWork( db: Db, @@ -19,6 +38,7 @@ export function registerRunEvidenceMigrationWork( tracesDone = false, blocked = 0, migrated = 0; + let deferrals: Record = {}; let retryMs = 5 * 60_000; let next: number | null = Date.now(); let pending: IdleLocalWork = "unknown"; @@ -38,9 +58,13 @@ export function registerRunEvidenceMigrationWork( signal, }); signal.throwIfAborted(); - for (const item of result.items) { + for (const found of result.items) { + const item = await archiveSettledNativeLog(db, found, signal); + signal.throwIfAborted(); + if (!item) continue; if (item.state === "blocked") { blocked++; + deferrals[deferralKey(item)] = (deferrals[deferralKey(item)] ?? 0) + 1; logger.warn( { kind: item.kind, id: item.id, reason: item.reason }, "Run evidence migration deferred; source storage is still required", @@ -70,6 +94,8 @@ export function registerRunEvidenceMigrationWork( runCursor = traceCursor = undefined; logsDone = tracesDone = false; blocked = migrated = 0; + completedSweepDeferrals = deferrals; + deferrals = {}; } }, next: async () => next, diff --git a/server/src/services/run-evidence-migration.ts b/server/src/services/run-evidence-migration.ts index 2aee95888ac..1b7c530f18d 100644 --- a/server/src/services/run-evidence-migration.ts +++ b/server/src/services/run-evidence-migration.ts @@ -43,6 +43,7 @@ export async function migrateRunEvidence( options: { apply: boolean; companyId?: string; + runId?: string; afterRunId?: string; afterTraceId?: string; limit?: number; @@ -124,6 +125,7 @@ export async function migrateRunEvidence( and( eq(heartbeatRuns.logStore, "local_file"), options.companyId ? eq(heartbeatRuns.companyId, options.companyId) : undefined, + options.runId ? eq(heartbeatRuns.id, options.runId) : undefined, options.afterRunId ? gt(heartbeatRuns.id, options.afterRunId) : undefined, ), ) diff --git a/server/src/services/skill-installation-maintenance.ts b/server/src/services/skill-installation-maintenance.ts index 2aef82362ba..5360d80a129 100644 --- a/server/src/services/skill-installation-maintenance.ts +++ b/server/src/services/skill-installation-maintenance.ts @@ -3,7 +3,7 @@ import path from "node:path"; import { and, asc, eq, gt, sql } from "drizzle-orm"; import { companySkills, companySkillVersions, type Db } from "@paperclipai/db"; import { parseFrontmatterMarkdown, type CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; -import { conflict } from "../errors.js"; +import { HttpError } from "../errors.js"; import { classifyInventoryKind, companySkillService } from "./company-skills.js"; import { managedSkillContent } from "./managed-skill-content.js"; import { skillPackageStore, skillVersionMetadataColumns } from "./skill-package-store.js"; @@ -11,6 +11,7 @@ import { skillPackageMaintenance } from "./skill-package-maintenance.js"; import { skillPackageManifest } from "./skill-package-files.js"; import { assertSkillSnapshotPath, MAX_SKILL_SNAPSHOT_BYTES, MAX_SKILL_SNAPSHOT_FILES, skillSnapshotHash, snapshotFile } from "./skill-snapshot.js"; import { protectedVersions, withSkillVersionCacheLock } from "./skill-version-cache.js"; +import { readPinnedCatalogSkillFiles } from "./skills-catalog.js"; import { getStorageService, type StorageService } from "../storage/index.js"; type Page = { companyId?: string; after?: string; id?: string; limit?: number; signal?: AbortSignal }; @@ -19,6 +20,11 @@ const limit = (options: Page) => Math.min(50, Math.max(1, options.limit ?? 10)); const legacyManaged = (skill: typeof companySkills.$inferSelect) => skill.sourceType === "local_path" && skill.metadata?.sourceKind === "managed_local"; const linkedLocal = (skill: typeof companySkills.$inferSelect) => skill.sourceType === "local_path" && !legacyManaged(skill) && skill.metadata?.sourceKind !== "paperclip_bundled"; +class SkillConversionBlocked extends HttpError { + constructor(readonly code: string, message: string, status = 422) { super(status, message); } +} +const blocked = (code: string, message: string, status?: number) => new SkillConversionBlocked(code, message, status); + function coversInstalledInventory(skill: typeof companySkills.$inferSelect, files: Array<{ path: string }>) { const paths = new Set(files.map(file => file.path)); return paths.has("SKILL.md") && skill.fileInventory.every(file => typeof file.path === "string" && paths.has(file.path)); @@ -32,7 +38,7 @@ async function legacySourceIdentity(skill: typeof companySkills.$inferSelect) { throw error; }); if (!stat) return null; - if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Legacy skill root is not a safe directory"); + if (!stat.isDirectory() || stat.isSymbolicLink()) throw blocked("legacy_source_unsafe", "Legacy skill root is not a safe directory"); return { root: await fs.realpath(root), device: stat.dev, inode: stat.ino }; } @@ -45,7 +51,7 @@ async function legacyFiles(skill: typeof companySkills.$inferSelect, signal?: Ab throw error; }); if (!stat) return null; - if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Legacy skill root is not a safe directory"); + if (!stat.isDirectory() || stat.isSymbolicLink()) throw blocked("legacy_source_unsafe", "Legacy skill root is not a safe directory"); const files: CompanySkillVersionFileInventoryEntry[] = []; let bytes = 0; async function visit(directory: string) { @@ -53,35 +59,47 @@ async function legacyFiles(skill: typeof companySkills.$inferSelect, signal?: Ab signal?.throwIfAborted(); if (entry.name === ".git" || entry.name === "node_modules") continue; const absolute = path.join(directory, entry.name); - if (entry.isSymbolicLink()) throw new Error("Legacy skill contains a symbolic link"); + if (entry.isSymbolicLink()) throw blocked("legacy_source_unsafe", "Legacy skill contains a symbolic link"); if (entry.isDirectory()) { await visit(absolute); continue; } - if (!entry.isFile()) throw new Error("Legacy skill contains an unsupported file"); + if (!entry.isFile()) throw blocked("legacy_source_unsafe", "Legacy skill contains an unsupported file"); const relative = assertSkillSnapshotPath(path.relative(root, absolute).split(path.sep).join("/")); const handle = await fs.open(absolute, constants.O_RDONLY | constants.O_NOFOLLOW); try { const stat = await handle.stat(); - if (!stat.isFile() || stat.nlink !== 1) throw new Error("Legacy skill contains an unsupported file"); + if (!stat.isFile() || stat.nlink !== 1) throw blocked("legacy_source_unsafe", "Legacy skill contains an unsupported file"); bytes += stat.size; - if (files.length >= MAX_SKILL_SNAPSHOT_FILES || bytes > MAX_SKILL_SNAPSHOT_BYTES) throw new Error("Legacy skill exceeds package limits"); + if (files.length >= MAX_SKILL_SNAPSHOT_FILES || bytes > MAX_SKILL_SNAPSHOT_BYTES) throw blocked("legacy_source_too_large", "Legacy skill exceeds package limits"); const body = Buffer.alloc(stat.size); let offset = 0; while (offset < body.length) { signal?.throwIfAborted(); const { bytesRead } = await handle.read(body, offset, body.length - offset, offset); - if (!bytesRead) throw new Error("Legacy skill changed during conversion"); + if (!bytesRead) throw blocked("source_changed", "Legacy skill changed during conversion"); offset += bytesRead; } const after = await handle.stat(); - if (after.size !== stat.size || after.mtimeMs !== stat.mtimeMs) throw new Error("Legacy skill changed during conversion"); + if (after.size !== stat.size || after.mtimeMs !== stat.mtimeMs) throw blocked("source_changed", "Legacy skill changed during conversion"); files.push(snapshotFile(relative, classifyInventoryKind(relative), body, Boolean(stat.mode & 0o111))); } finally { await handle.close(); } } } await visit(root); - if (!files.some(file => file.path === "SKILL.md" && file.encoding !== "base64")) throw new Error("Legacy skill is missing UTF-8 SKILL.md"); + if (!files.some(file => file.path === "SKILL.md" && file.encoding !== "base64")) throw blocked("legacy_skill_markdown_missing", "Legacy skill is missing UTF-8 SKILL.md"); return files; } +async function pinnedCatalogFiles(skill: typeof companySkills.$inferSelect, signal?: AbortSignal) { + const catalogId = skill.metadata?.catalogId, originHash = skill.metadata?.originHash; + if (skill.sourceType !== "catalog" || typeof catalogId !== "string" || typeof originHash !== "string") return null; + let files: Awaited>; + try { files = await readPinnedCatalogSkillFiles(catalogId, originHash, signal); } + catch (error) { + if (signal?.aborted) throw error; + throw blocked("catalog_package_unavailable", error instanceof Error ? error.message : "Catalog package is unavailable"); + } + return files?.map(file => snapshotFile(assertSkillSnapshotPath(file.path), classifyInventoryKind(file.path), file.bytes)) ?? null; +} + export function skillInstallationMaintenance(db: Db, storage?: StorageService) { const store = skillPackageStore(db, storage); const content = managedSkillContent(db, db, storage); @@ -93,7 +111,7 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) { } async function installations(options: Conversion = {}) { const skills = await db.select().from(companySkills).where(scope(options)).orderBy(asc(companySkills.id)).limit(limit(options)); - const results: Array<{ skillId: string; companyId: string; status: string; versionId?: string | null; reason?: string }> = []; + const results: Array<{ skillId: string; companyId: string; status: string; versionId?: string | null; reason?: string; code?: string }> = []; for (const skill of skills) { options.signal?.throwIfAborted(); const catalogOrigin = skill.sourceType === "catalog" && !skill.metadata?.originVersionId; @@ -115,18 +133,25 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) { const sourceIdentity = await legacySourceIdentity(skill); const local = await legacyFiles(skill, options.signal); const completeSaved = saved && coversInstalledInventory(skill, saved) ? saved : null; + const unmodifiedCatalog = skill.sourceType === "catalog" && !skill.metadata?.userModifiedAt + && typeof skill.metadata?.originHash === "string" && skill.metadata.installedHash === skill.metadata.originHash; const files = local ?? completeSaved ?? ((skill.sourceType === "github" || skill.sourceType === "skills_sh" || skill.sourceType === "url") - ? await companySkillService(db).captureLegacyInstallation(skill.companyId, skill.id) : null); + ? await companySkillService(db).captureLegacyInstallation(skill.companyId, skill.id).catch(error => { + if (options.signal?.aborted) throw error; + throw blocked("remote_source_unavailable", error instanceof Error ? error.message : "Remote source is unavailable"); + }) + : unmodifiedCatalog ? await pinnedCatalogFiles(skill, options.signal) : null); const originSource = catalogOrigin && typeof skill.metadata?.originSnapshotLocator === "string" ? { ...skill, sourceLocator: skill.metadata.originSnapshotLocator } : null; const originIdentity = originSource ? await legacySourceIdentity(originSource) : null; - const origin = originSource ? await legacyFiles(originSource, options.signal) : null; + const localOrigin = originSource ? await legacyFiles(originSource, options.signal) : null; + const origin = localOrigin ?? (catalogOrigin && skill.metadata?.userModifiedAt ? await pinnedCatalogFiles(skill, options.signal) : null); if (catalogOrigin && !origin && skill.metadata?.userModifiedAt) { - throw new Error("Catalog reset snapshot is unavailable; preserve the original installed package before conversion"); + throw blocked("catalog_origin_unavailable", "Catalog reset snapshot is unavailable; preserve the original installed package before conversion"); } - if (!files?.length || !files.some(file => file.path === "SKILL.md")) throw new Error("Neither complete legacy files nor a saved version is available"); + if (!files?.length || !files.some(file => file.path === "SKILL.md")) throw blocked("source_files_unavailable", "Neither complete legacy files nor a saved version is available"); if (!local && !coversInstalledInventory(skill, files)) { - throw new Error("Saved version does not cover the installed inventory"); + throw blocked("saved_version_incomplete", "Saved version does not cover the installed inventory"); } if (!options.apply) { return { skillId: skill.id, companyId: skill.companyId, status: "conversion_required", versionId: skill.currentVersionId }; @@ -137,12 +162,12 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) { } const candidate = await content.prepare(skill.companyId, files, options.signal); if (!candidate.packageSnapshotId || !await store.verifyDurable(skill.companyId, candidate.packageSnapshotId, options.signal)) { - throw new Error("Final snapshot is not independently recoverable from object storage"); + throw blocked("snapshot_not_durable", "Final snapshot is not independently recoverable from object storage"); } let originVersionId = typeof skill.metadata?.originVersionId === "string" ? skill.metadata.originVersionId : null; const originCandidate = catalogOrigin && origin && skillSnapshotHash(origin) !== skillSnapshotHash(files) ? await content.prepare(skill.companyId, origin, options.signal) : null; - if (originCandidate && !originCandidate.packageSnapshotId) throw new Error("Catalog origin is not durably available"); + if (originCandidate && !originCandidate.packageSnapshotId) throw blocked("catalog_origin_not_durable", "Catalog origin is not durably available"); const markdown = files.find(file => file.path === "SKILL.md")!.content; const parsed = parseFrontmatterMarkdown(markdown); const { missingSource: _missing, ...metadata } = skill.metadata ?? {}; @@ -167,14 +192,14 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) { if (!current || current.slug !== skill.slug || current.sourceLocator !== skill.sourceLocator || current.sourceRef !== skill.sourceRef || JSON.stringify(current.fileInventory) !== JSON.stringify(skill.fileInventory) || JSON.stringify(await legacySourceIdentity(current)) !== JSON.stringify(sourceIdentity)) - throw conflict("Legacy skill source changed during conversion. Retry the operation."); + throw blocked("source_changed", "Legacy skill source changed during conversion. Retry the operation.", 409); const latest = await legacyFiles(current, options.signal); if ((latest ? skillSnapshotHash(latest) : null) !== (local ? skillSnapshotHash(local) : null)) - throw conflict("Legacy skill files changed during conversion. Retry the operation."); + throw blocked("source_changed", "Legacy skill files changed during conversion. Retry the operation.", 409); if (originSource && (JSON.stringify(await legacySourceIdentity(originSource)) !== JSON.stringify(originIdentity) || (await legacyFiles(originSource, options.signal).then(value => value ? skillSnapshotHash(value) : null)) - !== (origin ? skillSnapshotHash(origin) : null))) - throw conflict("Catalog origin changed during conversion. Retry the operation."); + !== (localOrigin ? skillSnapshotHash(localOrigin) : null))) + throw blocked("source_changed", "Catalog origin changed during conversion. Retry the operation.", 409); if (origin && originCandidate?.packageSnapshotId) { originVersionId = (await companySkillService(db).createVersion(skill.companyId, skill.id, { label: "Original catalog package" }, { type: "system" }, { @@ -192,7 +217,10 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) { }; results.push(await convert()); } catch (error) { - results.push({ skillId: skill.id, companyId: skill.companyId, status: "blocked", reason: error instanceof Error ? error.message : "Conversion failed" }); + if (options.signal?.aborted) throw options.signal.reason; + const reason = error instanceof Error ? error.message : "Conversion failed"; + results.push({ skillId: skill.id, companyId: skill.companyId, status: "blocked", reason, + code: error instanceof SkillConversionBlocked ? error.code : "blocked" }); } } return { inspected: skills.length, after: skills.at(-1)?.id ?? null, results }; diff --git a/server/src/services/skills-catalog.ts b/server/src/services/skills-catalog.ts index ffe6a798686..1fd7a7de1d7 100644 --- a/server/src/services/skills-catalog.ts +++ b/server/src/services/skills-catalog.ts @@ -202,6 +202,7 @@ function resolveCatalogSourcePath(source: CatalogSkillSource, relativePath: stri async function fetchCatalogSourceFile( skill: CatalogSkill, relativePath: string, + signal?: AbortSignal, ): Promise { const source = skill.source; if (!source) { @@ -220,7 +221,7 @@ async function fetchCatalogSourceFile( const sourcePath = resolveCatalogSourcePath(source, relativePath); const url = resolveRawGitHubUrl(source.hostname, source.owner, source.repo, source.commit, sourcePath); - const response = await ghFetch(url); + const response = await ghFetch(url, signal ? { signal } : undefined); if (!response.ok) { throw unprocessable(`Failed to fetch pinned catalog file ${sourcePath} from ${source.owner}/${source.repo}@${source.commit}: HTTP ${response.status}`); } @@ -230,13 +231,14 @@ async function fetchCatalogSourceFile( async function readCatalogFileBytes( skill: CatalogSkill, relativePath: string, + signal?: AbortSignal, ): Promise { const fileEntry = skill.files.find((entry) => entry.path === relativePath); if (!fileEntry) { throw notFound("Catalog skill file not found"); } - const bytes = await fetchCatalogSourceFile(skill, relativePath); + const bytes = await fetchCatalogSourceFile(skill, relativePath, signal); const actualSha = createHash("sha256").update(bytes).digest("hex"); if (actualSha !== fileEntry.sha256) { throw unprocessable(`Pinned catalog file hash mismatch for ${skill.id}:${relativePath}.`); @@ -346,6 +348,23 @@ export async function copyCatalogSkillFile(reference: string, relativePath: stri await fs.writeFile(targetPath, await readCatalogFileBytes(skill, normalizedPath)); } +export async function readPinnedCatalogSkillFiles(reference: string, contentHash: string, signal?: AbortSignal) { + let skill: CatalogSkill | null; + try { skill = resolveCatalogSkillReference(reference).skill; } + catch (error) { + if (isCatalogManifestUnavailableError(error)) return null; + throw error; + } + if (!skill || skill.contentHash !== contentHash) return null; + const pinned = skill; + const files: Array<{ path: string; bytes: Buffer }> = []; + for (const entry of pinned.files) { + signal?.throwIfAborted(); + files.push({ path: entry.path, bytes: await readCatalogFileBytes(pinned, entry.path, signal) }); + } + return files; +} + export function getCatalogPackageMetadata() { const catalogManifest = getCatalogManifest(); return {