From 1b1861c29d13427bac196932c19cb683c930e205 Mon Sep 17 00:00:00 2001 From: Devin Foley Date: Sat, 21 Nov 2026 00:25:10 -0800 Subject: [PATCH] Fix skill upload conflicts blocking first-agent onboarding (#22477) ## Thinking Path > - Paperclip manages agents and the skills they need to work. > - First-agent setup resolves the bundled skills before onboarding can finish. > - Skill packages use durable upload claims to protect their object-storage receipts. > - A failed attempt kept its claim active for ten minutes. A concurrent caller received an immediate conflict. > - This pull request expires a failed owner's unpublished claim and waits for an active upload to settle. > - Onboarding can retry skill setup without a false upload-in-progress error. ## Linked Issues or Issue Description **What happened?** First-agent onboarding could show "This skill package is being saved. Retry after its upload settles." during the subscription connection step. The agent's bundled skill setup called the package store. A failed upload retained its active claim, and calls from another database client also returned a conflict while an upload was in progress. **Expected behavior** A failed attempt permits an immediate retry with new object keys. Concurrent callers reuse the completed receipt or claim the failed attempt. They preserve another owner's active lease and every published receipt. **Steps to reproduce** 1. Fail the first object upload while resolving the first agent's bundled skills. 2. Retry the same skill selection before the ten-minute lease expires. 3. Observe the upload-in-progress conflict. The new regression reproduces this through `companySkillService.resolveRequestedSkillEntries`. **Paperclip version or commit** Reproduced on `6bf312270c3e`. **Deployment mode** Hosted onboarding; the package-store fix also applies to self-hosted instances. Related implementation: #18082. Searched open and closed pull requests and open issues for related skill-upload and lease failures. No duplicate fix was found. ## What Changed - Expire only the failed attempt's token-matched, unpublished upload claim. Keep its exact keys for the existing deletion outbox. Cleanup has a one-second client deadline and a transaction-local statement timeout. Queued cleanup skips its write after that deadline; failed cleanup retains the original crash lease. - Wait for a competing package preparation within the existing three-minute deadline. Re-read the receipt every 250 ms while this concrete request remains active. Release the database connection between reads and stop on cancellation. This adds no idle worker or permanent polling. - Preserve cancellation reasons, published receipts after lost commit replies, and claims acquired by a newer owner. - Add real PostgreSQL regression coverage for first-agent bundled skill setup, cross-client contention, failed-owner takeover, crash expiry, cancellation, ambiguous publication, and ownership changes. - Document upload recovery behavior. No schema or public API change is required. ## Verification - Red: four new or updated checks failed against the original implementation. They reproduced the exact conflict and the retained ten-minute lease. - Green: 189 targeted tests passed across `skill-package-upload.test.ts`, `skill-package-store.test.ts`, `skill-package-work.test.ts`, both package-storage migration suites, and `agent-skills-routes.test.ts`. The combined run passed 188 tests. The final SQL-lock regression passed separately. - Command: `pnpm exec vitest run server/src/__tests__/skill-package-upload.test.ts server/src/__tests__/skill-package-store.test.ts server/src/services/skill-package-work.test.ts server/src/services/skill-package-storage-migration.test.ts server/src/services/skill-package-storage-migration-postgres.test.ts server/src/__tests__/agent-skills-routes.test.ts`. - `pnpm --filter @paperclipai/server typecheck` passed, including source and test layers. - The cancellation test first exposed Node's timer wrapping the caller's abort reason. The final implementation preserves that reason, and the full targeted run passed. - Greptile identified an unbounded cleanup write. Two new real-PostgreSQL tests reproduced delayed errors and cancellations behind an occupied one-connection pool. They now pass and prove that late cleanup skips its write. A further row-lock test proves that the PostgreSQL statement timeout frees the pool while another transaction still holds the package row. - Local setup initially lacked macOS embedded-Postgres library links. Repaired ignored dependency links before the red and green database tests. The initial server typecheck also lacked SDK links for excluded plugin packages; ran the maintained `node scripts/link-plugin-dev-sdk.mjs` helper before repeating it. - `pnpm verify --explain` returned exit 2 and requested broader CI coverage. No full local suite was substituted for that CI requirement. - `git diff --check` passed. Reviewed the full diff for secrets, private URLs, personal data, generated files, and run artifacts. - Final revision: `4b4814ac01522599901a3dc29f650a1b33c0b4e2`. [CI run](https://github.com/paperclipai/paperclip/actions/runs/44880245673) passed all applicable checks, including typecheck, build, all general and serialized server test shards, all e2e shards, Runner verification, and the final verify gate. [Greptile review](https://github.com/paperclipai/paperclip/pull/22477#issuecomment-7059199437) scored this revision 5/5 with no unresolved threads. GitHub reports the branch as mergeable and clean. ## Risks Contended preparations now wait within their existing request deadline. A crashed process or an unavailable cleanup database still uses the ten-minute lease expiry. Genuine storage failures remain errors; this change does not hide an unavailable backend. The existing object deletion grace remains intact. Live provider authorization and a deployed Cloud onboarding journey were not run. Tenant release and deployment are separate from this source fix. No production migration is needed. ## Model Used OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through the Codex harness, with xhigh reasoning effort and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/skill-package-storage.md | 11 +- .../src/__tests__/skill-package-store.test.ts | 5 +- .../__tests__/skill-package-upload.test.ts | 271 ++++++++++++++++++ server/src/services/skill-package-store.ts | 37 ++- 4 files changed, 313 insertions(+), 11 deletions(-) create mode 100644 server/src/__tests__/skill-package-upload.test.ts diff --git a/doc/skill-package-storage.md b/doc/skill-package-storage.md index 68dbe427614..958c82d8ef2 100644 --- a/doc/skill-package-storage.md +++ b/doc/skill-package-storage.md @@ -189,8 +189,15 @@ package connection to avoid a one-connection pool deadlock. No upload happens on unchanged warm preparation path. Uploads use unique attempt keys. An incomplete attempt has a durable claim with a -ten-minute expiry. Retries after expiry use fresh keys; an ambiguous database result -never triggers deletion of a possibly successful upload. A database trigger records +ten-minute crash expiry. When an attempt fails or is cancelled, it expires only its +own unpublished claim so a retry can use fresh keys immediately. This cleanup has +a one-second client deadline and a transaction-local statement timeout. Cleanup +queued behind a busy pool skips its write after that deadline. If the database is +unavailable, the original crash expiry remains. Concurrent preparation on another +connection or replica waits for the receipt or an expired claim within its existing +three-minute request deadline. These reads stop when the request settles or is cancelled; +they add no idle background polling. An ambiguous database result never triggers +deletion of a possibly successful upload. A database trigger records retired keys in `skill_package_deletions`, including company cascades. Promotion of an upload to a published receipt does not enqueue its objects for deletion. diff --git a/server/src/__tests__/skill-package-store.test.ts b/server/src/__tests__/skill-package-store.test.ts index 672ece8d9db..cd1bb86f700 100644 --- a/server/src/__tests__/skill-package-store.test.ts +++ b/server/src/__tests__/skill-package-store.test.ts @@ -892,7 +892,7 @@ describe.skipIf(!support.supported)("external skill packages", () => { } finally { clearTimeout(timeout); release(); await Promise.allSettled(pending); } }); - it("retries expired upload claims with fresh keys and only deletes the abandoned attempt", async () => { + it("retries failed upload claims immediately with fresh keys and only deletes the abandoned attempt", async () => { const companyId = await company(); const store = skillPackageStore(db, storage); const put = vi.spyOn(storage, "putFile").mockRejectedValueOnce(new Error("connection lost")); @@ -900,8 +900,7 @@ describe.skipIf(!support.supported)("external skill packages", () => { put.mockRestore(); const [pending] = await db.select().from(snapshots).where(eq(snapshots.companyId, companyId)); expect(pending!.storage).toBeNull(); - await expect(store.prepare(companyId, files)).rejects.toThrow("being saved"); - await db.update(snapshots).set({ upload: { ...pending!.upload!, expiresAt: new Date(0).toISOString() } }).where(eq(snapshots.id, pending!.id)); + expect(new Date(pending!.upload!.expiresAt).getTime()).toBeLessThanOrEqual(Date.now()); const id = await store.prepare(companyId, files); expect(id).toBe(pending!.id); const receipt = (await store.descriptor(companyId, id)).storage!; diff --git a/server/src/__tests__/skill-package-upload.test.ts b/server/src/__tests__/skill-package-upload.test.ts new file mode 100644 index 00000000000..f141515ffb1 --- /dev/null +++ b/server/src/__tests__/skill-package-upload.test.ts @@ -0,0 +1,271 @@ +import { randomUUID } from "node:crypto"; +import { promises as fs } from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { eq, sql } from "drizzle-orm"; +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest"; +import { companies, companySkillRepositorySnapshots as snapshots, createDb, skillPackageDeletions } from "@paperclipai/db"; +import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js"; +import * as storageModule from "../storage/index.js"; +import { createStorageService } from "../storage/service.js"; +import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; +import { companySkillService } from "../services/company-skills.js"; +import { skillPackageStore } from "../services/skill-package-store.js"; +import { skillPackageManifest } from "../services/skill-package-files.js"; + +const support = await getEmbeddedPostgresTestSupport(); +describe.skipIf(!support.supported)("skill package upload recovery", () => { + let database: Awaited>; + let db: ReturnType; + let otherDb: ReturnType; + let home: string; + let storage: storageModule.StorageService; + const previousHome = process.env.PAPERCLIP_HOME; + const files = [{ path: "SKILL.md", kind: "skill" as const, content: "---\nname: recovery\ndescription: Test recovery\n---\nTest instructions\n" }]; + beforeAll(async () => { + database = await startEmbeddedPostgresTestDatabase("skill-package-upload-"); + db = createDb(database.connectionString, { maxConnections: 1 }); + otherDb = createDb(database.connectionString, { maxConnections: 1 }); + home = await fs.mkdtemp(path.join(os.tmpdir(), "skill-package-upload-")); + process.env.PAPERCLIP_HOME = home; + storage = createStorageService(createLocalDiskStorageProvider(path.join(home, "objects"))); + }); + afterEach(() => vi.restoreAllMocks()); + afterAll(async () => { + if (previousHome === undefined) delete process.env.PAPERCLIP_HOME; + else process.env.PAPERCLIP_HOME = previousHome; + await Promise.all([db?.$client.end({ timeout: 1 }), otherDb?.$client.end({ timeout: 1 })]); + await database?.cleanup(); + if (home) await fs.rm(home, { recursive: true, force: true }); + }); + async function company() { + const id = randomUUID(); + await db.insert(companies).values({ id, name: "Upload recovery", issuePrefix: `P${id.slice(0, 6)}` }); + return id; + } + async function pendingUpload(companyId: string, expiresAt: Date) { + const token = randomUUID(); + const [row] = await db.insert(snapshots).values({ companyId, contentHash: skillPackageManifest(files).contentHash, + upload: { token, legacy: false, provider: storage.provider, archiveKey: `${companyId}/${token}/files.tar.gz`, + manifestKey: `${companyId}/${token}/manifest.json`, expiresAt: expiresAt.toISOString() } }).returning(); + return row!; + } + + it.each([false, true])("joins an upload on another database client and recovers a failed owner: %s", async failed => { + const companyId = await company(); + let release!: () => void; + let entered!: () => void; + let contended!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + const uploading = new Promise(resolve => { entered = resolve; }); + const contention = new Promise(resolve => { contended = resolve; }); + const put = storage.putFile.bind(storage); + const puts = vi.spyOn(storage, "putFile").mockImplementationOnce(async input => { + entered(); + await blocked; + if (failed) throw new Error("upload disconnected"); + return put(input); + }); + const debug = otherDb.$client.options.debug; + otherDb.$client.options.debug = (_connection, query) => { + if (query.includes('update "company_skill_repository_snapshots" set "upload"')) contended(); + }; + const owner = skillPackageStore(db, storage).prepare(companyId, files).catch(error => error as Error); + let joined: Promise | undefined; + const cancellation = new AbortController(); + const timeout = setTimeout(() => cancellation.abort(new Error("Upload join did not settle")), 5_000); + try { + await uploading; + joined = skillPackageStore(otherDb, storage).prepare(companyId, files, false, cancellation.signal).catch(error => error as Error); + await contention; + release(); + const [first, second] = await Promise.all([owner, joined]); + expect(second).toEqual(expect.any(String)); + if (failed) expect(first).toEqual(new Error("upload disconnected")); + else expect(second).toBe(first); + expect(puts).toHaveBeenCalledTimes(failed ? 3 : 2); + const rows = await db.select().from(snapshots).where(eq(snapshots.companyId, companyId)); + expect(rows).toHaveLength(1); + expect(rows[0]!.id).toBe(second); + expect(rows[0]!.upload).toBeNull(); + expect(await skillPackageStore(db, storage).verifyDurable(companyId, rows[0]!.id)).toBe(true); + const retired = await db.select().from(skillPackageDeletions).where(eq(skillPackageDeletions.companyId, companyId)); + expect(retired).toHaveLength(failed ? 1 : 0); + if (failed) expect(retired[0]!.archiveKey).not.toBe(rows[0]!.storage!.archiveKey); + } finally { + clearTimeout(timeout); + cancellation.abort(); + release(); + await Promise.allSettled([owner, joined]); + otherDb.$client.options.debug = debug; + } + }); + + it("retries the first agent's bundled skill initialization immediately after a storage failure", async () => { + const companyId = await company(); + vi.spyOn(storageModule, "getStorageService").mockReturnValue(storage); + vi.spyOn(storage, "putFile").mockRejectedValueOnce(new Error("upload disconnected")); + const skills = companySkillService(db); + const requested = ["paperclipai/paperclip/paperclip-create-agent", "paperclipai/paperclip/first-task"]; + await expect(skills.resolveRequestedSkillEntries(companyId, requested)).rejects.toThrow("upload disconnected"); + const result = await skills.resolveRequestedSkillEntries(companyId, requested); + expect(result.unresolved).toEqual([]); + expect(result.resolved.map(entry => entry.key)).toEqual(requested); + }); + + it("reclaims an expired crash lease with fresh keys", async () => { + const companyId = await company(); + const pending = await pendingUpload(companyId, new Date(0)); + const id = await skillPackageStore(db, storage).prepare(companyId, files); + expect(id).toBe(pending.id); + const [saved] = await db.select().from(snapshots).where(eq(snapshots.id, id)); + expect(saved!.upload).toBeNull(); + expect(saved!.storage!.archiveKey).not.toBe(pending.upload!.archiveKey); + const [retired] = await db.select().from(skillPackageDeletions).where(eq(skillPackageDeletions.companyId, companyId)); + expect(retired!.archiveKey).toBe(pending.upload!.archiveKey); + }); + + it("cancels a waiting caller without expiring another owner's active lease", async () => { + const companyId = await company(); + const pending = await pendingUpload(companyId, new Date("2099-01-01")); + const puts = vi.spyOn(storage, "putFile"); + const cancellation = new AbortController(); + let contended!: () => void; + const contention = new Promise(resolve => { contended = resolve; }); + const debug = db.$client.options.debug; + db.$client.options.debug = (_connection, query) => { + if (query.includes('update "company_skill_repository_snapshots" set "upload"')) contended(); + }; + const preparing = skillPackageStore(db, storage).prepare(companyId, files, false, cancellation.signal).catch(error => error); + try { + await contention; + cancellation.abort(new Error("Caller left onboarding")); + expect(await preparing).toEqual(new Error("Caller left onboarding")); + const [saved] = await db.select().from(snapshots).where(eq(snapshots.id, pending.id)); + expect(saved!.upload).toEqual(pending.upload); + expect(puts).not.toHaveBeenCalled(); + } finally { cancellation.abort(); await preparing; db.$client.options.debug = debug; } + }); + + it.each([false, true])("preserves a possibly published receipt after a lost commit reply: committed=%s", async committed => { + const companyId = await company(); + let transactionCount = 0; + const ambiguousDb = new Proxy(db, { + get(target, property, receiver) { + if (property === "transaction") return async (...args: Parameters) => { + const publishing = ++transactionCount === 2; + const result = await target.transaction(async tx => { + const value = await args[0](tx); + if (publishing && !committed) throw new Error("Lost commit reply"); + return value; + }, args[1]); + if (publishing) throw new Error("Lost commit reply"); + return result; + }; + return Reflect.get(target, property, receiver); + }, + }); + const puts = vi.spyOn(storage, "putFile"); + await expect(skillPackageStore(ambiguousDb, storage).prepare(companyId, files)).rejects.toThrow("Lost commit reply"); + const [beforeRetry] = await db.select().from(snapshots).where(eq(snapshots.companyId, companyId)); + expect(Boolean(beforeRetry!.storage)).toBe(committed); + const store = skillPackageStore(db, storage); + expect(await store.prepare(companyId, files)).toBe(beforeRetry!.id); + expect(puts).toHaveBeenCalledTimes(committed ? 2 : 4); + expect(await store.verifyDurable(companyId, beforeRetry!.id)).toBe(true); + const retired = await db.select().from(skillPackageDeletions).where(eq(skillPackageDeletions.companyId, companyId)); + expect(retired).toHaveLength(committed ? 0 : 1); + }); + + it("does not expire a newer owner's claim after the old upload fails", async () => { + const companyId = await company(); + const replacement = { token: randomUUID(), legacy: false, provider: storage.provider, archiveKey: `${companyId}/replacement/files.tar.gz`, + manifestKey: `${companyId}/replacement/manifest.json`, expiresAt: new Date("2099-01-01").toISOString() }; + vi.spyOn(storage, "putFile").mockImplementationOnce(async () => { + await db.update(snapshots).set({ upload: replacement }).where(eq(snapshots.companyId, companyId)); + throw new Error("Old owner disconnected"); + }); + await expect(skillPackageStore(db, storage).prepare(companyId, files)).rejects.toThrow("Old owner disconnected"); + const [saved] = await db.select().from(snapshots).where(eq(snapshots.companyId, companyId)); + expect(saved!.upload).toEqual(replacement); + }); + + it.each([false, true])("bounds failed-upload cleanup behind a busy pool and rejects late writes: cancellation=%s", async cancelled => { + const companyId = await company(); + const failure = new Error(cancelled ? "Upload cancelled" : "Upload failed"); + const cancellation = new AbortController(); + let release!: () => void; + let entered!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + const occupied = new Promise(resolve => { entered = resolve; }); + let holding: Promise | undefined; + vi.spyOn(storage, "putFile").mockImplementationOnce(async () => { + holding = db.transaction(async tx => { + await tx.execute(sql`select 1`); + entered(); + await blocked; + }); + await occupied; + if (cancelled) cancellation.abort(failure); + throw failure; + }); + const preparing = skillPackageStore(db, storage).prepare(companyId, files, false, cancellation.signal).catch(error => error); + let timer: ReturnType | undefined; + let pending: typeof snapshots.$inferSelect | undefined; + try { + await occupied; + [pending] = await otherDb.select().from(snapshots).where(eq(snapshots.companyId, companyId)); + const result = await Promise.race([preparing, new Promise(resolve => { + timer = setTimeout(() => resolve(new Error("Cleanup held the caller behind the pool")), 3_000); + })]); + expect(result).toBe(failure); + } finally { + clearTimeout(timer); + release(); + await holding; + await preparing; + } + await db.execute(sql`select 1`); + const [saved] = await db.select().from(snapshots).where(eq(snapshots.companyId, companyId)); + expect(saved!.upload).toEqual(pending!.upload); + }); + + it("cancels blocked cleanup SQL so the pool remains usable while the row is locked", async () => { + const companyId = await company(); + const failure = new Error("Upload failed before locked cleanup"); + let release!: () => void; + const blocked = new Promise(resolve => { release = resolve; }); + let holding: Promise | undefined; + let pending: typeof snapshots.$inferSelect | undefined; + vi.spyOn(storage, "putFile").mockImplementationOnce(async () => { + let entered!: () => void; + const locked = new Promise(resolve => { entered = resolve; }); + holding = otherDb.transaction(async tx => { + [pending] = await tx.select().from(snapshots).where(eq(snapshots.companyId, companyId)).for("update"); + entered(); + await blocked; + }); + await locked; + throw failure; + }); + let timer: ReturnType | undefined; + const preparing = skillPackageStore(db, storage).prepare(companyId, files).catch(error => error); + try { + const checked = (async () => { + expect(await preparing).toBe(failure); + await db.execute(sql`select 1`); + const [saved] = await db.select().from(snapshots).where(eq(snapshots.companyId, companyId)); + expect(saved!.upload).toEqual(pending!.upload); + return true; + })(); + expect(await Promise.race([checked, new Promise(resolve => { + timer = setTimeout(() => resolve(false), 3_000); + })])).toBe(true); + } finally { + clearTimeout(timer); + release(); + await holding; + await preparing; + } + }); +}); diff --git a/server/src/services/skill-package-store.ts b/server/src/services/skill-package-store.ts index 627e3ebe2ba..11c14f3765b 100644 --- a/server/src/services/skill-package-store.ts +++ b/server/src/services/skill-package-store.ts @@ -6,6 +6,7 @@ import { randomUUID } from "node:crypto"; import { constants, createReadStream, createWriteStream, promises as fs } from "node:fs"; import path from "node:path"; import { pipeline } from "node:stream/promises"; +import { setTimeout as delay } from "node:timers/promises"; import { and, eq, getTableColumns, inArray, sql } from "drizzle-orm"; import { companySkillRepositorySnapshots as snapshots, companySkillVersions, companySkillSources, withDedicatedDbConnection, type Db, type SkillPackageFileMapping, type SkillPackageStorageReceipt, type SkillPackageRestoreRecipe } from "@paperclipai/db"; import type { CompanySkillVersionFileInventoryEntry } from "@paperclipai/shared"; @@ -446,6 +447,24 @@ export function skillPackageStore(db: Db, suppliedStorage?: StorageService, cach return result; }); } + async function expireFailedUpload(companyId: string, id: string, token: string) { + const deadline = performance.now() + 1_000; + let timer: ReturnType | undefined; + try { + await Promise.race([ + db.transaction(async tx => { + if (performance.now() >= deadline) return; + const remainingMs = Math.max(1, Math.floor(deadline - performance.now())); + await tx.execute(sql`select set_config('statement_timeout', ${`${remainingMs}ms`}, true)`); + if (performance.now() >= deadline) return; + await tx.update(snapshots).set({ upload: sql`jsonb_set(${snapshots.upload}, '{expiresAt}', to_jsonb(now()))` }) + .where(and(scope(companyId, id), sql`${snapshots.storage} is null`, sql`${snapshots.upload}->>'token' = ${token}`)); + }), + new Promise(resolve => { timer = setTimeout(resolve, 1_000); }), + ]); + } catch { + } finally { clearTimeout(timer); } + } async function prepare(companyId: string, files: CompanySkillVersionFileInventoryEntry[], migrateLegacy = false, cancellation?: AbortSignal, restore?: SkillPackageRestoreRecipe): Promise { cancellation?.throwIfAborted(); const manifest = skillPackageManifest(files); @@ -461,7 +480,8 @@ export function skillPackageStore(db: Db, suppliedStorage?: StorageService, cach migrationAt: null, maintenanceAt: new Date(Date.now() + 24 * 60 * 60_000) }).onConflictDoNothing().returning(descriptorColumns); }); let existing = created ?? (await db.select(descriptorColumns).from(snapshots).where(and(eq(snapshots.companyId, companyId), eq(snapshots.contentHash, manifest.contentHash))))[0]!; - if (!created) { + while (!created) { + signal.throwIfAborted(); if (existing.storage || (!existing.upload && !migrateLegacy && !existing.restorable)) { // Content deduplication must not attach a recipe with different kind or // encoding metadata to an already published manifest. @@ -476,8 +496,12 @@ export function skillPackageStore(db: Db, suppliedStorage?: StorageService, cach const [claimed] = await db.update(snapshots).set({ upload }).where(and(scope(companyId, existing.id), sql`${snapshots.storage} is null`, sql`(${snapshots.upload} is null or (${snapshots.upload}->>'expiresAt')::timestamptz <= now())`)).returning(descriptorColumns); - if (!claimed) throw conflict("This skill package is being saved. Retry after its upload settles."); - existing = claimed; + if (claimed) { existing = claimed; break; } + try { await delay(250, undefined, { signal }); } + catch (error) { signal.throwIfAborted(); throw error; } + const [current] = await db.select(descriptorColumns).from(snapshots).where(scope(companyId, existing.id)); + if (!current) throw notFound("Skill package was removed during preparation"); + existing = current; } return withPackageIo(async () => { const root = skillPackagePaths(companyId, existing.id).parent; @@ -517,11 +541,12 @@ export function skillPackageStore(db: Db, suppliedStorage?: StorageService, cach if (!saved) throw conflict("Skill package upload ownership changed"); return saved.id; } finally { - // Failed/ambiguous uploads retain their durable claim. Maintenance reconciles - // exact owned keys; never delete a successful object on an uncertain SQL result. await fs.rm(staging, { recursive: true, force: true }); } - }, signal); + }, signal).catch(async error => { + await expireFailedUpload(companyId, existing.id, token); + throw error; + }); }); { const [retained] = await db.update(snapshots).set({ maintenanceAt: new Date(Date.now() + 24 * 60 * 60_000),