fix(storage): preserve agent and skill files during online conversion (#19656)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent files and installed skills need durable storage before a
legacy volume can be removed.
> - Their maintenance commands currently require every writer to stop.
> - Normal file edits already use agent row locks and skill publication
locks.
> - Shared agent heads also remain durable while their agents run.
> - This pull request uses those owners to copy settled files online and
keeps unresolved recovery states intact.
> - The benefit is useful migration progress while unrelated work
continues.

## Linked Issues or Issue Description

Related: #19642.

**What happened?**

Agent-file adoption and skill installation conversion reject apply
requests without a global stopped-writer acknowledgement. Agent-file
maintenance also reports an active agent as blocked when its shared head
is already durable. Live enrollment omits the existing
unresolved-local-instruction-copy guard.

**Expected behavior**

Copy settled agent files and skill installations under their existing
writer locks. Keep old skill directories, versions, and inline bytes.
Keep unresolved local copies and unknown historical pin obligations
visible to their owners.

**Steps to reproduce**

Run agent-file maintenance with `--apply` on the original host with
durable storage enabled. A managed agent without active execution is
rejected without `--writers-stopped`. The same gate rejects skill
installation conversion. An agent that already has a shared head is also
reported as blocked while it runs.

## What Changed

- Adopt settled agent files under the existing agent row lock. Recognize
an existing shared head before checking active execution, without
rewriting its projection.
- Reuse the local working-copy preservation predicate for first adoption
and live enrollment. Keep its original public export.
- Prepare skill uploads outside the shared run-pinning lock. Verify the
exact local tree and source identity under the cache, slug, and row
publication locks. Publish catalog origin history in that same
transaction.
- Retain old skill directories, versions, and inline bytes. Keep pin
retention and volume-removal guards unchanged.
- Add PostgreSQL and filesystem coverage for active durable heads,
unresolved copies, concurrent source edits, and linked files. Update the
maintenance docs.

## Verification

- Agent-file durability and live enrollment suites: 16 tests passed with
real PostgreSQL and filesystem fixtures.
- Managed skill suite: 22 tests passed. Seven final controls also
passed: upload-time file edits, source relocation, unknown historical
pins, hard-linked files, pinning during a blocked upload, and both
rejected/successful catalog-origin publication.
- Server source typecheck and focused changed-test typecheck passed.
- Module boundaries, diff whitespace, and added-line
credential/internal-reference scans passed.
- Baseline production rejected both online paths at their stopped-writer
guards. Review regressions also reproduced a busy pinning lock during
upload and extra catalog history after rejection; their fixes passed
real PostgreSQL controls. All expected red results are preserved.
- One final-control attempt failed the existing 30-second fixture
startup hook before executing tests. A sequential retry passed all four
controls without changing guards or timeouts; the failed attempt remains
recorded.
- Exact head `001a5beed9d0e55c7c178906fbad1f9a47f87788`: [required CI
passed](https://github.com/paperclipai/paperclip/actions/runs/42309446426),
including typecheck, build, tests, Runner and end-to-end gates. Greptile
completed successfully at 5/5 with zero unresolved review threads;
GitHub reports mergeable/clean.

## Risks

Changed or unsafe sources remain blocked. Storage failures keep the
original files and recovery state. This conversion does not settle
retained provider copies or historical execution. Unknown pin
obligations still block retention and volume removal; this PR alone does
not qualify removal of the original volume.

## Model Used

OpenAI GPT-6 (Codex). The runtime did not expose a more specific backend
model ID or context-window size. Used reasoning, code editing, and tool
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-11-06 17:31:04 -06:00
1 parent bee0f0e2fa
commit bb621880e6
10 files changed
+356 -97

No files matched your search

+8 -5
View File
@@ -339,12 +339,12 @@ unit or matcher results; a passing matcher does not prove a live run.
Run `server/src/maintenance/agent-files.ts` with the instance's database,
storage, secrets and original `PAPERCLIP_HOME`. The default migration phase is
read-only. Before applying, stop ingress and execution writers on that instance
and set `PAPERCLIP_DURABLE_AGENT_FILES=true`:
read-only. Set `PAPERCLIP_DURABLE_AGENT_FILES=true` before applying on the original
host. Migration uses the same agent row lock as file edits and admission:
```sh
pnpm exec tsx server/src/maintenance/agent-files.ts --phase=migration
pnpm exec tsx server/src/maintenance/agent-files.ts --phase=migration --apply --writers-stopped
pnpm exec tsx server/src/maintenance/agent-files.ts --phase=migration --apply
```
The production runtime build includes the same command without a TypeScript
@@ -355,8 +355,11 @@ Eligibility is a point-in-time check; keep writers stopped through volume remova
Follow each returned `next` cursor with `--after=<cursor>`. All pages must report
`pageReady: true` before removing the volume. A missing legacy directory, external
instructions or active execution prevents migration; resolve that condition on
the original host. Repeating apply verifies/restores the committed projection.
instructions, active execution without a durable head, or an unsettled private
local working copy prevents first adoption. An existing durable head needs no
local rewrite while its agent is active. Migration keeps copy recovery states
and retained provider references. The live eligibility gate separately refuses
unsettled local instruction copies, even after canonical agent files are durable.
This migrates agent files only; other storage families have their own eligibility
checks. Keep the original volume until the deployment migration is qualified.
+11 -8
View File
@@ -90,15 +90,17 @@ use the latest branch as a replacement for missing installed content.
`versions` phases. The latter also fills missing version metadata indexes so
subsequent library/history reads need no object downloads. Keep original bytes
until independent verification succeeds.
3. Stop admission and skill/source writers; retire every filesystem-authoritative
process. The `--writers-stopped` flag attests this operator action; it does not
stop the application itself.
4. Apply final installation conversion on the host with the legacy files:
3. Copy installations on the host with the legacy files. This phase shares the
cache, publication and row locks with normal writers. It checks the exact
local source again before publication:
```sh
pnpm --filter @paperclipai/server exec tsx scripts/migrate-skill-packages.ts --phase=installations --company=<uuid> --limit=10 --apply --writers-stopped
pnpm --filter @paperclipai/server exec tsx scripts/migrate-skill-packages.ts --phase=installations --company=<uuid> --limit=10 --apply
```
4. Keep all original directories, versions and inline bytes. Conversion does not
settle old execution, recovery or pin uncertainty. Preserve or settle every
remaining local dependency through its owner before removing the volume.
5. Sweep `eligibility` and `verification`, including all historical versions. The
latter verifies object bytes independently of caches. Follow all cursors; a
successful first batch is not a complete readiness report.
@@ -109,9 +111,10 @@ Managed conversion preserves skill IDs, keys, history, assignments, pins and acc
provenance. Identical complete snapshots reuse their version IDs. Catalog conversion
preserves the original reset version as well as installed content. A missing legacy
directory can use a complete verified saved version. A partial Markdown-only rescue,
unknown revision, unsafe source tree or active legacy selection without exact version
identity is a blocker. Settle or snapshot such runs while A is still available; do
not cancel them simply to pass preflight.
unknown revision or unsafe source tree blocks conversion. An active legacy selection
without exact version identity remains a retention and volume-removal blocker even
after copy-only conversion. Preserve or settle such runs while A is still available;
do not cancel them simply to pass preflight.
Report statuses distinguish linked-local sources, legacy managed/catalog records,
missing current versions, unknown active-run selections, missing/corrupt packages,
@@ -6,7 +6,7 @@ import { Readable } from "node:stream";
import { setTimeout as delay } from "node:timers/promises";
import { beforeAll, beforeEach, afterEach, afterAll, describe, expect, it, vi } from "vitest";
import { eq } from "drizzle-orm";
import { agents, agentFileRoots, authUsers, companies, companyMemberships, principalPermissionGrants, heartbeatRuns, createDb, closeFileStorageDb } from "@paperclipai/db";
import { agents, agentFileRoots, agentInstructionWorkingCopies, authUsers, companies, companyMemberships, principalPermissionGrants, heartbeatRuns, createDb, closeFileStorageDb } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { agentFileStore, adoptAgentFiles, fileHash } from "../services/agent-file-store.js";
import { invalidateAgentFileProjection, persistDurableAgentFiles } from "../services/agent-file-durability.js";
@@ -193,17 +193,42 @@ describe("durable agent file authority", () => {
expect(await store().read(companyId, agentId, "failed.txt", actor())).toBeNull();
});
it("dry-runs migration without adopting files and requires stopped writers for publication", async () => {
it("dry-runs without adopting files and publishes settled legacy files online", async () => {
await materializeInstructionBytes(root, "AGENTS.md", Buffer.from("Legacy"));
const input = { companyId, apply: false, writersStopped: false };
expect(await migrateAgentFiles(db, input)).toMatchObject({ results: [{ agentId, state: "needs_import" }], pageReady: false });
expect(await db.select().from(agentFileRoots).where(eq(agentFileRoots.agentId, agentId))).toEqual([]);
await expect(migrateAgentFiles(db, { ...input, apply: true })).rejects.toThrow("stopped writers");
expect(await migrateAgentFiles(db, { ...input, apply: true, writersStopped: true })).toMatchObject({ pageReady: true });
expect(await migrateAgentFiles(db, { ...input, apply: true })).toMatchObject({ pageReady: true });
await replaceFilesystem();
expect(await store().read(companyId, agentId, "AGENTS.md", actor())).toEqual(Buffer.from("Legacy"));
});
it("recognizes an active agent's existing durable head without restoring its filesystem projection", async () => {
await store().write({ companyId, agentId, path: "AGENTS.md", bytes: Buffer.from("Durable"), baseHash: null }, actor());
const [active] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running", runtimeMode: "native" }).returning();
const before = await db.select().from(agentFileRoots).where(eq(agentFileRoots.agentId, agentId));
await fs.rm(root, { recursive: true });
expect(await migrateAgentFiles(db, { companyId, apply: true, writersStopped: false }))
.toMatchObject({ results: [{ agentId, state: "durable" }], pageReady: true });
expect(await db.select().from(agentFileRoots).where(eq(agentFileRoots.agentId, agentId))).toEqual(before);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, active!.id)))[0]).toEqual(active);
await expect(fs.stat(root)).rejects.toMatchObject({ code: "ENOENT" });
});
it.each(["pending_collection", "unavailable"])("preserves a %s working copy and keeps first adoption blocked", async state => {
await materializeInstructionBytes(root, "AGENTS.md", Buffer.from("Canonical files"));
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", finishedAt: new Date() }).returning();
const [copy] = await db.insert(agentInstructionWorkingCopies).values({ companyId, agentId, runId: run!.id,
responsibleUserId: userId, entryFile: "AGENTS.md", baseHash: "a".repeat(64), localRoot: path.join(home, "retained-copy"),
executionRoot: "/retained/agent-files", location: "local", state,
errorCode: "INSTRUCTION_COLLECTION_UNAVAILABLE", receipt: { schema: "paperclip.agent-files.v1" } }).returning();
expect(await migrateAgentFiles(db, { companyId, apply: true, writersStopped: false }))
.toMatchObject({ results: [{ agentId, state: "working_copy_unsettled" }], pageReady: false });
expect(await db.select().from(agentFileRoots).where(eq(agentFileRoots.agentId, agentId))).toEqual([]);
expect((await db.select().from(agentInstructionWorkingCopies).where(eq(agentInstructionWorkingCopies.runId, run!.id)))[0]).toEqual(copy);
expect(await fs.readFile(path.join(root, "AGENTS.md"), "utf8")).toBe("Canonical files");
});
it("refuses migration while the agent has queued or running execution", async () => {
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "running" }).returning();
for (const status of ["running", "queued"]) {
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
import { beforeAll, afterAll, afterEach, expect, it, vi } from "vitest";
import { eq } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { agentFileRoots, agents, companies, companySkills, companySkillVersions, executionWorkspaces, heartbeatRuns, providerTraceRecords, createDb } from "@paperclipai/db";
import { agentFileRoots, agentInstructionWorkingCopies, agents, companies, companySkills, companySkillVersions, executionWorkspaces, heartbeatRuns, providerTraceRecords, createDb } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { assertEphemeralServerData, readEphemeralServerDataEligibility } from "../services/ephemeral-server-contract.js";
import { initializeDurableAgentFiles } from "../services/agent-file-durability.js";
@@ -66,3 +66,22 @@ it("keeps replacement readiness independent of a serving controller's unfinished
await db.update(companySkills).set({ currentVersionId: null }).where(eq(companySkills.id, skillId));
await expect(assertEphemeralServerData(db)).rejects.toThrow("skills_require_migration");
});
it("keeps an unsettled local instruction copy visible after its canonical files become durable", async () => {
vi.stubEnv("PAPERCLIP_DURABLE_AGENT_FILES", "true");
const companyId = randomUUID(), agentId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Private copy preservation", issuePrefix: "COPY" });
await db.insert(agents).values({ id: agentId, companyId, name: "File owner" });
await db.transaction(tx => initializeDurableAgentFiles(tx as unknown as Db, { id: agentId, companyId }));
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId, status: "failed", finishedAt: new Date() }).returning();
const [copy] = await db.insert(agentInstructionWorkingCopies).values({ runId: run.id, companyId, agentId,
responsibleUserId: "preservation-fixture", entryFile: "AGENTS.md", baseHash: "a".repeat(64),
localRoot: "/private/unsaved-copy", executionRoot: "/private/unsaved-copy", location: "local", state: "pending_collection" }).returning();
expect((await readEphemeralServerDataEligibility(db, { purpose: "startup" })).blockers)
.toContain("local_instruction_copies_require_settlement");
expect((await db.select().from(agentInstructionWorkingCopies).where(eq(agentInstructionWorkingCopies.runId, run.id)))[0]).toEqual(copy);
await db.update(agentInstructionWorkingCopies).set({ state: "unavailable", processStoppedAt: new Date(),
receipt: { schema: "paperclip.agent-files.v1", cleanupPending: false } }).where(eq(agentInstructionWorkingCopies.runId, run.id));
expect((await readEphemeralServerDataEligibility(db, { purpose: "startup" })).blockers)
.not.toContain("local_instruction_copies_require_settlement");
});
@@ -1,6 +1,8 @@
import { createServer } from "node:http";
import { createS3StorageProvider } from "../storage/s3-provider.js";
import { skillInstallationMaintenance } from "../services/skill-installation-maintenance.js";
import { withSkillVersionCacheLock } from "../services/skill-version-cache.js";
import { skillPackageStore } from "../services/skill-package-store.js";
import { removeRuntimeAssetTree } from "../services/native-runtime/runtime-asset-retention.js";
import { eq } from "drizzle-orm";
import { writePaperclipSkillSyncPreference } from "@paperclipai/adapter-utils/server-utils";
@@ -384,8 +386,7 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
await db.update(companySkillVersions).set({ packageFileMap: null }).where(eq(companySkillVersions.id, version.id));
const maintenance = skillInstallationMaintenance(db);
expect((await maintenance.installations({ id: legacy!.id })).results[0]!.status).toBe("conversion_required");
await expect(maintenance.installations({ companyId, apply: true })).rejects.toMatchObject({ status: 409 });
expect((await maintenance.installations({ id: legacy!.id, apply: true, writersStopped: true })).results)
expect((await maintenance.installations({ id: legacy!.id, apply: true })).results)
.toEqual([expect.objectContaining({ skillId: legacy!.id, versionId: version.id, status: "converted" })]);
await fs.rm(directory, { recursive: true });
expect(await service.getById(companyId, legacy!.id)).toMatchObject({ sourceType: "managed", sourceLocator: null, currentVersionId: version.id });
@@ -393,6 +394,163 @@ describe.skipIf(!support.supported)("managed skills use committed versions", ()
expect((await maintenance.eligibility({ id: legacy!.id })).results[0]!.status).toBe("version_backed");
});
it("copies an installation online while preserving unknown old pins, old inline bytes, files, and active work", async () => {
const companyId = await company(), directory = path.join(root, `copy-only-${randomUUID()}`);
await fs.mkdir(directory);
await fs.writeFile(path.join(directory, "SKILL.md"), "# Preserved skill");
const [skill] = await db.insert(companySkills).values({ companyId, key: "copy-only", slug: "copy-only", name: "Copy only",
markdown: "# Preserved skill", sourceType: "local_path", sourceLocator: directory,
fileInventory: [{ path: "SKILL.md", kind: "skill" }], metadata: { sourceKind: "managed_local" } }).returning();
const files = [{ path: "SKILL.md", kind: "skill" as const, content: "# Preserved skill" }];
const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill.id,
revisionNumber: 1, fileInventory: files }).returning();
await db.update(companySkills).set({ currentVersionId: version.id }).where(eq(companySkills.id, skill.id));
const [agent] = await db.insert(agents).values({ companyId, name: "Unrelated active worker" }).returning();
const [active] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, runtimeMode: "native",
status: "running", contextSnapshot: {} }).returning();
const maintenance = skillInstallationMaintenance(db);
expect((await maintenance.installations({ id: skill.id, apply: true })).results)
.toEqual([expect.objectContaining({ skillId: skill.id, status: "converted" })]);
expect((await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, version.id)))[0].fileInventory).toEqual(files);
expect(await fs.readFile(path.join(directory, "SKILL.md"), "utf8")).toBe("# Preserved skill");
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, active.id)))[0]).toEqual(active);
// Copy-only publication does not release retention or volume-removal guards.
expect((await maintenance.eligibility({ id: skill.id })).results[0].status).toBe("unsupported_active_run");
});
it.each(["files", "source_locator"])("keeps a concurrent %s change instead of publishing the earlier conversion", async change => {
const companyId = await company(), directory = path.join(root, `conversion-race-${randomUUID()}`);
await fs.mkdir(directory);
await fs.writeFile(path.join(directory, "SKILL.md"), "# Race");
await fs.writeFile(path.join(directory, "support.txt"), "old version");
const [skill] = await db.insert(companySkills).values({ companyId, key: "conversion-race", slug: "conversion-race", name: "Race",
markdown: "# Race", sourceType: "local_path", sourceLocator: directory,
fileInventory: [{ path: "SKILL.md", kind: "skill" }, { path: "support.txt", kind: "other" }],
metadata: { sourceKind: "managed_local" } }).returning();
const version = await companySkillService(db).createVersion(companyId, skill.id);
await fs.writeFile(path.join(directory, "support.txt"), "captured unsaved edit");
const storage = storageModule.getStorageService(), put = storage.putFile.bind(storage);
let changed = false;
const nextDirectory = path.join(root, `new-source-${randomUUID()}`);
const upload = vi.spyOn(storage, "putFile").mockImplementation(async input => {
const saved = await put(input);
if (!changed) {
changed = true;
if (change === "files") await fs.writeFile(path.join(directory, "support.txt"), "newer unsaved edit");
else {
await fs.mkdir(nextDirectory);
await fs.writeFile(path.join(nextDirectory, "SKILL.md"), "# New source");
await db.transaction(async tx => {
await tx.select().from(companySkills).where(eq(companySkills.id, skill.id)).for("update");
await tx.update(companySkills).set({ sourceLocator: nextDirectory }).where(eq(companySkills.id, skill.id));
});
}
}
return saved;
});
try {
const result = await skillInstallationMaintenance(db, storage).installations({ id: skill.id, apply: true });
expect(changed).toBe(true);
expect(result.results).toEqual([expect.objectContaining({ status: "blocked" })]);
const [current] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id));
expect(current).toMatchObject({ sourceType: "local_path", currentVersionId: version.id,
sourceLocator: change === "files" ? directory : nextDirectory });
expect(await fs.readFile(path.join(directory, "support.txt"), "utf8"))
.toBe(change === "files" ? "newer unsaved edit" : "captured unsaved edit");
} finally { upload.mockRestore(); }
});
it("holds a hard-linked local skill instead of adopting another file owner's mutable bytes", async () => {
const companyId = await company(), directory = path.join(root, `linked-conversion-${randomUUID()}`);
await fs.mkdir(directory);
const original = path.join(root, `shared-content-${randomUUID()}.md`);
await fs.writeFile(original, "# Shared source");
await fs.link(original, path.join(directory, "SKILL.md"));
const [skill] = await db.insert(companySkills).values({ companyId, key: "linked-conversion", slug: "linked-conversion", name: "Linked",
markdown: "# Shared source", sourceType: "local_path", sourceLocator: directory,
fileInventory: [{ path: "SKILL.md", kind: "skill" }], metadata: { sourceKind: "managed_local" } }).returning();
expect((await skillInstallationMaintenance(db).installations({ id: skill.id, apply: true })).results)
.toEqual([expect.objectContaining({ status: "blocked" })]);
expect((await db.select().from(companySkills).where(eq(companySkills.id, skill.id)))[0]).toEqual(skill);
expect(await fs.readFile(original, "utf8")).toBe("# Shared source");
});
it("allows new durable skill pinning while an installation upload is blocked", async () => {
const companyId = await company(), directory = path.join(root, `pinning-during-copy-${randomUUID()}`);
await fs.mkdir(directory);
await fs.writeFile(path.join(directory, "SKILL.md"), "# Old selection");
const [skill] = await db.insert(companySkills).values({ companyId, key: "online-pinning", slug: "online-pinning", name: "Pinning",
markdown: "# Old selection", sourceType: "local_path", sourceLocator: directory,
fileInventory: [{ path: "SKILL.md", kind: "skill" }], metadata: { sourceKind: "managed_local" } }).returning();
const version = await companySkillService(db).createVersion(companyId, skill.id);
await fs.writeFile(path.join(directory, "SKILL.md"), `# Current local files ${randomUUID()}`);
const [agent] = await db.insert(agents).values({ companyId, name: "New work" }).returning();
const [run] = await db.insert(heartbeatRuns).values({ companyId, agentId: agent.id, status: "queued" }).returning();
const storage = storageModule.getStorageService(), put = storage.putFile.bind(storage);
let release!: () => void, ready!: () => void, blocked = false;
const barrier = new Promise<void>(resolve => { release = resolve; }), uploading = new Promise<void>(resolve => { ready = resolve; });
const upload = vi.spyOn(storage, "putFile").mockImplementation(async input => {
if (!blocked) { blocked = true; ready(); await barrier; }
return put(input);
});
const conversion = skillInstallationMaintenance(db, storage).installations({ id: skill.id, apply: true });
try {
await uploading;
// This uses the real nonblocking PostgreSQL owner, not a timing assertion.
await withSkillVersionCacheLock(db, companyId, async () => {
await db.update(heartbeatRuns).set({ status: "running", contextSnapshot: { skillVersions: { [skill.key]: version.id } } })
.where(eq(heartbeatRuns.id, run.id));
}, true);
} finally { release(); await conversion; upload.mockRestore(); }
expect((await conversion).results).toEqual([expect.objectContaining({ status: "converted" })]);
expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, run.id)))[0])
.toMatchObject({ status: "running", contextSnapshot: { skillVersions: { [skill.key]: version.id } } });
expect((await db.select().from(companySkillVersions).where(eq(companySkillVersions.id, version.id)))[0]).toBeDefined();
});
it.each([true, false])("keeps catalog history unchanged on rejection and preserves its reset version (concurrent edit: %s)", async concurrentEdit => {
const companyId = await company(), directory = path.join(root, `catalog-copy-${randomUUID()}`), origin = `${directory}-original`;
await fs.mkdir(directory); await fs.mkdir(origin);
await fs.writeFile(path.join(directory, "SKILL.md"), "# Installed edit");
await fs.writeFile(path.join(origin, "SKILL.md"), "# Original package");
const [skill] = await db.insert(companySkills).values({ companyId, key: "catalog-copy", slug: "catalog-copy", name: "Catalog",
markdown: "# Installed edit", sourceType: "catalog", sourceLocator: directory,
fileInventory: [{ path: "SKILL.md", kind: "skill" }], metadata: {
sourceKind: "catalog", originSnapshotLocator: origin, userModifiedAt: new Date().toISOString(),
} }).returning();
const [version] = await db.insert(companySkillVersions).values({ companyId, companySkillId: skill.id, revisionNumber: 1,
fileInventory: [{ path: "SKILL.md", kind: "skill", content: "# Installed edit" }] }).returning();
await db.update(companySkills).set({ currentVersionId: version.id }).where(eq(companySkills.id, skill.id));
const storage = storageModule.getStorageService(), put = storage.putFile.bind(storage);
let changed = false;
const upload = vi.spyOn(storage, "putFile").mockImplementation(async input => {
const saved = await put(input);
if (!changed) {
changed = true;
if (concurrentEdit) await fs.writeFile(path.join(origin, "SKILL.md"), "# Concurrent original edit");
}
return saved;
});
try {
expect((await skillInstallationMaintenance(db, storage).installations({ id: skill.id, apply: true })).results)
.toEqual([expect.objectContaining({ status: concurrentEdit ? "blocked" : "converted" })]);
expect(changed).toBe(true);
const history = await db.select().from(companySkillVersions).where(eq(companySkillVersions.companySkillId, skill.id));
expect(history).toHaveLength(concurrentEdit ? 1 : 2);
expect(history.find(row => row.id === version.id)).toMatchObject({ fileInventory: version.fileInventory });
const [current] = await db.select().from(companySkills).where(eq(companySkills.id, skill.id));
if (concurrentEdit) expect(current.metadata?.originVersionId).toBeUndefined();
else {
const original = history.find(row => row.id !== version.id)!;
expect(current.metadata?.originVersionId).toBe(original.id);
const store = skillPackageStore(db, storage), saved = await store.version(companyId, skill.id, original.id);
expect((await store.versionFiles(saved!)).find(file => file.path === "SKILL.md")?.content).toBe("# Original package");
}
expect(await fs.readFile(path.join(origin, "SKILL.md"), "utf8"))
.toBe(concurrentEdit ? "# Concurrent original edit" : "# Original package");
} finally { upload.mockRestore(); }
});
it("reports an unrecoverable legacy installation and leaves linked sources alone", async () => {
const companyId = await company();
const [legacy] = await db.insert(companySkills).values({ companyId, key: "legacy-missing", slug: "legacy-missing", name: "Missing",
+12 -8
View File
@@ -8,17 +8,18 @@ import type { StorageProvider, StorageService } from "../storage/types.js";
import { adoptAgentFiles, inspectAgentDirectory } from "./agent-file-store.js";
import { agentInstructionsBundleMode, deriveBundleState, resolveManagedInstructionsRoot } from "./agent-instructions.js";
import { durableAgentFilesEnabled } from "./agent-file-durability.js";
import { hasUnsettledLocalInstructionCopies } from "./agent-instruction-copy-preservation.js";
export const AGENT_FILE_OBJECT_GRACE_MS = 30 * 24 * 60 * 60_000;
/** One-time operator migration on the host that owns the volume. Cloud must
* stop ingress and execution writers before apply; row locks also serialize
* ordinary file mutations. A dry run never adopts or rewrites local files. */
/** Run on the host that owns the legacy volume. The canonical agent row lock
* serializes file writers and new admission. First adoption also requires its
* local working copies to be settled. A dry run never adopts local files. */
export async function migrateAgentFiles(db: Db, input: {
apply: boolean; writersStopped: boolean; companyId?: string; after?: string; limit?: number;
}) {
if (input.apply && (!input.writersStopped || !durableAgentFilesEnabled())) {
throw new Error("Agent file migration requires stopped writers and PAPERCLIP_DURABLE_AGENT_FILES=true");
if (input.apply && !durableAgentFilesEnabled()) {
throw new Error("Agent file migration requires PAPERCLIP_DURABLE_AGENT_FILES=true");
}
const limit = input.limit ?? 25;
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 100) throw new Error("Migration limit must be between 1 and 100");
@@ -32,19 +33,22 @@ export async function migrateAgentFiles(db: Db, input: {
const [agent] = await tx.select().from(agents).where(eq(agents.id, candidate.id)).for("update");
if (!agent) return "removed";
if (agentInstructionsBundleMode(agent) === "external") return "external_instructions";
const [head] = await tx.select({ generation: agentFileRoots.generation }).from(agentFileRoots)
.where(and(eq(agentFileRoots.companyId, agent.companyId), eq(agentFileRoots.agentId, agent.id)));
// Current shared authority needs no local rewrite, even during a run.
if (head) return "durable";
const [run] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
eq(heartbeatRuns.companyId, agent.companyId), eq(heartbeatRuns.agentId, agent.id),
input.writersStopped && process.env.PAPERCLIP_EPHEMERAL_MIGRATION === "true"
? offlineMigrationExecutionWhere() : inArray(heartbeatRuns.status, ["running", "queued"]),
)).limit(1);
if (run) return "active_execution";
const [head] = await tx.select({ generation: agentFileRoots.generation }).from(agentFileRoots)
.where(and(eq(agentFileRoots.companyId, agent.companyId), eq(agentFileRoots.agentId, agent.id)));
if (await hasUnsettledLocalInstructionCopies(tx, { companyId: agent.companyId, agentId: agent.id }))
return "working_copy_unsettled";
if (input.apply) {
await adoptAgentFiles(tx, agent);
return "durable";
}
if (head) return "durable";
const root = resolveManagedInstructionsRoot(agent);
const stored = agent.adapterConfig.instructionsRootPath;
if (typeof stored === "string" && (path.resolve(stored) !== root
@@ -0,0 +1,16 @@
import { and, eq, isNotNull, not, notInArray, sql } from "drizzle-orm";
import { agentInstructionWorkingCopies as copies, type Db } from "@paperclipai/db";
/** Terminal run status does not settle private edits. Only the copy owner's
* save/resolution or confirmed stopped no-save cleanup releases local bytes. */
export async function hasUnsettledLocalInstructionCopies(db: Pick<Db, "select">,
scope?: { companyId: string; agentId?: string }): Promise<boolean> {
return (await db.select({ runId: copies.runId }).from(copies).where(and(
scope ? eq(copies.companyId, scope.companyId) : undefined,
scope?.agentId ? eq(copies.agentId, scope.agentId) : undefined,
eq(copies.location, "local"), notInArray(copies.state, ["saved", "unchanged", "resolved", "superseded"]),
not(and(eq(copies.state, "unavailable"), isNotNull(copies.processStoppedAt),
sql`coalesce(${copies.receipt}->>'schema', '') = 'paperclip.agent-files.v1'`,
sql`coalesce(${copies.receipt}->>'cleanupPending', '') = 'false'`)!),
)).limit(1)).length > 0;
}
@@ -11,7 +11,7 @@ import { execFile as execFileCallback } from "node:child_process";
import { promisify } from "node:util";
import fs from "node:fs/promises";
import path from "node:path";
import { and, asc, desc, eq, gt, inArray, not, notInArray, lte, or, isNull, isNotNull, sql, type SQL } from "drizzle-orm";
import { and, asc, desc, eq, gt, inArray, lte, or, isNull, sql, type SQL } from "drizzle-orm";
import { agents, heartbeatRuns, agentInstructionWorkingCopies as copies, databaseWorkScopeKey, type Db } from "@paperclipai/db";
import {
prepareAdapterExecutionTargetRuntime,
@@ -41,18 +41,7 @@ const liveTargets = new Map<string, AdapterExecutionTarget>();
const targetKey = (companyId: string, runId: string) => `${companyId}:${runId}`;
const copySaves = new WeakMap<object, Map<string, Promise<void>>>();
/** A terminal run does not establish that its private local edits were saved.
* Enrollment may discard these paths only after this owner's save/resolution
* or explicit ownership transfer. A rejected directory save is settled only
* after this owner confirms both process stop and completed no-save cleanup. */
export async function hasUnsettledLocalInstructionCopies(db: Db): Promise<boolean> {
return (await db.select({ runId: copies.runId }).from(copies).where(and(
eq(copies.location, "local"), notInArray(copies.state, ["saved", "unchanged", "resolved", "superseded"]),
not(and(eq(copies.state, "unavailable"), isNotNull(copies.processStoppedAt),
sql`coalesce(${copies.receipt}->>'schema', '') = 'paperclip.agent-files.v1'`,
sql`coalesce(${copies.receipt}->>'cleanupPending', '') = 'false'`)!),
)).limit(1)).length > 0;
}
export { hasUnsettledLocalInstructionCopies } from "./agent-instruction-copy-preservation.js";
function serializeCopySave<T>(db: Db, companyId: string, runId: string, save: () => Promise<T>): Promise<T> {
const scope = databaseWorkScopeKey(db);
@@ -6,6 +6,7 @@ import { agentInstructionsBundleMode } from "./agent-instructions.js";
import { skillInstallationMaintenance } from "./skill-installation-maintenance.js";
import { hasLocalNativeHarnessObligations } from "./native-runtime/native-harness-backup-migration.js";
import { hasLocalNativeControllerObligations } from "./native-runtime/native-controller-migration.js";
import { hasUnsettledLocalInstructionCopies } from "./agent-instruction-copy-preservation.js";
export const ephemeralServerEnabled = () => process.env.PAPERCLIP_EPHEMERAL_SERVER === "true";
@@ -49,6 +50,7 @@ export async function readEphemeralServerDataEligibility(db: Db, options: { purp
and(eq(agentFileRoots.companyId, agents.companyId), eq(agentFileRoots.agentId, agents.id)))
.where(isNull(agentFileRoots.agentId)).limit(1);
if (missingFiles) blockers.push("agent_files_require_migration");
if (await hasUnsettledLocalInstructionCopies(db)) blockers.push("local_instruction_copies_require_settlement");
// Only inspect configuration here; filesystem reads must never adopt an
// empty directory on the replacement machine as the old server's content.
const configured = await db.select({ id: agents.id, companyId: agents.companyId, name: agents.name, adapterConfig: agents.adapterConfig }).from(agents);
@@ -24,6 +24,18 @@ function coversInstalledInventory(skill: typeof companySkills.$inferSelect, file
return paths.has("SKILL.md") && skill.fileInventory.every(file => typeof file.path === "string" && paths.has(file.path));
}
async function legacySourceIdentity(skill: typeof companySkills.$inferSelect) {
if (!skill.sourceLocator || !path.isAbsolute(skill.sourceLocator)) return null;
const source = path.resolve(skill.sourceLocator), root = path.basename(source) === "SKILL.md" ? path.dirname(source) : source;
const stat = await fs.lstat(root).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return null;
throw error;
});
if (!stat) return null;
if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("Legacy skill root is not a safe directory");
return { root: await fs.realpath(root), device: stat.dev, inode: stat.ino };
}
async function legacyFiles(skill: typeof companySkills.$inferSelect, signal?: AbortSignal) {
if (!skill.sourceLocator || !path.isAbsolute(skill.sourceLocator)) return null;
const source = path.resolve(skill.sourceLocator);
@@ -48,7 +60,7 @@ async function legacyFiles(skill: typeof companySkills.$inferSelect, signal?: Ab
const handle = await fs.open(absolute, constants.O_RDONLY | constants.O_NOFOLLOW);
try {
const stat = await handle.stat();
if (!stat.isFile()) throw new Error("Legacy skill contains an unsupported file");
if (!stat.isFile() || stat.nlink !== 1) throw new Error("Legacy skill contains an unsupported file");
bytes += stat.size;
if (files.length >= MAX_SKILL_SNAPSHOT_FILES || bytes > MAX_SKILL_SNAPSHOT_BYTES) throw new Error("Legacy skill exceeds package limits");
const body = Buffer.alloc(stat.size);
@@ -80,7 +92,6 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) {
options.after ? gt(companySkills.id, options.after) : undefined);
}
async function installations(options: Conversion = {}) {
if (options.apply && !options.writersStopped) throw conflict("Installation conversion requires stopped admission and retired legacy writers (--writers-stopped).");
const skills = await db.select().from(companySkills).where(scope(options)).orderBy(asc(companySkills.id)).limit(limit(options));
const results: Array<{ skillId: string; companyId: string; status: string; versionId?: string | null; reason?: string }> = [];
for (const skill of skills) {
@@ -97,60 +108,89 @@ export function skillInstallationMaintenance(db: Db, storage?: StorageService) {
results.push({ skillId: skill.id, companyId: skill.companyId, status: "unchanged", versionId: skill.currentVersionId });
continue;
}
const pins = await protectedVersions(db, skill.companyId);
if (pins.versions === null) throw new Error("Active run lacks an exact durable skill selection; settle or snapshot it on the legacy server first");
const saved = version ? await store.versionFiles(version) : null;
const local = await legacyFiles(skill, options.signal);
const completeSaved = saved && coversInstalledInventory(skill, saved) ? saved : null;
const files = local ?? completeSaved ?? ((skill.sourceType === "github" || skill.sourceType === "skills_sh" || skill.sourceType === "url")
? await companySkillService(db).captureLegacyInstallation(skill.companyId, skill.id) : null);
const origin = catalogOrigin && typeof skill.metadata?.originSnapshotLocator === "string"
? await legacyFiles({ ...skill, sourceLocator: skill.metadata.originSnapshotLocator }, options.signal) : null;
if (catalogOrigin && !origin && skill.metadata?.userModifiedAt) {
throw new Error("Catalog reset snapshot is unavailable; preserve the original installed package before conversion");
}
if (!files?.length || !files.some(file => file.path === "SKILL.md")) throw new Error("Neither complete legacy files nor a saved version is available");
if (!local && !coversInstalledInventory(skill, files)) {
throw new Error("Saved version does not cover the installed inventory");
}
if (!options.apply) {
results.push({ skillId: skill.id, companyId: skill.companyId, status: "conversion_required", versionId: skill.currentVersionId });
continue;
}
if (version && saved && !version.packageSnapshotId && skillSnapshotHash(saved) === skillSnapshotHash(files)) {
await skillPackageMaintenance(db, storage).migrateVersions({ id: version.id, apply: true, clearLegacy: true, signal: options.signal });
}
const candidate = await content.prepare(skill.companyId, files, options.signal);
if (!candidate.packageSnapshotId || !await store.verifyDurable(skill.companyId, candidate.packageSnapshotId, options.signal)) {
throw new Error("Final snapshot is not independently recoverable from object storage");
}
let originVersionId = typeof skill.metadata?.originVersionId === "string" ? skill.metadata.originVersionId : null;
if (catalogOrigin && origin && skillSnapshotHash(origin) !== skillSnapshotHash(files)) {
originVersionId = (await companySkillService(db).createVersion(skill.companyId, skill.id,
{ label: "Original catalog package" }, { type: "system" }, {
fileInventory: origin, updateCurrentVersion: false, skipInventoryRefresh: true,
})).id;
}
const markdown = files.find(file => file.path === "SKILL.md")!.content;
const parsed = parseFrontmatterMarkdown(markdown);
const { missingSource: _missing, ...metadata } = skill.metadata ?? {};
const result = await withSkillVersionCacheLock(db, skill.companyId, () => content.publish(skill, candidate, {
...(legacyManaged(skill) ? { sourceType: "managed", sourceLocator: null } : {}), markdown,
name: typeof parsed.frontmatter.name === "string" ? parsed.frontmatter.name : skill.name,
description: typeof parsed.frontmatter.description === "string" ? parsed.frontmatter.description : skill.description,
fileInventory: files.map(({ path, kind }) => ({ path, kind })),
trustLevel: files.some(file => file.kind === "script" || file.executable) ? "scripts_executables"
: files.some(file => file.kind === "asset" || file.kind === "other") ? "assets" : "markdown_only",
metadata: { ...metadata, ...(legacyManaged(skill) ? { sourceKind: "managed" } : {}),
...(originVersionId ? { originVersionId } : {}) },
}, { type: "system" }, { label: "Legacy installation conversion",
reuseCurrentVersion: Boolean(saved && skillSnapshotHash(saved) === skillSnapshotHash(files)),
afterPublish: async (tx, row, version) => {
if (catalogOrigin && !originVersionId) await tx.update(companySkills)
.set({ metadata: { ...row.metadata, originVersionId: version.id } }).where(eq(companySkills.id, row.id));
},
}));
results.push({ skillId: skill.id, companyId: skill.companyId, status: "converted", versionId: result.version.id });
// Conversion copies current content. Keep old directories, versions and
// inline bytes; unknown historical pins still block retention/eviction.
const convert = async () => {
const saved = version ? await store.versionFiles(version) : null;
const sourceIdentity = await legacySourceIdentity(skill);
const local = await legacyFiles(skill, options.signal);
const completeSaved = saved && coversInstalledInventory(skill, saved) ? saved : null;
const files = local ?? completeSaved ?? ((skill.sourceType === "github" || skill.sourceType === "skills_sh" || skill.sourceType === "url")
? await companySkillService(db).captureLegacyInstallation(skill.companyId, skill.id) : null);
const originSource = catalogOrigin && typeof skill.metadata?.originSnapshotLocator === "string"
? { ...skill, sourceLocator: skill.metadata.originSnapshotLocator } : null;
const originIdentity = originSource ? await legacySourceIdentity(originSource) : null;
const origin = originSource ? await legacyFiles(originSource, options.signal) : null;
if (catalogOrigin && !origin && skill.metadata?.userModifiedAt) {
throw new Error("Catalog reset snapshot is unavailable; preserve the original installed package before conversion");
}
if (!files?.length || !files.some(file => file.path === "SKILL.md")) throw new Error("Neither complete legacy files nor a saved version is available");
if (!local && !coversInstalledInventory(skill, files)) {
throw new Error("Saved version does not cover the installed inventory");
}
if (!options.apply) {
return { skillId: skill.id, companyId: skill.companyId, status: "conversion_required", versionId: skill.currentVersionId };
}
if (version && saved && !version.packageSnapshotId && !version.repositorySnapshotId
&& skillSnapshotHash(saved) === skillSnapshotHash(files)) {
await skillPackageMaintenance(db, storage).migrateVersions({ id: version.id, apply: true, clearLegacy: false, signal: options.signal });
}
const candidate = await content.prepare(skill.companyId, files, options.signal);
if (!candidate.packageSnapshotId || !await store.verifyDurable(skill.companyId, candidate.packageSnapshotId, options.signal)) {
throw new Error("Final snapshot is not independently recoverable from object storage");
}
let originVersionId = typeof skill.metadata?.originVersionId === "string" ? skill.metadata.originVersionId : null;
const originCandidate = catalogOrigin && origin && skillSnapshotHash(origin) !== skillSnapshotHash(files)
? await content.prepare(skill.companyId, origin, options.signal) : null;
if (originCandidate && !originCandidate.packageSnapshotId) throw new Error("Catalog origin is not durably available");
const markdown = files.find(file => file.path === "SKILL.md")!.content;
const parsed = parseFrontmatterMarkdown(markdown);
const { missingSource: _missing, ...metadata } = skill.metadata ?? {};
// Object preparation is private. Hold the shared cache lock only for
// publication so new run pinning does not wait for network uploads.
const result = await withSkillVersionCacheLock(db, skill.companyId, () => content.publish(skill, candidate, {
...(legacyManaged(skill) ? { sourceType: "managed", sourceLocator: null } : {}), markdown,
name: typeof parsed.frontmatter.name === "string" ? parsed.frontmatter.name : skill.name,
description: typeof parsed.frontmatter.description === "string" ? parsed.frontmatter.description : skill.description,
fileInventory: files.map(({ path, kind }) => ({ path, kind })),
trustLevel: files.some(file => file.kind === "script" || file.executable) ? "scripts_executables"
: files.some(file => file.kind === "asset" || file.kind === "other") ? "assets" : "markdown_only",
metadata: { ...metadata, ...(legacyManaged(skill) ? { sourceKind: "managed" } : {}),
...(originVersionId ? { originVersionId } : {}) },
}, { type: "system" }, { label: "Legacy installation conversion",
reuseCurrentVersion: Boolean(saved && skillSnapshotHash(saved) === skillSnapshotHash(files)),
validate: async tx => {
// publish already holds the canonical slug and skill-row locks.
// Recheck filesystem writers after upload, before changing authority.
const [current] = await tx.select().from(companySkills).where(and(
eq(companySkills.companyId, skill.companyId), eq(companySkills.id, skill.id)));
if (!current || current.slug !== skill.slug || current.sourceLocator !== skill.sourceLocator
|| current.sourceRef !== skill.sourceRef || JSON.stringify(current.fileInventory) !== JSON.stringify(skill.fileInventory)
|| JSON.stringify(await legacySourceIdentity(current)) !== JSON.stringify(sourceIdentity))
throw conflict("Legacy skill source changed during conversion. Retry the operation.");
const latest = await legacyFiles(current, options.signal);
if ((latest ? skillSnapshotHash(latest) : null) !== (local ? skillSnapshotHash(local) : null))
throw conflict("Legacy skill files changed during conversion. Retry the operation.");
if (originSource && (JSON.stringify(await legacySourceIdentity(originSource)) !== JSON.stringify(originIdentity)
|| (await legacyFiles(originSource, options.signal).then(value => value ? skillSnapshotHash(value) : null))
!== (origin ? skillSnapshotHash(origin) : null)))
throw conflict("Catalog origin changed during conversion. Retry the operation.");
if (origin && originCandidate?.packageSnapshotId) {
originVersionId = (await companySkillService(db).createVersion(skill.companyId, skill.id,
{ label: "Original catalog package" }, { type: "system" }, {
database: tx, fileInventory: origin, packageSnapshotId: originCandidate.packageSnapshotId,
packageFileMap: originCandidate.packageFileMap, updateCurrentVersion: false, skipInventoryRefresh: true,
})).id;
}
},
afterPublish: async (tx, row, version) => {
if (catalogOrigin) await tx.update(companySkills)
.set({ metadata: { ...row.metadata, originVersionId: originVersionId ?? version.id } }).where(eq(companySkills.id, row.id));
},
}));
return { skillId: skill.id, companyId: skill.companyId, status: "converted", versionId: result.version.id };
};
results.push(await convert());
} catch (error) {
results.push({ skillId: skill.id, companyId: skill.companyId, status: "blocked", reason: error instanceof Error ? error.message : "Conversion failed" });
}