mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 16:02:36 +01:00
DottaandPaperclip
a3eecf1fc9
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs keep the exact skill files admitted for execution and recovery. > - Older inputs can name immutable local bundles without a skill version ID. > - Installed skill heads alone do not preserve those admitted bundles. > - Server replacement needs a durable source for each retained bundle. > - This pull request copies authenticated bundles through the existing package owner and restores their exact projections. > - The benefit is preservation during online migration without stopping active work or settling recovery warnings. ## Linked Issues or Issue Description Refs #19656. Related: #19660. **What happened?** Older native execution inputs can contain skill bundles with a null version ID. Recovery skips those bundles. Startup installation checks can pass even when those retained inputs still require the old filesystem. **Expected behavior** Preserve each retained admitted bundle in object storage. Restore its original bytes, modes and digests on an empty replacement host. Keep its null pin and all run, controller and recovery state. Block startup and enrollment while a concrete local asset obligation remains. **Steps to reproduce** Retain a native run with an admitted local skill bundle and a null version ID. Convert its installed skill head to durable storage. Replace the server home and attempt recovery. The old bundle path remains in the recovered context. ## What Changed - Add a bounded `runtime-assets` phase to the existing skill package maintenance CLI. It copies exact admitted files under the runtime asset publication lock, verifies the durable package, and appends a run-bound receipt after source revalidation. - Keep original files, null pins, task intent, warnings and controller leases. Support an active run through copy-only preservation. Do not start a turn or claim provider ownership. - Verify and reuse a healthy canonical local bundle before storage reads. Restore missing or damaged bundles through the existing package and native asset materializers. Use the authenticated original manifest to preserve exact read and execute modes. Accept a later contract revision only when the run binding and exact skill selection remain unchanged. - Check retained recovery obligations and current task sessions during both startup and migration eligibility. Ignore ordinary admission that has no completed execution input. - Protect receipt-backed packages from orphan collection. Serialize receipt publication and deletion on the package row, then check references in a fresh statement after the lock. - Add real PostgreSQL and filesystem regressions. Replace a delete-method proxy with an actual upload-renewal row-lock race. Publish mutation activity only after commit. ## Verification - Baseline production startup gate failed the new concrete-asset regression as expected. The fixed gate passed. The actual before-fix preservation owner also failed recovery from a healthy local bundle during a simulated storage outage. A verified local-first path fixes that regression. - Eight preservation tests passed with real PostgreSQL, streaming storage, archive extraction and native materialization. They cover cold recovery, exact modes and binary bytes, active ownership, changed sources and bindings, contract revisions, missing or corrupt receipts, and publication/collection races. - Six affected package collection/deletion controls and three existing startup controls passed. The existing versioned recovery consumer passed a sequential retry. - One fixture attempt failed on its cleanup foreign-key order; the fixture was corrected. One existing suite hit its unchanged 30-second database startup hook before test execution. The unchanged single consumer passed on retry. These failures remain recorded. - Source, changed-test and CLI TypeScript checks passed. Rebase preserved both the local instruction-copy and runtime-asset guards. Current-head changed-test checks passed. Final PostgreSQL rerun passed all eleven preservation/startup cases. Current CLI check passed. Required CI remains pending at the trusted-runner gate. - Current canonical bundled creation uses package preparation plus atomic content/head publication. Native preparation requests immutable version snapshots before delivery. Converted installations and new canonical bundled creation therefore do not recreate the historical null-version asset dependency. - Fresh Greptile review on exact head `54c7dd048c9016c1af9fc68213d5ae053ac16499` passed at 5/5 with zero unresolved threads. - Module boundaries, whitespace, and credential/internal-reference scans passed. No live source writes, provider actions or migration apply were performed. ## Risks Missing, changed or unsafe local files remain blockers. This command does not repair a previously published package whose objects later disappear or become corrupt. It also refuses malformed preservation receipts. It retains the original local files and prior evidence and keeps removal blocked. Automatic repair requires a separate canonical package-owner recovery operation; direct database edits are not an accepted recovery path. Failed publication keeps the original files and any unreferenced package for normal collection. The readiness check is explicit startup or enrollment work, not a background scan. Its cost grows with retained admitted asset obligations. Older readers must be upgraded before the old volume is removed. This change preserves asset bytes; it does not settle business recovery or qualify unrelated volume dependencies. ## Model Used OpenAI GPT-6 (Codex). The runtime did not expose a more specific backend model ID or context-window size. Used reasoning, code editing, and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
42 lines
3.0 KiB
TypeScript
42 lines
3.0 KiB
TypeScript
/** Run from the server package with its configured storage and DATABASE_URL. */
|
|
import { createDb } from "@paperclipai/db";
|
|
import { loadStorageConfig } from "../src/storage/config.js";
|
|
import { createStorageProviderFromConfig } from "../src/storage/provider-registry.js";
|
|
import { createStorageService } from "../src/storage/service.js";
|
|
import { skillPackageMaintenance } from "../src/services/skill-package-maintenance.js";
|
|
import { skillInstallationMaintenance } from "../src/services/skill-installation-maintenance.js";
|
|
import { migrateRuntimeSkillAssets } from "../src/services/native-runtime/runtime-skill-preservation.js";
|
|
import { isCloudManagedInstance } from "../src/services/cloud-instance.js";
|
|
import { cloudRuntimeConfigService } from "../src/services/cloud-runtime-config.js";
|
|
import { setCloudRuntimeConfigSnapshot } from "../src/services/cloud-runtime-config-snapshot.js";
|
|
|
|
const value = (flag: string) => process.argv.find(arg => arg.startsWith(`${flag}=`))?.slice(flag.length + 1);
|
|
const phase = value("--phase") ?? "snapshots";
|
|
const url = process.env.DATABASE_URL;
|
|
if (!url) throw new Error("DATABASE_URL must explicitly select the database to migrate");
|
|
const limit = Number(value("--limit") ?? 10);
|
|
if (!Number.isSafeInteger(limit) || limit < 1 || limit > 50) throw new Error("--limit must be between 1 and 50");
|
|
const db = createDb(url, { maxConnections: 2 });
|
|
try {
|
|
if (isCloudManagedInstance()) {
|
|
const current = await cloudRuntimeConfigService(db).inspectStored();
|
|
if (!current) throw new Error("Initialize Cloud runtime configuration before running skill package maintenance");
|
|
setCloudRuntimeConfigSnapshot(current);
|
|
}
|
|
// Maintenance must inspect the old provider before Cloud cutover. Server
|
|
// startup prerequisites still apply to the normal global storage service.
|
|
const storage = createStorageService(createStorageProviderFromConfig(loadStorageConfig()));
|
|
const service = skillPackageMaintenance(db, storage);
|
|
const installations = skillInstallationMaintenance(db, storage);
|
|
const actions = { snapshots: service.migrateSnapshots, versions: service.migrateVersions,
|
|
orphans: service.collectOrphans, deletions: service.deleteObjects,
|
|
installations: installations.installations, eligibility: installations.eligibility, verification: installations.verifyVersions,
|
|
"runtime-assets": (options: Parameters<typeof migrateRuntimeSkillAssets>[1]) => migrateRuntimeSkillAssets(db, options, storage) };
|
|
const action = actions[phase as keyof typeof actions];
|
|
if (!action) throw new Error("--phase must be snapshots, versions, orphans, deletions, installations, eligibility, verification, or runtime-assets");
|
|
const apply = process.argv.includes("--apply");
|
|
const result = await action({ apply, writersStopped: process.argv.includes("--writers-stopped"),
|
|
clearLegacy: process.argv.includes("--clear-legacy"), limit, after: value("--after"), companyId: value("--company") });
|
|
console.log(JSON.stringify({ phase, apply, ...result }));
|
|
} finally { await db.$client.end({ timeout: 1 }); }
|