mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 11:20:42 +01:00
## Thinking Path > - Paperclip is the open source control plane people use to manage AI-agent companies > - Operators need a predictable installation path that survives beyond an ephemeral `npx` process > - A durable installation needs an owned per-user payload store, stable command shim, safe shell integration, and supported service lifecycle > - Updates must preserve recoverability by backing up data, installing side-by-side, verifying the new payload, and retaining rollback state > - Bootstrap scripts and privileged service operations must fail closed across download, filesystem, ownership, and consent boundaries > - This pull request integrates managed install, update, rollback, service, uninstall, doctor, bootstrap-installer, and runtime-serving support into one workflow > - The benefit is a recoverable, inspectable, and documented installation lifecycle with explicit safety boundaries across Linux, macOS, containers, WSL, npm, npx, and source checkouts ## Linked Issues or Issue Description ### Problem Paperclip lacks a first-class durable installation and lifecycle workflow. Operators currently have to assemble npm/npx installation, PATH setup, background-service management, updates, rollback, diagnostics, and uninstall behavior themselves. That makes upgrades harder to recover, creates inconsistent behavior across platforms, and leaves shell/download/service trust boundaries without one documented implementation. ### Proposed Solution Add a managed per-user install store and stable shim, a verified shell bootstrap installer, service lifecycle commands, install-mode-aware update/rollback behavior, doctor checks, and documentation. Managed updates back up the database, install and smoke-test a side-by-side payload, atomically switch `current`, and retain prior payloads. The shell installer pins registry/download trust boundaries and requires explicit consent for non-interactive privileged actions. ### Alternatives Considered - Keep recommending `npx`: simple for evaluation, but ephemeral and unsuitable for stable services, atomic updates, or rollback. - Require global npm installation only: familiar, but cannot provide the owned side-by-side payload store and retained rollback semantics. - Split the capability across multiple PRs: rejected because install, update, service, uninstall, bootstrap, and serving behavior share contracts and security boundaries that need review together. ### Related Pull Requests - Supersedes #10042 and #10044 with one integrated final diff. - Incorporates and replaces the closed preparatory work in #10032 and #10034. ## What Changed - Added `paperclipai install`, `update`/`upgrade`, rollback, uninstall, service lifecycle, onboarding integration, and managed-install doctor checks. - Added a private managed payload store, verified manifest/marker ownership, exclusive mutation locks, atomic manifest/current/shim writes, retained previous payloads, and provenance validation. - Added npm and GitHub-ref install sources with exact target resolution, registry isolation, database backup, side-by-side verification, atomic activation, service restart coordination, and failure rollback. - Made managed-update backups report actionable service-start and `--no-backup` recovery guidance for unreachable databases, while clean never-onboarded instances skip an empty backup. - Added systemd user and launchd service definitions, status/health/log commands, single-instance coordination, stale-port recovery, and explicit sudo/lingering consent handling. - Added the `scripts/install.sh` bootstrap path with checked two-stage downloads, pinned public npm registry usage, platform checks, dry-run/non-interactive controls, and Docker fixtures. - Added embedded Postgres/native bootstrap integration, hot-restart/systemd-notify serving support, passive update notices, configuration contracts, README/CLI/install documentation, and focused regression tests. - Security re-review should explicitly re-verify: (1) `addManagedPathBlock`/`removeManagedPathBlock` reject symlinked or non-regular rc files, assert current-user ownership, preserve restrictive modes, and replace atomically; (2) managed shim replacement rejects unsafe parents, foreign-owned or multiply linked files, and uses checked atomic replacement; (3) the shell installer and sudo path preserve explicit consent and checked downloads; and (4) installed service/runtime serving remains bound to the validated managed shim and instance configuration. ## Verification - `bash -n scripts/install.sh scripts/clean-install-git.sh scripts/clean-install-npm.sh scripts/test-install-sh-docker.sh` - `pnpm exec vitest run cli/src/__tests__/install-store.test.ts cli/src/__tests__/install-command.test.ts cli/src/__tests__/managed-install-check.test.ts cli/src/__tests__/onboard-service.test.ts cli/src/__tests__/service-health-check.test.ts cli/src/__tests__/service-manager.test.ts cli/src/__tests__/update-command.test.ts cli/src/__tests__/update-notice.test.ts packages/db/src/embedded-postgres-native.test.ts` — 9 files, 66 tests passed - `pnpm --dir cli typecheck` - `pnpm --dir cli build` - Follow-up verification: `pnpm exec vitest run cli/src/__tests__/update-command.test.ts` (14/14), `pnpm --dir cli typecheck`, `pnpm --dir cli build`, and `pnpm --filter @paperclipai/server typecheck`. - `pnpm -r typecheck` - `pnpm build` - Full `pnpm test:run` exercised all suites; an injected static AWS credential changed one unrelated doctor expectation, which passed when those credentials were removed. A second run cleared that case and exposed stale pre-existing adapter-utils `dist` output; rebuilding `@paperclipai/adapter-utils` made the isolated test pass. The updated PR CI is the authoritative clean-workspace full-suite run. ## Risks - Installer/update code writes executable shims, symlinks, shell rc blocks, service definitions, and managed payloads; ownership, regular-file, symlink, hard-link, marker, and path-containment checks fail closed before destructive changes. - The bootstrap installer executes downloaded tooling; downloads are staged and checked before execution, npm traffic is pinned to the public registry, and non-interactive privileged behavior requires explicit consent. - Linux lingering may invoke `sudo`; the command is surfaced and confirmed before execution, and unsupported service managers fall back to foreground-run guidance. - Database migrations remain forward-only; payload rollback does not reverse migrations, so managed updates create a backup before activation unless explicitly disabled. - Service restart and runtime serving touch process/port ownership; lifecycle locks, health/version checks, and stable-shim service definitions reduce split-brain and stale-process risk. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex coding agents using GPT-5.5 and GPT-5.6-sol, with reasoning, repository/API access, shell execution, and test tooling. The runtime did not expose a reliable context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
207 lines
8.1 KiB
TypeScript
207 lines
8.1 KiB
TypeScript
import fs from "node:fs";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
|
import {
|
|
INSTALL_MANIFEST_VERSION,
|
|
MANAGED_SHIM_MARKER,
|
|
addManagedPathBlock,
|
|
buildNextManifest,
|
|
flipCurrentAtomic,
|
|
isManagedExecutable,
|
|
payloadPathFor,
|
|
pruneInstallPayloads,
|
|
readInstallManifest,
|
|
removeManagedPathBlock,
|
|
removeManagedShim,
|
|
resolveInstallStorePaths,
|
|
withInstallStoreLock,
|
|
writeInstallManifestAtomic,
|
|
writeManagedShim,
|
|
type InstallManifest,
|
|
type InstallRecord,
|
|
} from "../install-store.js";
|
|
|
|
function record(payloadPath: string, version: string): InstallRecord {
|
|
return {
|
|
source: "npm",
|
|
version,
|
|
channel: "latest",
|
|
payloadPath,
|
|
installedAt: `2026-07-${version.padStart(2, "0")}T00:00:00.000Z`,
|
|
};
|
|
}
|
|
|
|
describe("managed install store", () => {
|
|
let root: string;
|
|
let paths: ReturnType<typeof resolveInstallStorePaths>;
|
|
|
|
beforeEach(() => {
|
|
root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-install-store-"));
|
|
paths = resolveInstallStorePaths({
|
|
homeDir: path.join(root, "home"),
|
|
paperclipHome: path.join(root, "home", ".paperclip"),
|
|
});
|
|
});
|
|
|
|
afterEach(() => {
|
|
fs.rmSync(root, { recursive: true, force: true });
|
|
});
|
|
|
|
it("resolves the documented npm and git payload layout", () => {
|
|
expect(payloadPathFor(paths, "npm", "2026.720.0")).toBe(
|
|
path.join(paths.cliRoot, "installs", "npm", "2026.720.0"),
|
|
);
|
|
expect(payloadPathFor(paths, "git", "ab12cd34ef56")).toBe(
|
|
path.join(paths.cliRoot, "installs", "git", "ab12cd34ef56"),
|
|
);
|
|
});
|
|
|
|
it("writes and reads the manifest atomically with private permissions", () => {
|
|
const payloadPath = payloadPathFor(paths, "npm", "1.2.3");
|
|
const manifest: InstallManifest = {
|
|
schemaVersion: INSTALL_MANIFEST_VERSION,
|
|
...record(payloadPath, "1.2.3"),
|
|
previous: [],
|
|
};
|
|
writeInstallManifestAtomic(manifest, paths);
|
|
expect(readInstallManifest(paths)).toEqual(manifest);
|
|
expect(fs.statSync(paths.manifestPath).mode & 0o777).toBe(0o600);
|
|
});
|
|
|
|
it("leaves the old current payload working when interrupted before rename", () => {
|
|
const oldPayload = payloadPathFor(paths, "npm", "1.0.0");
|
|
const newPayload = payloadPathFor(paths, "npm", "2.0.0");
|
|
fs.mkdirSync(oldPayload, { recursive: true });
|
|
fs.mkdirSync(newPayload, { recursive: true });
|
|
flipCurrentAtomic(oldPayload, paths);
|
|
|
|
expect(() =>
|
|
flipCurrentAtomic(newPayload, paths, {
|
|
beforeRename: () => {
|
|
throw new Error("simulated crash");
|
|
},
|
|
}),
|
|
).toThrow("simulated crash");
|
|
|
|
expect(fs.realpathSync(paths.currentPath)).toBe(fs.realpathSync(oldPayload));
|
|
expect(fs.readdirSync(paths.cliRoot).filter((entry) => entry.startsWith(".current-"))).toEqual([]);
|
|
});
|
|
|
|
it("retains current plus two previous payloads and prunes older entries", () => {
|
|
const payloads = ["1", "2", "3", "4"].map((version) => payloadPathFor(paths, "npm", version));
|
|
for (const payload of payloads) fs.mkdirSync(payload, { recursive: true });
|
|
const previousManifest: InstallManifest = {
|
|
schemaVersion: INSTALL_MANIFEST_VERSION,
|
|
...record(payloads[2], "3"),
|
|
previous: [record(payloads[1], "2"), record(payloads[0], "1")],
|
|
};
|
|
const next = buildNextManifest(record(payloads[3], "4"), previousManifest);
|
|
|
|
expect(next.previous.map((entry) => entry.version)).toEqual(["3", "2"]);
|
|
expect(pruneInstallPayloads(next, paths)).toEqual([payloads[0]]);
|
|
expect(fs.existsSync(payloads[0])).toBe(false);
|
|
expect(payloads.slice(1).every((payload) => fs.existsSync(payload))).toBe(true);
|
|
});
|
|
|
|
it("writes a stable shim with the validated runtime and custom store path", () => {
|
|
writeManagedShim(paths);
|
|
const shim = fs.readFileSync(paths.shimPath, "utf8");
|
|
expect(shim).toContain(process.execPath);
|
|
expect(shim).toContain(paths.currentPath);
|
|
expect(shim).not.toContain("PAPERCLIP_HOME");
|
|
expect(fs.statSync(paths.shimPath).mode & 0o777).toBe(0o755);
|
|
|
|
const rcPath = path.join(root, "home", ".bashrc");
|
|
expect(addManagedPathBlock(rcPath)).toBe(true);
|
|
expect(addManagedPathBlock(rcPath)).toBe(false);
|
|
fs.chmodSync(rcPath, 0o640);
|
|
expect(removeManagedPathBlock(rcPath)).toBe(true);
|
|
expect(fs.readFileSync(rcPath, "utf8")).not.toContain("paperclipai managed PATH");
|
|
expect(fs.statSync(rcPath).mode & 0o777).toBe(0o640);
|
|
});
|
|
|
|
it("rejects marker substrings that are not the exact managed shim format", () => {
|
|
fs.mkdirSync(path.dirname(paths.shimPath), { recursive: true });
|
|
fs.writeFileSync(paths.shimPath, `#!/bin/sh\necho '${MANAGED_SHIM_MARKER}'\n`);
|
|
|
|
expect(removeManagedShim(paths)).toBe(false);
|
|
expect(fs.existsSync(paths.shimPath)).toBe(true);
|
|
expect(() => writeManagedShim(paths)).toThrow("non-managed command");
|
|
});
|
|
|
|
it("serializes install-store mutations with an exclusive lock", async () => {
|
|
await expect(
|
|
withInstallStoreLock(
|
|
() => withInstallStoreLock(async () => undefined, paths),
|
|
paths,
|
|
),
|
|
).rejects.toThrow("already running");
|
|
expect(fs.existsSync(paths.lockPath)).toBe(false);
|
|
});
|
|
|
|
it("recovers a lock owned by a process that no longer exists", async () => {
|
|
const staleToken = "2147483647:stale";
|
|
await withInstallStoreLock(async () => undefined, paths);
|
|
fs.writeFileSync(paths.lockPath, `${staleToken}\n`, { mode: 0o600 });
|
|
|
|
await expect(withInstallStoreLock(async () => undefined, paths)).resolves.toBeUndefined();
|
|
expect(fs.existsSync(paths.lockPath)).toBe(false);
|
|
});
|
|
|
|
it("reports managed provenance only for the payload selected by current", () => {
|
|
const manifestPayload = payloadPathFor(paths, "npm", "1.0.0");
|
|
const currentPayload = payloadPathFor(paths, "npm", "2.0.0");
|
|
const executable = path.join(manifestPayload, "node_modules", "paperclipai", "dist", "index.js");
|
|
fs.mkdirSync(path.dirname(executable), { recursive: true });
|
|
fs.writeFileSync(executable, "");
|
|
fs.mkdirSync(currentPayload, { recursive: true });
|
|
flipCurrentAtomic(currentPayload, paths);
|
|
const manifest: InstallManifest = {
|
|
schemaVersion: INSTALL_MANIFEST_VERSION,
|
|
...record(manifestPayload, "1.0.0"),
|
|
previous: [],
|
|
};
|
|
|
|
expect(isManagedExecutable(executable, manifest, paths)).toBe(false);
|
|
});
|
|
|
|
it("refuses symlinked payload roots and pre-existing non-managed shims", () => {
|
|
const outside = path.join(root, "outside");
|
|
fs.mkdirSync(outside, { recursive: true });
|
|
fs.mkdirSync(paths.installsRoot, { recursive: true });
|
|
fs.symlinkSync(outside, path.join(paths.installsRoot, "npm"), "dir");
|
|
const escapedPayload = path.join(paths.installsRoot, "npm", "1.2.3");
|
|
fs.mkdirSync(path.join(outside, "1.2.3"));
|
|
expect(() => flipCurrentAtomic(escapedPayload, paths)).toThrow("resolves outside");
|
|
|
|
fs.mkdirSync(path.dirname(paths.shimPath), { recursive: true });
|
|
fs.writeFileSync(paths.shimPath, "#!/bin/sh\necho other-command\n");
|
|
expect(() => writeManagedShim(paths)).toThrow("non-managed command");
|
|
});
|
|
|
|
it("refuses symlinked rc files, unsafe shim parents, and multiply linked shims", () => {
|
|
const outsideRc = path.join(root, "outside-rc");
|
|
fs.writeFileSync(outsideRc, "keep\n");
|
|
const rcPath = path.join(root, "home", ".bashrc");
|
|
fs.mkdirSync(path.dirname(rcPath), { recursive: true });
|
|
fs.symlinkSync(outsideRc, rcPath);
|
|
expect(() => addManagedPathBlock(rcPath)).toThrow("non-regular shell rc file");
|
|
expect(() => removeManagedPathBlock(rcPath)).toThrow("non-regular shell rc file");
|
|
expect(fs.readFileSync(outsideRc, "utf8")).toBe("keep\n");
|
|
|
|
fs.rmSync(rcPath);
|
|
const localDir = path.join(root, "home", ".local");
|
|
const outsideBin = path.join(root, "outside-bin");
|
|
fs.mkdirSync(outsideBin);
|
|
fs.symlinkSync(outsideBin, localDir, "dir");
|
|
expect(() => writeManagedShim(paths)).toThrow("unsafe shim directory");
|
|
|
|
fs.rmSync(localDir);
|
|
fs.mkdirSync(path.dirname(paths.shimPath), { recursive: true });
|
|
fs.writeFileSync(paths.shimPath, `# ${MANAGED_SHIM_MARKER}\n`);
|
|
fs.linkSync(paths.shimPath, path.join(root, "linked-shim"));
|
|
expect(() => writeManagedShim(paths)).toThrow("multiply linked shim");
|
|
});
|
|
});
|