mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-06 21:05:21 +02:00
## Thinking Path > - Paperclip is the open source control plane people use to manage AI-agent companies > - Operators need a predictable installation path that survives beyond an ephemeral `npx` process > - A durable installation needs an owned per-user payload store, stable command shim, safe shell integration, and supported service lifecycle > - Updates must preserve recoverability by backing up data, installing side-by-side, verifying the new payload, and retaining rollback state > - Bootstrap scripts and privileged service operations must fail closed across download, filesystem, ownership, and consent boundaries > - This pull request integrates managed install, update, rollback, service, uninstall, doctor, bootstrap-installer, and runtime-serving support into one workflow > - The benefit is a recoverable, inspectable, and documented installation lifecycle with explicit safety boundaries across Linux, macOS, containers, WSL, npm, npx, and source checkouts ## Linked Issues or Issue Description ### Problem Paperclip lacks a first-class durable installation and lifecycle workflow. Operators currently have to assemble npm/npx installation, PATH setup, background-service management, updates, rollback, diagnostics, and uninstall behavior themselves. That makes upgrades harder to recover, creates inconsistent behavior across platforms, and leaves shell/download/service trust boundaries without one documented implementation. ### Proposed Solution Add a managed per-user install store and stable shim, a verified shell bootstrap installer, service lifecycle commands, install-mode-aware update/rollback behavior, doctor checks, and documentation. Managed updates back up the database, install and smoke-test a side-by-side payload, atomically switch `current`, and retain prior payloads. The shell installer pins registry/download trust boundaries and requires explicit consent for non-interactive privileged actions. ### Alternatives Considered - Keep recommending `npx`: simple for evaluation, but ephemeral and unsuitable for stable services, atomic updates, or rollback. - Require global npm installation only: familiar, but cannot provide the owned side-by-side payload store and retained rollback semantics. - Split the capability across multiple PRs: rejected because install, update, service, uninstall, bootstrap, and serving behavior share contracts and security boundaries that need review together. ### Related Pull Requests - Supersedes #10042 and #10044 with one integrated final diff. - Incorporates and replaces the closed preparatory work in #10032 and #10034. ## What Changed - Added `paperclipai install`, `update`/`upgrade`, rollback, uninstall, service lifecycle, onboarding integration, and managed-install doctor checks. - Added a private managed payload store, verified manifest/marker ownership, exclusive mutation locks, atomic manifest/current/shim writes, retained previous payloads, and provenance validation. - Added npm and GitHub-ref install sources with exact target resolution, registry isolation, database backup, side-by-side verification, atomic activation, service restart coordination, and failure rollback. - Made managed-update backups report actionable service-start and `--no-backup` recovery guidance for unreachable databases, while clean never-onboarded instances skip an empty backup. - Added systemd user and launchd service definitions, status/health/log commands, single-instance coordination, stale-port recovery, and explicit sudo/lingering consent handling. - Added the `scripts/install.sh` bootstrap path with checked two-stage downloads, pinned public npm registry usage, platform checks, dry-run/non-interactive controls, and Docker fixtures. - Added embedded Postgres/native bootstrap integration, hot-restart/systemd-notify serving support, passive update notices, configuration contracts, README/CLI/install documentation, and focused regression tests. - Security re-review should explicitly re-verify: (1) `addManagedPathBlock`/`removeManagedPathBlock` reject symlinked or non-regular rc files, assert current-user ownership, preserve restrictive modes, and replace atomically; (2) managed shim replacement rejects unsafe parents, foreign-owned or multiply linked files, and uses checked atomic replacement; (3) the shell installer and sudo path preserve explicit consent and checked downloads; and (4) installed service/runtime serving remains bound to the validated managed shim and instance configuration. ## Verification - `bash -n scripts/install.sh scripts/clean-install-git.sh scripts/clean-install-npm.sh scripts/test-install-sh-docker.sh` - `pnpm exec vitest run cli/src/__tests__/install-store.test.ts cli/src/__tests__/install-command.test.ts cli/src/__tests__/managed-install-check.test.ts cli/src/__tests__/onboard-service.test.ts cli/src/__tests__/service-health-check.test.ts cli/src/__tests__/service-manager.test.ts cli/src/__tests__/update-command.test.ts cli/src/__tests__/update-notice.test.ts packages/db/src/embedded-postgres-native.test.ts` — 9 files, 66 tests passed - `pnpm --dir cli typecheck` - `pnpm --dir cli build` - Follow-up verification: `pnpm exec vitest run cli/src/__tests__/update-command.test.ts` (14/14), `pnpm --dir cli typecheck`, `pnpm --dir cli build`, and `pnpm --filter @paperclipai/server typecheck`. - `pnpm -r typecheck` - `pnpm build` - Full `pnpm test:run` exercised all suites; an injected static AWS credential changed one unrelated doctor expectation, which passed when those credentials were removed. A second run cleared that case and exposed stale pre-existing adapter-utils `dist` output; rebuilding `@paperclipai/adapter-utils` made the isolated test pass. The updated PR CI is the authoritative clean-workspace full-suite run. ## Risks - Installer/update code writes executable shims, symlinks, shell rc blocks, service definitions, and managed payloads; ownership, regular-file, symlink, hard-link, marker, and path-containment checks fail closed before destructive changes. - The bootstrap installer executes downloaded tooling; downloads are staged and checked before execution, npm traffic is pinned to the public registry, and non-interactive privileged behavior requires explicit consent. - Linux lingering may invoke `sudo`; the command is surfaced and confirmed before execution, and unsupported service managers fall back to foreground-run guidance. - Database migrations remain forward-only; payload rollback does not reverse migrations, so managed updates create a backup before activation unless explicitly disabled. - Service restart and runtime serving touch process/port ownership; lifecycle locks, health/version checks, and stable-shim service definitions reduce split-brain and stale-process risk. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex coding agents using GPT-5.5 and GPT-5.6-sol, with reasoning, repository/API access, shell execution, and test tooling. The runtime did not expose a reliable context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
231 lines
10 KiB
Bash
Executable File
231 lines
10 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# End-to-end proof of the paperclipai managed install lifecycle on a CLEAN machine.
|
|
#
|
|
# Exercises the real user journey against real GitHub + real npm:
|
|
# bootstrap build -> install (npm latest) -> install --ref (build-from-source)
|
|
# -> update --check -> update --rollback -> reinstall (payload reuse)
|
|
# -> bad-ref failure hygiene -> service lifecycle -> uninstall (data preserved)
|
|
#
|
|
# Machine requirements: bash, curl, tar, node >= 20 (with corepack), npm.
|
|
# The machine's $HOME must not already contain a managed install.
|
|
#
|
|
# Env knobs:
|
|
# E2E_REPO GitHub repo to install from (default: paperclipai/paperclip)
|
|
# E2E_REF branch/tag/sha to install (default: master)
|
|
# E2E_SKIP_NPM=1 skip the npm-channel install step (canary is tested separately;
|
|
# the npm leg uses the latest channel)
|
|
# E2E_SKIP_SERVICE=1 skip the service lifecycle step
|
|
# E2E_SERVICE_TIMEOUT_SECS how long to wait for the service to go active (default 300)
|
|
set -uo pipefail
|
|
|
|
E2E_REPO="${E2E_REPO:-paperclipai/paperclip}"
|
|
E2E_REF="${E2E_REF:-master}"
|
|
E2E_SERVICE_TIMEOUT_SECS="${E2E_SERVICE_TIMEOUT_SECS:-300}"
|
|
|
|
# A clean environment: no inherited Paperclip or build-mode state.
|
|
for var in $(env | grep -o '^PAPERCLIP_[A-Z_]*' || true); do unset "$var"; done
|
|
unset NODE_ENV npm_config_prefix 2>/dev/null || true
|
|
export COREPACK_ENABLE_DOWNLOAD_PROMPT=0
|
|
export CI="${CI:-1}"
|
|
|
|
SHIM="$HOME/.local/bin/paperclipai"
|
|
STORE="$HOME/.paperclip/cli"
|
|
RESULTS=()
|
|
FAILED=0
|
|
|
|
note() { printf '\n\033[1;34m== %s ==\033[0m\n' "$*"; }
|
|
pass() { RESULTS+=("PASS $1"); printf '\033[1;32mPASS\033[0m %s\n' "$1"; }
|
|
fail_() { RESULTS+=("FAIL $1"); printf '\033[1;31mFAIL\033[0m %s\n' "$1"; FAILED=1; }
|
|
skip_() { RESULTS+=("SKIP $1${2:+ — $2}"); printf '\033[1;33mSKIP\033[0m %s%s\n' "$1" "${2:+ — $2}"; }
|
|
|
|
shim() { "$SHIM" "$@"; }
|
|
current_target() { readlink "$STORE/current" 2>/dev/null || echo "<missing>"; }
|
|
|
|
note "0. Preflight — this machine"
|
|
uname -a
|
|
node --version && npm --version && curl --version | head -1
|
|
command -v corepack >/dev/null || npm install -g corepack
|
|
[ -e "$SHIM" ] && { echo "shim already exists at $SHIM — not a clean machine"; exit 2; }
|
|
[ -d "$STORE" ] && { echo "store already exists at $STORE — not a clean machine"; exit 2; }
|
|
echo "repo=$E2E_REPO ref=$E2E_REF home=$HOME"
|
|
|
|
note "1. Bootstrap: build the new CLI from the GitHub tarball of $E2E_REF"
|
|
# Nothing published on npm has the install/update/service commands yet, so the
|
|
# bootstrap simulates what `npx paperclipai@<channel> install` will run post-release:
|
|
# the same CLI code, built from the exact ref under test.
|
|
BOOT="$HOME/e2e-bootstrap"
|
|
mkdir -p "$BOOT"
|
|
if curl --fail --silent --show-error --location \
|
|
"https://codeload.github.com/$E2E_REPO/tar.gz/$E2E_REF" \
|
|
| tar -xz --strip-components=1 -C "$BOOT"; then
|
|
pass "1a bootstrap tarball downloaded from codeload"
|
|
else
|
|
fail_ "1a bootstrap tarball download"; exit 1
|
|
fi
|
|
cd "$BOOT"
|
|
if corepack pnpm install --frozen-lockfile > "$HOME/e2e-bootstrap-install.log" 2>&1; then
|
|
pass "1b bootstrap pnpm install"
|
|
else
|
|
tail -40 "$HOME/e2e-bootstrap-install.log"; fail_ "1b bootstrap pnpm install"; exit 1
|
|
fi
|
|
if bash scripts/build-npm.sh --skip-checks --skip-typecheck > "$HOME/e2e-bootstrap-build.log" 2>&1; then
|
|
pass "1c bootstrap build-npm.sh"
|
|
else
|
|
tail -40 "$HOME/e2e-bootstrap-build.log"; fail_ "1c bootstrap build-npm.sh"; exit 1
|
|
fi
|
|
# The in-checkout dist resolves externals against the publishable package.json,
|
|
# so run the bootstrap exactly the way npm users get it: pack + install the tarball.
|
|
TARBALL="$(cd "$BOOT/cli" && npm pack --silent 2>/dev/null | tail -1)"
|
|
mkdir -p "$HOME/e2e-bootstrap-cli"
|
|
if (cd "$HOME/e2e-bootstrap-cli" && npm install --no-fund --no-audit "$BOOT/cli/$TARBALL" > "$HOME/e2e-bootstrap-npm.log" 2>&1); then
|
|
pass "1d bootstrap CLI packed + npm-installed ($TARBALL)"
|
|
else
|
|
tail -40 "$HOME/e2e-bootstrap-npm.log"; fail_ "1d bootstrap CLI npm install"; exit 1
|
|
fi
|
|
BOOTSTRAP_CLI="$HOME/e2e-bootstrap-cli/node_modules/paperclipai/dist/index.js"
|
|
node "$BOOTSTRAP_CLI" --version >/dev/null || { fail_ "1e bootstrap CLI smoke"; exit 1; }
|
|
cd "$HOME"
|
|
|
|
if [ "${E2E_SKIP_NPM:-0}" != "1" ]; then
|
|
note "2. install (published npm latest channel; proves the npm install mechanism)"
|
|
if node "$BOOTSTRAP_CLI" install --yes; then
|
|
pass "2a install (latest) exits 0"
|
|
else
|
|
fail_ "2a install (latest) exits 0"
|
|
fi
|
|
[ -x "$SHIM" ] && pass "2b shim created at ~/.local/bin/paperclipai" || fail_ "2b shim created"
|
|
case "$(current_target)" in
|
|
*"installs/npm/"*) pass "2c current -> installs/npm/<version> ($(basename "$(current_target)"))" ;;
|
|
*) fail_ "2c current -> installs/npm/<version> (got: $(current_target))" ;;
|
|
esac
|
|
[ -f "$STORE/install.json" ] && pass "2d install.json manifest present" || fail_ "2d install.json manifest present"
|
|
NPM_VERSION="$("$SHIM" --version 2>/dev/null || true)"
|
|
[ -n "$NPM_VERSION" ] && pass "2e shim runs: paperclipai --version = $NPM_VERSION" || fail_ "2e shim runs paperclipai --version"
|
|
else
|
|
skip_ "2 install (npm latest)" "E2E_SKIP_NPM=1"
|
|
fi
|
|
|
|
note "3. install --ref $E2E_REF (real build-from-GitHub-source into the managed store)"
|
|
if node "$BOOTSTRAP_CLI" install --repo "$E2E_REPO" --ref "$E2E_REF" --yes; then
|
|
pass "3a install --ref exits 0"
|
|
else
|
|
fail_ "3a install --ref exits 0"
|
|
fi
|
|
case "$(current_target)" in
|
|
*"installs/git/"*) pass "3b current -> installs/git/<sha> ($(basename "$(current_target)"))" ;;
|
|
*) fail_ "3b current -> installs/git/<sha> (got: $(current_target))" ;;
|
|
esac
|
|
GIT_VERSION="$("$SHIM" --version 2>/dev/null || true)"
|
|
[ -n "$GIT_VERSION" ] && pass "3c shim runs git payload: --version = $GIT_VERSION" || fail_ "3c shim runs git payload"
|
|
[ -x "$SHIM" ] && pass "3d shim still in place" || fail_ "3d shim still in place"
|
|
|
|
note "4. update --check from the managed shim"
|
|
shim update --check --json; CHECK_EXIT=$?
|
|
if [ "$CHECK_EXIT" -eq 0 ] || [ "$CHECK_EXIT" -eq 10 ]; then
|
|
pass "4a update --check exits $CHECK_EXIT (0=current, 10=update available)"
|
|
else
|
|
fail_ "4a update --check exit code (got $CHECK_EXIT)"
|
|
fi
|
|
|
|
if [ "${E2E_SKIP_NPM:-0}" != "1" ]; then
|
|
note "5. update --rollback (git payload -> previous npm payload)"
|
|
if shim update --rollback; then
|
|
pass "5a update --rollback exits 0"
|
|
else
|
|
fail_ "5a update --rollback exits 0"
|
|
fi
|
|
case "$(current_target)" in
|
|
*"installs/npm/"*) pass "5b rollback restored npm payload ($(basename "$(current_target)"))" ;;
|
|
*) fail_ "5b rollback restored npm payload (got: $(current_target))" ;;
|
|
esac
|
|
ROLLED_VERSION="$("$SHIM" --version 2>/dev/null || true)"
|
|
[ "$ROLLED_VERSION" = "$NPM_VERSION" ] \
|
|
&& pass "5c version after rollback matches npm payload ($ROLLED_VERSION)" \
|
|
|| fail_ "5c version after rollback ($ROLLED_VERSION != $NPM_VERSION)"
|
|
|
|
note "6. reinstall the git ref (payload retained -> reused, no rebuild)"
|
|
REINSTALL_START=$(date +%s)
|
|
if node "$BOOTSTRAP_CLI" install --repo "$E2E_REPO" --ref "$E2E_REF" --yes; then
|
|
REINSTALL_SECS=$(( $(date +%s) - REINSTALL_START ))
|
|
pass "6a reinstall exits 0 (${REINSTALL_SECS}s — reused payload should be fast)"
|
|
else
|
|
fail_ "6a reinstall exits 0"
|
|
fi
|
|
case "$(current_target)" in
|
|
*"installs/git/"*) pass "6b back on git payload" ;;
|
|
*) fail_ "6b back on git payload (got: $(current_target))" ;;
|
|
esac
|
|
else
|
|
skip_ "5-6 rollback/reinstall" "E2E_SKIP_NPM=1"
|
|
fi
|
|
|
|
note "7. failure hygiene: install --ref <nonexistent> must fail cleanly"
|
|
BEFORE_DIRS="$(ls "$STORE/installs/git" 2>/dev/null | sort)"
|
|
if node "$BOOTSTRAP_CLI" install --ref e2e-definitely-not-a-ref-xyz --yes 2>&1; then
|
|
fail_ "7a bad ref rejected (command unexpectedly succeeded)"
|
|
else
|
|
pass "7a bad ref rejected with nonzero exit"
|
|
fi
|
|
AFTER_DIRS="$(ls "$STORE/installs/git" 2>/dev/null | sort)"
|
|
[ "$BEFORE_DIRS" = "$AFTER_DIRS" ] && pass "7b no partial install dir left behind" || fail_ "7b no partial install dir left behind"
|
|
"$SHIM" --version >/dev/null 2>&1 && pass "7c existing install still healthy" || fail_ "7c existing install still healthy"
|
|
|
|
if [ "${E2E_SKIP_SERVICE:-0}" = "1" ]; then
|
|
skip_ "8 service lifecycle" "E2E_SKIP_SERVICE=1"
|
|
else
|
|
if [ "$(uname -s)" = "Linux" ] && [ ! -S "/run/user/$(id -u)/bus" ]; then
|
|
skip_ "8 service lifecycle" "no systemd user bus at /run/user/$(id -u)/bus"
|
|
else
|
|
note "8. service lifecycle ($(uname -s): systemd/launchd)"
|
|
# Real quickstart path: onboard with defaults, then install + start the service.
|
|
if shim onboard --yes --install-service; then
|
|
pass "8a onboard --yes --install-service exits 0"
|
|
else
|
|
fail_ "8a onboard --yes --install-service exits 0"
|
|
fi
|
|
DEADLINE=$(( $(date +%s) + E2E_SERVICE_TIMEOUT_SECS ))
|
|
ACTIVE=0
|
|
while [ "$(date +%s)" -lt "$DEADLINE" ]; do
|
|
STATUS_JSON="$(shim service status --json 2>/dev/null || true)"
|
|
if echo "$STATUS_JSON" | grep -q '"active"[[:space:]]*:[[:space:]]*true'; then ACTIVE=1; break; fi
|
|
sleep 5
|
|
done
|
|
if [ "$ACTIVE" = "1" ]; then
|
|
pass "8b service reached active within ${E2E_SERVICE_TIMEOUT_SECS}s"
|
|
else
|
|
echo "last status: ${STATUS_JSON:-<none>}"
|
|
shim service logs -n 60 || true
|
|
fail_ "8b service reached active"
|
|
fi
|
|
shim service logs -n 20 >/dev/null 2>&1 && pass "8c service logs readable" || fail_ "8c service logs readable"
|
|
if shim service stop; then pass "8d service stop exits 0"; else fail_ "8d service stop exits 0"; fi
|
|
if shim service uninstall; then pass "8e service uninstall exits 0"; else fail_ "8e service uninstall exits 0"; fi
|
|
fi
|
|
fi
|
|
|
|
note "9. installer script guardrails (from the bootstrap checkout)"
|
|
# Capture first: under pipefail, install.sh's expected exit 1 would fail the pipeline.
|
|
GUARD_OUT="$(bash "$BOOT/scripts/install.sh" --ref deadbeef 2>&1 || true)"
|
|
if echo "$GUARD_OUT" | grep -qi "not supported"; then
|
|
pass "9a install.sh rejects --ref with guidance to npx path"
|
|
else
|
|
echo "$GUARD_OUT" | tail -3
|
|
fail_ "9a install.sh rejects --ref"
|
|
fi
|
|
|
|
note "10. uninstall preserves user data"
|
|
mkdir -p "$HOME/.paperclip" && touch "$HOME/.paperclip/e2e-user-data-marker"
|
|
if shim uninstall; then
|
|
pass "10a uninstall exits 0"
|
|
else
|
|
fail_ "10a uninstall exits 0"
|
|
fi
|
|
[ ! -e "$SHIM" ] && pass "10b shim removed" || fail_ "10b shim removed"
|
|
[ ! -d "$STORE" ] && pass "10c managed store removed" || fail_ "10c managed store removed"
|
|
[ -f "$HOME/.paperclip/e2e-user-data-marker" ] && pass "10d user data under ~/.paperclip preserved" || fail_ "10d user data preserved"
|
|
|
|
note "RESULTS ($E2E_REPO@$E2E_REF on $(uname -sm))"
|
|
printf '%s\n' "${RESULTS[@]}"
|
|
if [ "$FAILED" = "1" ]; then echo; echo "OVERALL: FAIL"; exit 1; fi
|
|
echo; echo "OVERALL: PASS"
|