4 Commits

Include renames

Author	SHA1	Message	Date
3 people	fc5a30805e	feat(cli): add managed install, update, and service lifecycle (#10045) ... ## Thinking Path > - Paperclip is the open source control plane people use to manage AI-agent companies > - Operators need a predictable installation path that survives beyond an ephemeral `npx` process > - A durable installation needs an owned per-user payload store, stable command shim, safe shell integration, and supported service lifecycle > - Updates must preserve recoverability by backing up data, installing side-by-side, verifying the new payload, and retaining rollback state > - Bootstrap scripts and privileged service operations must fail closed across download, filesystem, ownership, and consent boundaries > - This pull request integrates managed install, update, rollback, service, uninstall, doctor, bootstrap-installer, and runtime-serving support into one workflow > - The benefit is a recoverable, inspectable, and documented installation lifecycle with explicit safety boundaries across Linux, macOS, containers, WSL, npm, npx, and source checkouts ## Linked Issues or Issue Description ### Problem Paperclip lacks a first-class durable installation and lifecycle workflow. Operators currently have to assemble npm/npx installation, PATH setup, background-service management, updates, rollback, diagnostics, and uninstall behavior themselves. That makes upgrades harder to recover, creates inconsistent behavior across platforms, and leaves shell/download/service trust boundaries without one documented implementation. ### Proposed Solution Add a managed per-user install store and stable shim, a verified shell bootstrap installer, service lifecycle commands, install-mode-aware update/rollback behavior, doctor checks, and documentation. Managed updates back up the database, install and smoke-test a side-by-side payload, atomically switch `current`, and retain prior payloads. The shell installer pins registry/download trust boundaries and requires explicit consent for non-interactive privileged actions. ### Alternatives Considered - Keep recommending `npx`: simple for evaluation, but ephemeral and unsuitable for stable services, atomic updates, or rollback. - Require global npm installation only: familiar, but cannot provide the owned side-by-side payload store and retained rollback semantics. - Split the capability across multiple PRs: rejected because install, update, service, uninstall, bootstrap, and serving behavior share contracts and security boundaries that need review together. ### Related Pull Requests - Supersedes #10042 and #10044 with one integrated final diff. - Incorporates and replaces the closed preparatory work in #10032 and #10034. ## What Changed - Added `paperclipai install`, `update`/`upgrade`, rollback, uninstall, service lifecycle, onboarding integration, and managed-install doctor checks. - Added a private managed payload store, verified manifest/marker ownership, exclusive mutation locks, atomic manifest/current/shim writes, retained previous payloads, and provenance validation. - Added npm and GitHub-ref install sources with exact target resolution, registry isolation, database backup, side-by-side verification, atomic activation, service restart coordination, and failure rollback. - Made managed-update backups report actionable service-start and `--no-backup` recovery guidance for unreachable databases, while clean never-onboarded instances skip an empty backup. - Added systemd user and launchd service definitions, status/health/log commands, single-instance coordination, stale-port recovery, and explicit sudo/lingering consent handling. - Added the `scripts/install.sh` bootstrap path with checked two-stage downloads, pinned public npm registry usage, platform checks, dry-run/non-interactive controls, and Docker fixtures. - Added embedded Postgres/native bootstrap integration, hot-restart/systemd-notify serving support, passive update notices, configuration contracts, README/CLI/install documentation, and focused regression tests. - Security re-review should explicitly re-verify: (1) `addManagedPathBlock`/`removeManagedPathBlock` reject symlinked or non-regular rc files, assert current-user ownership, preserve restrictive modes, and replace atomically; (2) managed shim replacement rejects unsafe parents, foreign-owned or multiply linked files, and uses checked atomic replacement; (3) the shell installer and sudo path preserve explicit consent and checked downloads; and (4) installed service/runtime serving remains bound to the validated managed shim and instance configuration. ## Verification - `bash -n scripts/install.sh scripts/clean-install-git.sh scripts/clean-install-npm.sh scripts/test-install-sh-docker.sh` - `pnpm exec vitest run cli/src/__tests__/install-store.test.ts cli/src/__tests__/install-command.test.ts cli/src/__tests__/managed-install-check.test.ts cli/src/__tests__/onboard-service.test.ts cli/src/__tests__/service-health-check.test.ts cli/src/__tests__/service-manager.test.ts cli/src/__tests__/update-command.test.ts cli/src/__tests__/update-notice.test.ts packages/db/src/embedded-postgres-native.test.ts` — 9 files, 66 tests passed - `pnpm --dir cli typecheck` - `pnpm --dir cli build` - Follow-up verification: `pnpm exec vitest run cli/src/__tests__/update-command.test.ts` (14/14), `pnpm --dir cli typecheck`, `pnpm --dir cli build`, and `pnpm --filter @paperclipai/server typecheck`. - `pnpm -r typecheck` - `pnpm build` - Full `pnpm test:run` exercised all suites; an injected static AWS credential changed one unrelated doctor expectation, which passed when those credentials were removed. A second run cleared that case and exposed stale pre-existing adapter-utils `dist` output; rebuilding `@paperclipai/adapter-utils` made the isolated test pass. The updated PR CI is the authoritative clean-workspace full-suite run. ## Risks - Installer/update code writes executable shims, symlinks, shell rc blocks, service definitions, and managed payloads; ownership, regular-file, symlink, hard-link, marker, and path-containment checks fail closed before destructive changes. - The bootstrap installer executes downloaded tooling; downloads are staged and checked before execution, npm traffic is pinned to the public registry, and non-interactive privileged behavior requires explicit consent. - Linux lingering may invoke `sudo`; the command is surfaced and confirmed before execution, and unsupported service managers fall back to foreground-run guidance. - Database migrations remain forward-only; payload rollback does not reverse migrations, so managed updates create a backup before activation unless explicitly disabled. - Service restart and runtime serving touch process/port ownership; lifecycle locks, health/version checks, and stable-shim service definitions reduce split-brain and stale-process risk. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex coding agents using GPT-5.5 and GPT-5.6-sol, with reasoning, repository/API access, shell execution, and test tooling. The runtime did not expose a reliable context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-31 18:52:23 -07:00
DottaandPaperclip	fd2f82ac5b	[codex] Add built-in Hermes adapters (#8543) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agent adapters are the boundary between the control plane and the runtimes that actually do work. > - Hermes support needs to be available as first-class local and gateway adapters while still preserving the adapter-manager override path for external packages. > - The adapter work touches runtime execution, UI adapter metadata, onboarding prompts, scoped credentials, release packaging, and smoke coverage, so the handoff needs concrete verification rather than only unit tests. > - This pull request adds built-in Hermes local and Hermes gateway support, keeps external adapter overrides compatible, and documents/tests the gateway flow end to end. > - The benefit is that operators can hire Hermes-backed agents without a manual plugin install, while self-hosted installs can still override/shadow the built-ins through Adapter manager packages. ## Linked Issues or Issue Description No public GitHub issue exists for this exact Hermes built-in adapter, gateway onboarding, and release-source work. Problem description: - Hermes local and gateway adapters need a public, reviewable source path in the monorepo so package artifacts and built-in adapter behavior match the application source. - Operators need built-in `hermes_local` and `hermes_gateway` adapter choices without losing the ability to install external Hermes packages as overrides. - Gateway onboarding needs secure defaults for API server URLs, API keys, and generated agent setup text. - Hermes-originated task bridge credentials need narrower API-key scope configuration. - Related public PRs found during duplicate search include #3027, #2363, #7544, #7950, #8095, and #8543. ## What Changed - Added the unified Hermes adapter package with local and gateway server/UI/CLI exports, config schemas, transcript parsing, model detection, and package metadata. - Registered `hermes_local` and `hermes_gateway` as built-in adapters across shared constants, server registries, CLI packaging, and UI adapter registries. - Kept the external adapter override path compatible so installed Hermes packages can shadow built-ins and restore the built-in parser when disabled. - Added Hermes gateway onboarding docs, board-operator docs, Docker smoke assets, and shell smoke harnesses for join/e2e validation. - Added scoped task-bridge API-key support, authorization checks, issue-origin handling, and tests for Hermes-created Paperclip tasks. - Hardened gateway transport and redaction behavior for API keys, headers, session data, and smoke diagnostics. - Updated release packaging/bootstrap checks for the Hermes packages while leaving `pnpm-lock.yaml` out of the PR per repository policy. ## Verification Targeted local verification recorded before PR handoff: - `pnpm --filter @paperclipai/hermes-paperclip-adapter exec vitest run src/gateway/server/execute.test.ts` — 14/14 passed. - `pnpm test:hermes-gateway-smoke` — 6/6 passed. - Hermes package typecheck/build checks passed. - Focused server/UI adapter tests passed — 31/31. - Release helper Node tests passed — 18/18. - `git diff --check origin/master..HEAD` passed. Fresh Docker E2E smoke evidence: - Ran `pnpm smoke:hermes-gateway-e2e` on 2026-06-26 with a fresh state directory and fresh Docker container against a live Paperclip dev server. - Hermes direct execution reached `completed`. - Hermes stop/cancel path reached `cancelled`. - Hermes gateway created a Paperclip task, Paperclip ran the Hermes agent, and the task reached `done` with the expected marker response. - Temporary board auth keys, token files, smoke state, and Docker containers were cleaned up after the run. PR checks on head `b5eae40ce`: - GitHub Actions passed: `policy`, `review`, `Typecheck + Release Registry`, all general test shards, all serialized server shards, `Build`, `Canary Dry Run`, `e2e`, and aggregate `verify`. - External checks passed: Snyk and Socket Project Report. - External Socket Pull Request Alerts remained pending after the first-party CI matrix completed. ## Risks - Medium risk: this spans adapter registration, package publishing, gateway execution, onboarding docs, API-key scoping, and UI adapter metadata. - Migration risk is low: the scope-config migration adds a nullable column and does not rewrite existing keys. - Gateway execution depends on operator-provided Hermes API configuration; the smoke covers the Docker gateway path but real deployments may differ by network/auth setup. - Direct Greptile review on the latest expanded diff is file-count limited, although the commitperclip review gate passed. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5 coding agent, tool use enabled in a local repository workspace. Context window size is not exposed in this environment. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Commitperclip review gate is green; direct Greptile review is file-count limited on the latest expanded diff - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-06-26 16:04:58 -05:00
Devin Foley	4518f272b2	fix(release): publish all remaining @paperclipai workspace packages from CI + guard unpublishable edges (#8365) ... ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work, distributed primarily via the `paperclipai` npm package and its `@paperclipai/*` workspace packages. > - The release subsystem (`scripts/release-package-*`) decides which workspace packages CI republishes at the unified calver version each release, and `replaceWorkspaceDeps()` rewrites every internal `workspace:` dependency to that calver version at publish time. > - The gap: a package that publishes from CI can declare a `workspace:` dependency on a package that is NOT enrolled for CI publish. The dependency spec gets rewritten to a calver version that is never actually published, so the dependent becomes uninstallable. > - This shipped for real: a recent change made `@paperclipai/server` depend on `@paperclipai/skills-catalog`, but skills-catalog was not on the calver release train — so canary builds after that merge failed to resolve `@paperclipai/skills-catalog@<calver>` and `npx paperclipai@canary run` broke. > - This PR addresses it durably by (a) putting every remaining internal package on the CI release train so no internal dependency can dangle, and (b) adding a fail-fast guard so this class of break can never ship again. > - The benefit is that canary and stable installs resolve all internal dependencies, and any future unpublishable workspace edge fails the release build with a clear, named error instead of producing a broken package. ## Linked Issues or Issue Description Refs #8327 (introduced the `server -> skills-catalog` runtime dependency that surfaced the gap). No public issue tracks this; describing it in-PR: - **Problem:** After #8327, `npx paperclipai@canary run` failed for builds past the merge because `@paperclipai/skills-catalog` was rewritten to a calver version that was never published (the package was not enrolled for CI publish). Versions before the merge still ran. - **Root cause:** A `publishFromCi:true` package can declare a `workspace:` dependency on a package that is not `publishFromCi:true`; the release-time version rewrite then points at a non-existent published version. ## What Changed - Enrolled every remaining internal package on the calver release train by setting `publishFromCi: true` in `scripts/release-package-manifest.json`: `skills-catalog`, `teams-catalog`, `plugin-workspace-diff`, `plugin-kubernetes`, `plugin-novita-sandbox`. There are now zero `publishFromCi:false` entries. - `skills-catalog`, `teams-catalog`, `plugin-workspace-diff` already existed on npm — CI simply republishes them at calver. - `plugin-kubernetes` and `plugin-novita-sandbox` were not on npm; their one-time first publish was bootstrapped so the `check-release-package-bootstrap` gate passes. - Added `findUnpublishableWorkspaceEdges()` to `scripts/release-package-map.mjs`, wired into `buildReleasePackagePlan()`. The release map build now fails fast (surfaced by the `check` CI already runs) whenever a `publishFromCi:true` package declares a runtime `workspace:` dependency (`dependencies`/`optionalDependencies`/`peerDependencies`) on a non-`publishFromCi:true` `@paperclipai/*` package, naming the offending edge. - Added tests covering positive/negative detection, all three dependency sections, unknown-package edges, off-train edges, and the live manifest. ## Verification - `node --test scripts/release-package-map.test.mjs` → 9/9 pass (includes a test asserting the live manifest has no unpublishable edges). - `node scripts/check-release-package-bootstrap.mjs scripts/release-package-manifest.json` → passes, naming all five newly-enabled packages (all confirmed present on npm). - Confirmed via `npm view` that all five packages resolve on the public registry. ## Risks - Low risk. The change only enrolls already-existing (or freshly-bootstrapped) packages onto the existing release train and adds a build-time validation. No runtime code paths change. The new guard can only *fail* a release that was already going to ship a broken package. ## Model Used Claude Opus 4.7 (claude-opus-4-7), extended thinking, with tool use / code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots (N/A — no UI changes) - [x] I have updated relevant documentation to reflect my changes (N/A — no doc-facing behavior change) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge	2026-06-19 19:11:20 -07:00
Devin Foley	29401b231b	fix(ci): gate new release packages on npm bootstrap (#5146) ... ## Thinking Path > - Paperclip is a control plane for autonomous agent companies, so its release automation is part of the core operator trust boundary. > - The affected subsystem is npm/GitHub Actions release publishing for the public monorepo packages. > - The concrete failure was that a newly added package reached `master`, the canary workflow attempted its first publish, and npm trusted publishing was not yet bootstrapped for that package. > - That means the problem is not just one broken run; it is a missing pre-merge guard that lets release-ineligible packages land and only fail once `publish_canary` runs. > - This pull request makes release enrollment explicit, validates that enrollment in CI, and adds a PR-time bootstrap check against npm for changed release-enabled package manifests. > - The result is that we keep trusted publishing, avoid teaching CI to `npm adduser`, and move this class of failure from post-merge canary time to pre-merge review time. ## What Changed - Added `scripts/release-package-manifest.json` so release-managed public packages are explicitly enrolled instead of being inferred from every non-private workspace package. - Hardened `scripts/release-package-map.mjs` to validate the manifest before release workflows rewrite versions or assemble publish payloads. - Added `scripts/check-release-package-bootstrap.mjs` and wired it into `.github/workflows/pr.yml` so PRs that change a release-enabled package manifest fail if that package does not already exist on npm. - Added release-package manifest coverage tests to `scripts/release-package-map.test.mjs` and included them in `pnpm run test:release-registry`. - Wired manifest validation into `.github/workflows/release.yml` and documented the first-publish bootstrap policy in `doc/PUBLISHING.md` and `doc/RELEASE-AUTOMATION-SETUP.md`. ## Verification - `pnpm run test:release-registry` - `./scripts/release.sh canary --skip-verify --dry-run` - Confirmed the committed diff contains no obvious PII/secrets via targeted pattern scan before pushing. ## Risks - Low risk overall: this is CI/release-policy code, not product runtime logic. - The new PR bootstrap check depends on npm metadata availability, so a transient npm outage could block a PR that changes a release-enabled package manifest. - The manifest introduces a new source of truth that must stay aligned with public package additions, but that is intentional and now enforced. ## Model Used - OpenAI Codex via the `codex_local` Paperclip adapter; GPT-5-based coding agent with tool use, terminal execution, git, and GitHub CLI. Exact served model ID/context window are not exposed by the local runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] I will address all Greptile and reviewer comments before requesting merge	2026-05-03 19:31:28 -07:00