From fd61dc1dd538e36b88f01dddf34eaf8df89feeca Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Tue, 13 Oct 2026 07:02:37 -0500 Subject: [PATCH] docs(skills): distinguish Paperclip verification targets (#16145) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Verification must show which runtime and user journey passed. > - The existing skill centers on an empty instance launched from source. > - Seeded data, reused databases, published packages, and hosted stacks prove different outcomes. > - This pull request makes the target and starting conditions explicit. > - It routes deployment to private Cloud skills and keeps feature checks in Verify Paperclip. ## Linked Issues or Issue Description **Issue type** Missing documentation. **Where is the issue?** `.agents/skills/verify-paperclip/SKILL.md`. **What's wrong?** The skill does not distinguish source development, npm installation, staging, and production. It also needs to distinguish fresh onboarding from seeded or reused data and make platform scope explicit. **Suggested fix** Add target selection and focused references. Preserve the existing empty-instance helper. Name the private Cloud deployment skills as a separate prerequisite when deployment is requested. No duplicate documentation PR was found in the open-PR search. ## What Changed - Record artifact, environment, starting data, platform, and journey before testing. - Document dev servers, fresh instances, seeded/reused test drives, and cloned worktrees. - Separate exact-version canary and stable npm verification from source and Cloud evidence. - Document staging and production checks and route deployment to `deploy-paperclip-stack` or `deploy-cloud-harness` in the private Cloud repo. - Limit local ownership helpers to instances created by their launcher. ## Verification - Current-head GitHub CI: 55 passed, 2 skipped, no failures. Typecheck, tests, build, and the aggregate gate passed. Greptile: 5/5, all threads resolved. - Skill validator: passed. - Relative Markdown file links: passed across the changed skills and references (29 links in final validation). - `python3 -m unittest discover -s .agents/skills/verify-paperclip/tests -v`: 2 passed. - `node scripts/check-maps.mjs`: passed (structural coverage only). - `git diff --check`: passed. - `pnpm -r typecheck`, `pnpm test:run`, and `pnpm build`: attempted; blocked by missing workspace dependencies (`node_modules`, Node type definitions, and the checkout tsx entry point). These are not passing full-suite results. - Existing published-launcher test attempt was blocked by sandbox loopback permissions; it does not provide package-install evidence. - Reviewed commands against development docs, release workflow, and current Cloud runbooks. No live deployment, publication, or production mutation was performed for these documentation changes. ## Risks - Low runtime risk: only skill documentation changes. - Private deployment skills require access to the Cloud repo and must be available before that handoff works. The verifier reports the prerequisite when unavailable. - Workflow changes can make command guidance stale. The skills direct readers to maintained workflows and runbooks before use. ## Model Used OpenAI GPT-6 via Codex, with reasoning, file editing, shell execution, and GitHub CLI access. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (the focused ownership suite; full-suite limits are listed above) - [x] I have added or updated tests where applicable (no runtime change; existing tests retained) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .agents/skills/verify-paperclip/SKILL.md | 82 +++++++++++++++-- .../references/npm-install.md | 78 +++++++++++++++++ .../references/verification-targets.md | 87 +++++++++++++++++++ 3 files changed, 240 insertions(+), 7 deletions(-) create mode 100644 .agents/skills/verify-paperclip/references/npm-install.md create mode 100644 .agents/skills/verify-paperclip/references/verification-targets.md diff --git a/.agents/skills/verify-paperclip/SKILL.md b/.agents/skills/verify-paperclip/SKILL.md index b16467140a..9e10d7c2f9 100644 --- a/.agents/skills/verify-paperclip/SKILL.md +++ b/.agents/skills/verify-paperclip/SKILL.md @@ -1,6 +1,6 @@ --- name: verify-paperclip -description: Launch an isolated Paperclip instance, drive its real web UI or CLI using the existing feature-map recipes, capture evidence, and clean up owned processes. Use to verify user-facing Paperclip behavior. +description: Verify Paperclip from a source checkout, published npm install, or existing staging or production stack. Select the environment, data state, version, and platform; drive real UI or CLI journeys and capture evidence. Route deployment to the private Cloud skills. --- # Verify Paperclip @@ -15,7 +15,66 @@ recipe or access drift with evidence; map updates belong to [maintain-verification-skill](../maintain-verification-skill/SKILL.md). Read [development instructions](../../../doc/DEVELOPING.md) for current commands. -## Launch +## Select the verification target + +State what is being tested before starting. Use the requested target; for ordinary +feature development, default to the current checkout on the current host with +isolated data. Expand into cross-platform or release qualification only when +requested or needed to reproduce the problem. + +Record these independent dimensions together: + +- **Artifact:** source checkout (path, commit, dirty state), published npm package + (requested channel and resolved exact version), or deployed image/release. +- **Environment:** local, staging, or production; target URL and intended company + or workspace; board/user/agent identity and authentication mode. +- **Starting data:** fresh and unonboarded, freshly seeded, cloned, or reused; + embedded versus external PostgreSQL; configuration/data location or a redacted + database identifier. Never record credentials or connection strings. +- **Platform:** server OS/architecture, Node/npm versions, and browser/client OS + when relevant. A Mac browser against a Linux server is Linux server evidence; + WSL is Linux evidence, not native Windows evidence. +- **Journey:** first installation/onboarding, an existing-user feature, upgrade, + or a live agent run; expected outcome and real versus simulated dependencies. + +| Target | Use it to prove | Procedure | +| --- | --- | --- | +| Checkout dev server, existing or cloned data | Source behavior against existing state | [Checkout modes](references/verification-targets.md#checkout-and-data-modes) | +| Checkout with fresh empty data | First-run onboarding and subsequent features | [Launch](#launch-an-empty-checkout-instance) below | +| Checkout test drive, fresh seed or reused data | Features with a configured company/CEO | [Checkout modes](references/verification-targets.md#checkout-and-data-modes) | +| Published npm canary | End-user installation of a specific prerelease | [npm verification](references/npm-install.md) | +| Published npm stable (`latest`) | End-user installation of the released package | [npm verification](references/npm-install.md) | +| Existing staging stack | Deployed build, hosted auth, flags, integrations | [Hosted verification](references/verification-targets.md#staging-and-production) | +| Existing production stack | Current production journey within the authorized scope | [Hosted verification](references/verification-targets.md#staging-and-production) | + +Fresh data does not imply a fresh install. A seeded test drive does not prove +onboarding. A source build does not prove npm packaging. A passing staging check +does not prove production, and a production Cloud stack does not prove the stable +npm install. Report only the combinations actually exercised. + +## Deployment handoff + +When a requested revision must first be built or deployed, use the skills in the +private `paperclip-cloud` repo: + +- `deploy-paperclip-stack` at `.agents/skills/deploy-paperclip-stack/SKILL.md` + for a tenant workspace or an explicitly scoped fleet rollout. +- `deploy-cloud-harness` at `.agents/skills/deploy-cloud-harness/SKILL.md` + for the Cloud control plane itself. + +Locate the authorized Cloud checkout or installed skill by name; do not assume a +sibling directory exists. Those skills own build preparation, deployment, +migrations, and rollout recovery. Return here with environment, target, resolved +source SHA, release/image identity, deployment result, and evidence link. If +private access is unavailable, report that prerequisite as blocked and continue +checks possible on the supplied target without inventing deploy commands. + +A verification request alone does not authorize deployment, npm publication, +fleet-default changes, or a production rollout. Preserve explicit deployment +instructions already given. Publishing npm packages belongs to the existing +[release skill](../release/SKILL.md); installing one does not require publishing it. + +## Launch an empty checkout instance Run from the repository root with Node 24.11+, Python 3, `lsof`, and dependencies installed with `pnpm install --frozen-lockfile`. Browser driving uses the existing @@ -53,7 +112,11 @@ For Paperclip-managed reusable services, follow instead of launching a second server. Its cloned-data/login requirements differ from this empty local instance. -## Doctor +## Doctor (owned checkout helper only) + +The doctor/stop commands require the real `launch.json` created by `launch.py`. +Do not fabricate one for dev servers, npm installs, test drives, or hosted stacks. +Use their own readiness and lifecycle procedures instead. After startup, and again after any surprising behavior, run: @@ -98,14 +161,16 @@ guessing resource URLs. The unprefixed `/onboarding` route starts naming; use th company-prefixed onboarding route to resume its agent setup. Check relevant experimental gates and actor permissions. -For CLI checks, discover exact flags with +For source-checkout CLI checks, discover exact flags with `node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts --help`, then -select the disposable data directory/API/company explicitly. API reads can +select the disposable data directory/API/company explicitly. For published npm +checks, use the installed version’s CLI and help instead. API reads can corroborate UI writes; API-created state does not prove a creation UI. ## Evidence -Record feature file, sub-feature and entry-point ID, commit/dirty state, owned URL, +Include the target dimensions above, then record feature file, sub-feature and +entry-point ID, commit/dirty state, target URL, company/user, flags, runtime/harness, live versus simulated dependencies, command, exit code, expected and observed behavior, and evidence links. Report each entry as passed, failed, blocked with its prerequisite, or not run. @@ -126,7 +191,10 @@ documentation coverage rather than live test health. ## Cleanup -Close the browser context you created (the smoke does this in `finally`). Stop the owned runtime with: +Close the browser context you created (the smoke does this in `finally`). Leave +shared dev, staging, and production services running. The remaining cleanup steps +in this section apply only to `launch.py` instances; use the selected target’s +procedure for other launchers. Stop the helper-owned runtime with: ```sh python3 .agents/skills/verify-paperclip/scripts/stop.py \ diff --git a/.agents/skills/verify-paperclip/references/npm-install.md b/.agents/skills/verify-paperclip/references/npm-install.md new file mode 100644 index 0000000000..fe4491948d --- /dev/null +++ b/.agents/skills/verify-paperclip/references/npm-install.md @@ -0,0 +1,78 @@ +# Verify the published npm experience + +Use this route for “can a user install and run this package?” Source startup, +workspace links, `pnpm build`, and local tarballs do not answer that question. + +## Resolve the exact artifact + +Read [.github/workflows/release.yml](../../../../.github/workflows/release.yml) +for publication behavior. Canary publication follows verified pushes to `master`; +an arbitrary feature-branch SHA need not have a published canary. Publication can +fail or still be running, so prove registry availability before claiming coverage. + +Resolve the requested channel once against the public registry: + +```sh +npm view paperclipai@canary version --registry=https://registry.npmjs.org +npm view paperclipai@latest version --registry=https://registry.npmjs.org +``` + +Use the one requested by the task: `canary` for prerelease verification, `latest` +for stable production npm verification, or an explicit version. Record resolved +version, registry, dist integrity, and source provenance from release metadata, +workflow, and tags. Pin that version for all later commands. If testing a SHA, +confirm the package maps to it; never substitute the current canary silently. +Missing publication is a blocked check, not a reason to publish during verification. + +## Install like a new user + +Use a disposable directory outside the checkout and fresh npm cache, +instance/config, and user-config paths. Use an isolated OS user/container when +proving absence of all pre-existing user state. Inspect inherited routing and +registry overrides without exposing secrets. Avoid workspace dependencies, local +`file:` packages, preinstalled Paperclip binaries, and cached `npx` installs. + +Run the documented end-user entry point with the pinned package, for example +`npx --yes --registry=https://registry.npmjs.org paperclipai@`. +Read that version's `--help` before selecting `onboard`, `install`, or service +flags. For a foreground disposable check use `onboard --yes --bind loopback +--no-install-service --run --data-dir ` when supported by that +version; keep ordinary interactive onboarding when that is the journey requested. +A managed CLI/service installation is separate from an ephemeral `npx` launch; +test it explicitly if requested. + +Prove each relevant stage separately: registry download/install, CLI execution, +server readiness, visible onboarding, saved company/setup state, restart/reload, +and a first agent task when authorized. Successful `--version` alone does not +prove a usable install. For upgrades, first create state with the previous exact +version, then install the target against that same disposable data and verify +state survives; report this separately from a fresh install. + +## Existing automation and its limits + +The repo has a published-package onboarding suite: + +```sh +PAPERCLIPAI_VERSION= pnpm test:canary-onboarding-smoke +``` + +Read [its configuration](../../../../tests/canary-onboarding/playwright.config.ts) +and [launcher](../../../../tests/canary-onboarding/start-published-canary.mjs) +before running. It installs the published package into isolated temporary state, +uses a fresh npm cache and loopback server, and retains failure diagnostics. +It uses checkout Playwright tooling to drive a published runtime; record both +revisions. Its name and automated release wiring target canaries. For stable +verification, inspect compatibility with the selected version or use the manual +pinned-package path above; do not relabel a canary result as stable proof. + +The Docker release-smoke suite is separate image evidence. Likewise, +`scripts/clean-install-npm.sh` defaults to source, and its published mode uses a +moving version and source-driven uninstall; it does not by itself prove the +whole lifecycle of a specific published version. + +Capture OS/architecture, Node/npm versions, exact commands and exit codes, +installed version, serving revision when available, initial data state, screenshots, +logs, and observed outcomes. Redact credentials. Stop owned foreground processes +or uninstall an owned test service with the tested version's supported command; +retain evidence outside disposable data. Never run checkout ownership helpers +with invented metadata to stop a package process. diff --git a/.agents/skills/verify-paperclip/references/verification-targets.md b/.agents/skills/verify-paperclip/references/verification-targets.md new file mode 100644 index 0000000000..837f3803a1 --- /dev/null +++ b/.agents/skills/verify-paperclip/references/verification-targets.md @@ -0,0 +1,87 @@ +# Checkout, data, and hosted targets + +## Checkout and data modes + +Read [DEVELOPING.md](../../../../doc/DEVELOPING.md) and +[DATABASE.md](../../../../doc/DATABASE.md) for the current startup contract. + +- **Existing dev server:** use `pnpm dev` (watch) or `pnpm dev:once` from the + intended checkout. Inspect its instance routing and data configuration first; + do not assume the default port belongs to this checkout. Record the actual + served commit and dirty state. Reuse a matching running server when appropriate. +- **Fresh empty instance:** use the skill's `launch.py` helper with a new data + directory. Complete the UI or CLI onboarding journey under test. This is the + credential-free route; a live model run needs separate provider setup. +- **Seeded test drive:** run the documented CLI `test-drive --no-browser` with + an authorized environment-backed provider credential. It creates a temporary + embedded instance, completes setup, and seeds Test Company and a CEO. It does + not execute the first task. Verify that separately when required. +- **Reused test drive:** pass `--data-dir `. If it already + contains a company, bootstrap flags do not replace companies, agents, or + secrets. Record existing state; do not label the result fresh onboarding. + Test drive rejects external PostgreSQL configuration and database URL overrides. +- **Managed or cloned worktree:** use + [dev-workspace run/verify/fix](../../paperclip-dev-workspace-run-verify-fix/SKILL.md). + Verify seed completion, instance routing, and login handoff before driving it. + Cloned data is existing-user evidence even when the checkout is new. + +`--data-dir` alone is not proof of a fresh database: check effective config and +whether `DATABASE_URL` routes to an external database. Do not print its value. +For ordinary dev, inspect both environment overrides and the active database +config: with `DATABASE_URL` unset, `database.mode: postgres` and a saved +`connectionString` still select external PostgreSQL. Embedded PostgreSQL is the +default only when neither an environment override nor the active config selects +an external database; its instance directory may already hold data. Do not reset +shared data to make a test pass. Choose separate disposable data for first-run +checks and verify the effective backend before calling it isolated. + +Record runtime PID, URL, port owner, checkout, and data location for a process you +start. Check `/api/health`, expected served revision, bootstrap/auth readiness, +and the actual page. Capture starting state and verify persistence after reload. +Use that launcher's documented stop procedure for an owned process; leave a +server you borrowed running. + +## Platform scope + +Ordinary development covers the current host unless the request or failure calls +for more. For a platform issue or release matrix, list required combinations: +macOS, Linux, native Windows, architecture, runtime, and install path. Mark each +combination passed, failed, blocked, or not run separately. + +The local helper needs POSIX tools such as `lsof`; do not assume it qualifies +native Windows. Use the supported CLI and native shell for that platform, with +isolated instance/config/cache paths and platform-appropriate process cleanup. +A container proves its guest OS and selected architecture, not the host's native +installer, service manager, file permissions, or path behavior. Keep browser and +server platforms distinct when testing a remote Linux box from a Mac. + +## Staging and production + +Use the supplied tenant URL and intended identity. Determine whether the target +is the Paperclip tenant application or the Cloud control plane. Confirm the +environment from the operator's deployment record and authenticated routing; +a page title or local Git checkout is not evidence of the serving revision. + +If a build/deploy is needed, follow the private skill handoff in `SKILL.md` first. +Record the exact serving SHA and release/image identity after deployment, not +just the requested branch, CI result, or fleet default. Recheck after a long run: +a staging auto-roll or another deployment can change the serving version. Mark +unknown or changing provenance as inconclusive for a claim about a specific SHA. + +For staging, use a designated test workspace and its real hosted sign-in/entry +flow. Record fresh versus existing workspace state, experimental flags, relevant +provider connections, and whether dependencies are live. Check tenant readiness +and then the feature journey, including persistence or continuation. A control +plane `/healthz` response or tenant HTML 200 is not feature acceptance. + +For production, keep verification within the requested stack, identity, and +journey. Prefer observation of existing state unless writes are authorized; use +an agreed test workspace for mutating checks. Do not run the empty-instance +onboarding smoke, reset data, change flags, or trigger paid agent work merely to +make a production check pass. If a necessary write is outside the request, +report that missing authorization and complete the read-only checks. + +Never use local ownership helpers against a remote server. Close only browser +sessions you created and clean up only test records whose removal is authorized. +Keep shared services running. Report production Cloud verification separately +from stable npm verification; the shipped artifacts and configuration differ.