diff --git a/.agents/skills/verify-paperclip/SKILL.md b/.agents/skills/verify-paperclip/SKILL.md index b16467140a..9e10d7c2f9 100644 --- a/.agents/skills/verify-paperclip/SKILL.md +++ b/.agents/skills/verify-paperclip/SKILL.md @@ -1,6 +1,6 @@ --- name: verify-paperclip -description: Launch an isolated Paperclip instance, drive its real web UI or CLI using the existing feature-map recipes, capture evidence, and clean up owned processes. Use to verify user-facing Paperclip behavior. +description: Verify Paperclip from a source checkout, published npm install, or existing staging or production stack. Select the environment, data state, version, and platform; drive real UI or CLI journeys and capture evidence. Route deployment to the private Cloud skills. --- # Verify Paperclip @@ -15,7 +15,66 @@ recipe or access drift with evidence; map updates belong to [maintain-verification-skill](../maintain-verification-skill/SKILL.md). Read [development instructions](../../../doc/DEVELOPING.md) for current commands. -## Launch +## Select the verification target + +State what is being tested before starting. Use the requested target; for ordinary +feature development, default to the current checkout on the current host with +isolated data. Expand into cross-platform or release qualification only when +requested or needed to reproduce the problem. + +Record these independent dimensions together: + +- **Artifact:** source checkout (path, commit, dirty state), published npm package + (requested channel and resolved exact version), or deployed image/release. +- **Environment:** local, staging, or production; target URL and intended company + or workspace; board/user/agent identity and authentication mode. +- **Starting data:** fresh and unonboarded, freshly seeded, cloned, or reused; + embedded versus external PostgreSQL; configuration/data location or a redacted + database identifier. Never record credentials or connection strings. +- **Platform:** server OS/architecture, Node/npm versions, and browser/client OS + when relevant. A Mac browser against a Linux server is Linux server evidence; + WSL is Linux evidence, not native Windows evidence. +- **Journey:** first installation/onboarding, an existing-user feature, upgrade, + or a live agent run; expected outcome and real versus simulated dependencies. + +| Target | Use it to prove | Procedure | +| --- | --- | --- | +| Checkout dev server, existing or cloned data | Source behavior against existing state | [Checkout modes](references/verification-targets.md#checkout-and-data-modes) | +| Checkout with fresh empty data | First-run onboarding and subsequent features | [Launch](#launch-an-empty-checkout-instance) below | +| Checkout test drive, fresh seed or reused data | Features with a configured company/CEO | [Checkout modes](references/verification-targets.md#checkout-and-data-modes) | +| Published npm canary | End-user installation of a specific prerelease | [npm verification](references/npm-install.md) | +| Published npm stable (`latest`) | End-user installation of the released package | [npm verification](references/npm-install.md) | +| Existing staging stack | Deployed build, hosted auth, flags, integrations | [Hosted verification](references/verification-targets.md#staging-and-production) | +| Existing production stack | Current production journey within the authorized scope | [Hosted verification](references/verification-targets.md#staging-and-production) | + +Fresh data does not imply a fresh install. A seeded test drive does not prove +onboarding. A source build does not prove npm packaging. A passing staging check +does not prove production, and a production Cloud stack does not prove the stable +npm install. Report only the combinations actually exercised. + +## Deployment handoff + +When a requested revision must first be built or deployed, use the skills in the +private `paperclip-cloud` repo: + +- `deploy-paperclip-stack` at `.agents/skills/deploy-paperclip-stack/SKILL.md` + for a tenant workspace or an explicitly scoped fleet rollout. +- `deploy-cloud-harness` at `.agents/skills/deploy-cloud-harness/SKILL.md` + for the Cloud control plane itself. + +Locate the authorized Cloud checkout or installed skill by name; do not assume a +sibling directory exists. Those skills own build preparation, deployment, +migrations, and rollout recovery. Return here with environment, target, resolved +source SHA, release/image identity, deployment result, and evidence link. If +private access is unavailable, report that prerequisite as blocked and continue +checks possible on the supplied target without inventing deploy commands. + +A verification request alone does not authorize deployment, npm publication, +fleet-default changes, or a production rollout. Preserve explicit deployment +instructions already given. Publishing npm packages belongs to the existing +[release skill](../release/SKILL.md); installing one does not require publishing it. + +## Launch an empty checkout instance Run from the repository root with Node 24.11+, Python 3, `lsof`, and dependencies installed with `pnpm install --frozen-lockfile`. Browser driving uses the existing @@ -53,7 +112,11 @@ For Paperclip-managed reusable services, follow instead of launching a second server. Its cloned-data/login requirements differ from this empty local instance. -## Doctor +## Doctor (owned checkout helper only) + +The doctor/stop commands require the real `launch.json` created by `launch.py`. +Do not fabricate one for dev servers, npm installs, test drives, or hosted stacks. +Use their own readiness and lifecycle procedures instead. After startup, and again after any surprising behavior, run: @@ -98,14 +161,16 @@ guessing resource URLs. The unprefixed `/onboarding` route starts naming; use th company-prefixed onboarding route to resume its agent setup. Check relevant experimental gates and actor permissions. -For CLI checks, discover exact flags with +For source-checkout CLI checks, discover exact flags with `node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts --help`, then -select the disposable data directory/API/company explicitly. API reads can +select the disposable data directory/API/company explicitly. For published npm +checks, use the installed version’s CLI and help instead. API reads can corroborate UI writes; API-created state does not prove a creation UI. ## Evidence -Record feature file, sub-feature and entry-point ID, commit/dirty state, owned URL, +Include the target dimensions above, then record feature file, sub-feature and +entry-point ID, commit/dirty state, target URL, company/user, flags, runtime/harness, live versus simulated dependencies, command, exit code, expected and observed behavior, and evidence links. Report each entry as passed, failed, blocked with its prerequisite, or not run. @@ -126,7 +191,10 @@ documentation coverage rather than live test health. ## Cleanup -Close the browser context you created (the smoke does this in `finally`). Stop the owned runtime with: +Close the browser context you created (the smoke does this in `finally`). Leave +shared dev, staging, and production services running. The remaining cleanup steps +in this section apply only to `launch.py` instances; use the selected target’s +procedure for other launchers. Stop the helper-owned runtime with: ```sh python3 .agents/skills/verify-paperclip/scripts/stop.py \ diff --git a/.agents/skills/verify-paperclip/references/npm-install.md b/.agents/skills/verify-paperclip/references/npm-install.md new file mode 100644 index 0000000000..fe4491948d --- /dev/null +++ b/.agents/skills/verify-paperclip/references/npm-install.md @@ -0,0 +1,78 @@ +# Verify the published npm experience + +Use this route for “can a user install and run this package?” Source startup, +workspace links, `pnpm build`, and local tarballs do not answer that question. + +## Resolve the exact artifact + +Read [.github/workflows/release.yml](../../../../.github/workflows/release.yml) +for publication behavior. Canary publication follows verified pushes to `master`; +an arbitrary feature-branch SHA need not have a published canary. Publication can +fail or still be running, so prove registry availability before claiming coverage. + +Resolve the requested channel once against the public registry: + +```sh +npm view paperclipai@canary version --registry=https://registry.npmjs.org +npm view paperclipai@latest version --registry=https://registry.npmjs.org +``` + +Use the one requested by the task: `canary` for prerelease verification, `latest` +for stable production npm verification, or an explicit version. Record resolved +version, registry, dist integrity, and source provenance from release metadata, +workflow, and tags. Pin that version for all later commands. If testing a SHA, +confirm the package maps to it; never substitute the current canary silently. +Missing publication is a blocked check, not a reason to publish during verification. + +## Install like a new user + +Use a disposable directory outside the checkout and fresh npm cache, +instance/config, and user-config paths. Use an isolated OS user/container when +proving absence of all pre-existing user state. Inspect inherited routing and +registry overrides without exposing secrets. Avoid workspace dependencies, local +`file:` packages, preinstalled Paperclip binaries, and cached `npx` installs. + +Run the documented end-user entry point with the pinned package, for example +`npx --yes --registry=https://registry.npmjs.org paperclipai@`. +Read that version's `--help` before selecting `onboard`, `install`, or service +flags. For a foreground disposable check use `onboard --yes --bind loopback +--no-install-service --run --data-dir ` when supported by that +version; keep ordinary interactive onboarding when that is the journey requested. +A managed CLI/service installation is separate from an ephemeral `npx` launch; +test it explicitly if requested. + +Prove each relevant stage separately: registry download/install, CLI execution, +server readiness, visible onboarding, saved company/setup state, restart/reload, +and a first agent task when authorized. Successful `--version` alone does not +prove a usable install. For upgrades, first create state with the previous exact +version, then install the target against that same disposable data and verify +state survives; report this separately from a fresh install. + +## Existing automation and its limits + +The repo has a published-package onboarding suite: + +```sh +PAPERCLIPAI_VERSION= pnpm test:canary-onboarding-smoke +``` + +Read [its configuration](../../../../tests/canary-onboarding/playwright.config.ts) +and [launcher](../../../../tests/canary-onboarding/start-published-canary.mjs) +before running. It installs the published package into isolated temporary state, +uses a fresh npm cache and loopback server, and retains failure diagnostics. +It uses checkout Playwright tooling to drive a published runtime; record both +revisions. Its name and automated release wiring target canaries. For stable +verification, inspect compatibility with the selected version or use the manual +pinned-package path above; do not relabel a canary result as stable proof. + +The Docker release-smoke suite is separate image evidence. Likewise, +`scripts/clean-install-npm.sh` defaults to source, and its published mode uses a +moving version and source-driven uninstall; it does not by itself prove the +whole lifecycle of a specific published version. + +Capture OS/architecture, Node/npm versions, exact commands and exit codes, +installed version, serving revision when available, initial data state, screenshots, +logs, and observed outcomes. Redact credentials. Stop owned foreground processes +or uninstall an owned test service with the tested version's supported command; +retain evidence outside disposable data. Never run checkout ownership helpers +with invented metadata to stop a package process. diff --git a/.agents/skills/verify-paperclip/references/verification-targets.md b/.agents/skills/verify-paperclip/references/verification-targets.md new file mode 100644 index 0000000000..837f3803a1 --- /dev/null +++ b/.agents/skills/verify-paperclip/references/verification-targets.md @@ -0,0 +1,87 @@ +# Checkout, data, and hosted targets + +## Checkout and data modes + +Read [DEVELOPING.md](../../../../doc/DEVELOPING.md) and +[DATABASE.md](../../../../doc/DATABASE.md) for the current startup contract. + +- **Existing dev server:** use `pnpm dev` (watch) or `pnpm dev:once` from the + intended checkout. Inspect its instance routing and data configuration first; + do not assume the default port belongs to this checkout. Record the actual + served commit and dirty state. Reuse a matching running server when appropriate. +- **Fresh empty instance:** use the skill's `launch.py` helper with a new data + directory. Complete the UI or CLI onboarding journey under test. This is the + credential-free route; a live model run needs separate provider setup. +- **Seeded test drive:** run the documented CLI `test-drive --no-browser` with + an authorized environment-backed provider credential. It creates a temporary + embedded instance, completes setup, and seeds Test Company and a CEO. It does + not execute the first task. Verify that separately when required. +- **Reused test drive:** pass `--data-dir `. If it already + contains a company, bootstrap flags do not replace companies, agents, or + secrets. Record existing state; do not label the result fresh onboarding. + Test drive rejects external PostgreSQL configuration and database URL overrides. +- **Managed or cloned worktree:** use + [dev-workspace run/verify/fix](../../paperclip-dev-workspace-run-verify-fix/SKILL.md). + Verify seed completion, instance routing, and login handoff before driving it. + Cloned data is existing-user evidence even when the checkout is new. + +`--data-dir` alone is not proof of a fresh database: check effective config and +whether `DATABASE_URL` routes to an external database. Do not print its value. +For ordinary dev, inspect both environment overrides and the active database +config: with `DATABASE_URL` unset, `database.mode: postgres` and a saved +`connectionString` still select external PostgreSQL. Embedded PostgreSQL is the +default only when neither an environment override nor the active config selects +an external database; its instance directory may already hold data. Do not reset +shared data to make a test pass. Choose separate disposable data for first-run +checks and verify the effective backend before calling it isolated. + +Record runtime PID, URL, port owner, checkout, and data location for a process you +start. Check `/api/health`, expected served revision, bootstrap/auth readiness, +and the actual page. Capture starting state and verify persistence after reload. +Use that launcher's documented stop procedure for an owned process; leave a +server you borrowed running. + +## Platform scope + +Ordinary development covers the current host unless the request or failure calls +for more. For a platform issue or release matrix, list required combinations: +macOS, Linux, native Windows, architecture, runtime, and install path. Mark each +combination passed, failed, blocked, or not run separately. + +The local helper needs POSIX tools such as `lsof`; do not assume it qualifies +native Windows. Use the supported CLI and native shell for that platform, with +isolated instance/config/cache paths and platform-appropriate process cleanup. +A container proves its guest OS and selected architecture, not the host's native +installer, service manager, file permissions, or path behavior. Keep browser and +server platforms distinct when testing a remote Linux box from a Mac. + +## Staging and production + +Use the supplied tenant URL and intended identity. Determine whether the target +is the Paperclip tenant application or the Cloud control plane. Confirm the +environment from the operator's deployment record and authenticated routing; +a page title or local Git checkout is not evidence of the serving revision. + +If a build/deploy is needed, follow the private skill handoff in `SKILL.md` first. +Record the exact serving SHA and release/image identity after deployment, not +just the requested branch, CI result, or fleet default. Recheck after a long run: +a staging auto-roll or another deployment can change the serving version. Mark +unknown or changing provenance as inconclusive for a claim about a specific SHA. + +For staging, use a designated test workspace and its real hosted sign-in/entry +flow. Record fresh versus existing workspace state, experimental flags, relevant +provider connections, and whether dependencies are live. Check tenant readiness +and then the feature journey, including persistence or continuation. A control +plane `/healthz` response or tenant HTML 200 is not feature acceptance. + +For production, keep verification within the requested stack, identity, and +journey. Prefer observation of existing state unless writes are authorized; use +an agreed test workspace for mutating checks. Do not run the empty-instance +onboarding smoke, reset data, change flags, or trigger paid agent work merely to +make a production check pass. If a necessary write is outside the request, +report that missing authorization and complete the read-only checks. + +Never use local ownership helpers against a remote server. Close only browser +sessions you created and clean up only test records whose removal is authorized. +Keep shared services running. Report production Cloud verification separately +from stable npm verification; the shipped artifacts and configuration differ.