diff --git a/.agents/skills/create-paperclip-bundled-skill/SKILL.md b/.agents/skills/create-paperclip-bundled-skill/SKILL.md new file mode 100644 index 00000000000..50b4dd57ab4 --- /dev/null +++ b/.agents/skills/create-paperclip-bundled-skill/SKILL.md @@ -0,0 +1,265 @@ +--- +name: create-paperclip-bundled-skill +description: > + Turn an idea, tweet, or task into a skill in the Paperclip skills catalog + (packages/skills-catalog). Use when asked to FIND or MAKE a skill and publish + it as a bundled/optional catalog skill: research prior art, reference or + author it, add examples, regenerate the manifest, open a PR. +--- + +# Create a Paperclip Bundled Skill + +Take source material — a tweet, a task description, a blog post, "make a skill +that does X" — and land it as a skill in the Paperclip skills catalog +(`packages/skills-catalog/`), delivered as a reviewed PR. The catalog is the +shelf every Paperclip company browses and installs from, so the bar is: correct +metadata, useful instructions, worked examples, and a clean validation run. + +The core rule is **FIND before MAKE**: if a good skill already exists (in the +catalog, in this repo, or published on GitHub), reference or adapt it instead +of writing a duplicate from scratch. + +## When to use + +- A human sends a tweet/link/idea and asks for it to become a Paperclip skill. +- A task asks to bundle an existing repo skill into the catalog. +- A task asks to add an external published skill to the catalog. + +## When not to use + +- The skill is company-private (belongs in that company's library via the + Skills UI/API, not the shipped catalog). +- You only need a repo-internal agent skill for working on Paperclip itself — + that goes in `.agents/skills/` or `skills/`, with no catalog machinery. + +## Step 0 — Capture the source material + +Understand exactly what the skill should teach before writing anything. + +**Tweets / X links.** Use the `xc` CLI (X API client). Paperclip engineering +agent environments ship it preinstalled and pre-authenticated; it is not a +tool you install or mint credentials for yourself. Check availability before +relying on it: + +```sh +command -v xc && xc whoami # on PATH and authenticated? if not, use the fallback below +``` + +```sh +xc get --json # the post itself (conversation_id, author) +xc search 'conversation_id:' --archive --json # rest of the thread (>7 days old needs --archive) +xc user # author context +xc search '' -n 30 # related discussion +``` + +If `xc` is not on PATH, is unauthenticated, or the account lacks read access +(the check above fails for any reason), delegate the +fetch to a teammate with X/Twitter access (e.g. the Content Strategist agent) +via a child issue: give them the URL and ask for full text of the post + thread ++ any linked content. + +**Other sources.** Fetch linked articles/READMEs directly. Record the source +URL — it goes in the skill body or PR description as attribution. + +Distill: what is the repeatable procedure? What inputs does it take? What does +"done" look like? If the source is just an aspiration ("agents should write +better commit messages"), you are authoring the procedure yourself — say so in +the PR. + +## Step 1 — FIND: search for an existing skill + +Search in this order; stop when you have a clear winner. + +1. **Already in the catalog?** Avoid duplicates (duplicate slugs fail the + build): + ```sh + grep -i '' packages/skills-catalog/generated/catalog.json + ls packages/skills-catalog/catalog/{bundled,optional}/*/ + ``` +2. **Already in this repo?** Check `.agents/skills/`, `skills/`, and issue + history (`gh search issues` / Paperclip board) for prior work on the topic. +3. **Published on GitHub?** Skills are conventionally a directory with a + `SKILL.md`: + ```sh + gh search code --filename SKILL.md "" --limit 20 + gh search repos " skill" --limit 20 + ``` + Also check known collections (e.g. `anthropics/skills`) and do a web search + for ` agent skill SKILL.md`. + +Judge candidates by: does the SKILL.md actually contain the procedure (not a +stub)? Is it maintained? What does it bundle (scripts raise the trust level)? +Is the license compatible with redistribution? Then pick a path: + +- **Good external skill exists** → add it as an **external reference** + (Step 2A). It stays attributed to and pinned at the upstream repo. +- **Partial match** → author a local skill (Step 2B) that adapts the idea; + credit the source with a link in the SKILL.md body. +- **Nothing usable** → author a new local skill (Step 2B). + +## Step 2 — Choose kind, category, and slug + +- **kind**: default to `optional`. Use `bundled` only when the skill should + ship to every Paperclip company by default — that needs explicit human/board + direction, not your judgment call. +- **category**: reuse an existing directory when one fits (`browser`, + `content`, `docs`, `finance`, `paperclip-operations`, `product`, `quality`, + `research`, `software-development`). New categories are allowed but must be + lowercase kebab-case slugs. +- **slug**: lowercase kebab-case (`^[a-z0-9]+(-[a-z0-9]+)*$`), unique across + the whole catalog (both kinds). + +The skill lives at +`packages/skills-catalog/catalog////` and its canonical +key is `paperclipai///`. + +## Step 2A — External reference path (`catalog-ref.json`) + +The directory contains **only** `catalog-ref.json` (a directory with both +`catalog-ref.json` and `SKILL.md` fails the build). The manifest builder +fetches the pinned files from GitHub at build time and inventories them. + +```sh +# Pin the exact commit for the chosen ref (tag or branch) +gh api repos///commits/ --jq .sha +``` + +```json +{ + "source": { + "type": "github", + "hostname": "github.com", + "owner": "", + "repo": "", + "ref": "", + "commit": "<40-char sha from above>", + "path": "" + }, + "files": ["SKILL.md", "references/**", "scripts/run.py"], + "defaultInstall": false, + "recommendedForRoles": ["researcher"], + "requires": ["python3"], + "tags": ["topic", "keywords"] +} +``` + +Rules the builder enforces: + +- `files` entries are exact relative paths or `dir/**` globs; `SKILL.md` must + be included and must have frontmatter with `name` and `description`. +- If the upstream frontmatter declares `key`/`slug`, they must match the + catalog placement — otherwise pick a matching slug or use the local path. +- `commit` must be a full 40-hex SHA; every listed file must be ≤ 1 MiB. +- `recommendedForRoles`, `requires`, `tags` live in the JSON (there is no + local SKILL.md to carry them). + +See `catalog/optional/research/last30days/catalog-ref.json` for the live +example, and `examples/external-reference.md` next to this skill. + +## Step 2B — Author a local catalog skill + +Layout: + +``` +catalog//// +├── SKILL.md # required entrypoint +├── examples/ # 1–2 worked examples (Step 3) +├── references/ # optional deep-dive docs +├── scripts/ # optional — raises trust level, avoid unless needed +└── assets/ # optional templates/images +``` + +`SKILL.md` frontmatter (all validated by the builder): + +```markdown +--- +name: +description: > + 40–300 chars. Routing logic, not marketing: what it does, when to use it, + when not to. +key: paperclipai/// +recommendedForRoles: + - engineer # non-empty; used for staffing suggestions +tags: + - topic # non-empty; used for browse/search +--- +``` + +Optional frontmatter: `defaultInstall: true` (only for skills every new +company should get), `requires: [node, python3, ...]` for runtime deps. + +Body: follow `docs/guides/agent-developer/writing-a-skill.md` — "When to use" +/ "When not to use" sections, concrete commands over prose, supporting detail +in `references/`. If the skill came from a tweet or external source, link it +in the body for attribution. + +Trust level is derived from files, not declared: any `scripts/` file makes the +skill `scripts_executables` (install becomes audit-gated and you must extend +the `scriptBearing` expectation in `src/shipped-catalog.test.ts`); `assets/` +or non-markdown files make it `assets`; markdown-only skills stay +`markdown_only`. Prefer markdown-only. + +## Step 3 — Write 1–2 worked examples + +Create `examples/` inside the skill directory with one or two markdown files, +each a complete input → application → output walkthrough (realistic input, the +skill's steps applied, the finished artifact). These ship with the skill so +installers can judge it before running it, and they keep the trust level at +`markdown_only` because they are `.md` files. + +Name them by scenario, e.g. `examples/rewrite-release-note.md`. + +## Step 4 — Regenerate the manifest and update tests + +Never hand-edit `generated/catalog.json`; it is deterministic build output. + +```sh +pnpm --filter @paperclipai/skills-catalog build:manifest # regenerates generated/catalog.json +pnpm --filter @paperclipai/skills-catalog validate # must report no errors +``` + +(External references need network access to GitHub during these steps.) + +Then update `packages/skills-catalog/src/shipped-catalog.test.ts`: + +- add the new key to `EXPECTED_BUNDLED_KEYS` or `EXPECTED_OPTIONAL_KEYS` + (alphabetical order); +- if the skill bears scripts, add it to the `scriptBearing` expectation. + +```sh +pnpm --filter @paperclipai/skills-catalog test +``` + +The test suite also enforces the ≤300-char frontmatter description budget +across the repo and the ≥40-char description / non-empty roles+tags rules for +every catalog skill. + +## Step 5 — Open the PR + +Follow the `prepare-paperclip-pr` skill (`.agents/skills/prepare-paperclip-pr/`) +against `paperclipai/paperclip` master. The diff should contain exactly: + +1. the new skill directory (SKILL.md + examples/ + supporting files, **or** + catalog-ref.json), +2. the regenerated `generated/catalog.json`, +3. the `shipped-catalog.test.ts` expectation update. + +In the PR body: link the source material (tweet URL, upstream repo), state +whether this is a new skill / adaptation / external reference, and note the +trust level. Reference PR #10410 (simplified-english) as the shape of a +minimal optional-skill PR. + +## Gotchas + +- `generated/catalog.json` staleness is a validation error — always rerun + `build:manifest` after any file change inside the skill directory (the + inventory carries per-file sha256 hashes). +- Duplicate `slug` across bundled *and* optional fails the build, not just + duplicate keys. +- Symlinks inside a skill directory must resolve within it; directory + symlinks are rejected — copy files in. +- The `bundled` kind and `defaultInstall` are independent axes; don't set + `defaultInstall: true` casually on optional skills. +- For external references the builder fetches from GitHub on every manifest + build; a moved/deleted upstream breaks the build, which is why `commit` is + pinned — prefer upstream tags for `ref`. diff --git a/.agents/skills/create-paperclip-bundled-skill/examples/external-reference.md b/.agents/skills/create-paperclip-bundled-skill/examples/external-reference.md new file mode 100644 index 00000000000..6c2084794df --- /dev/null +++ b/.agents/skills/create-paperclip-bundled-skill/examples/external-reference.md @@ -0,0 +1,84 @@ +# Example — FIND path: tweet → existing skill → external reference + +Real artifact: `packages/skills-catalog/catalog/optional/research/last30days/`. + +## Input + +Dotta sends a tweet praising a "last 30 days" research workflow that sweeps +Reddit/X/YouTube for what changed recently on a topic. + +## Step 0 — Capture + +```sh +xc get https://x.com//status/ --json # post text + conversation_id +xc search 'conversation_id:' --archive --json # the rest of the thread +``` + +The thread links a GitHub repo: `mvanhorn/last30days-skill`, which already +contains a proper skill (`skills/last30days/SKILL.md` plus scripts and +references). + +## Step 1 — FIND + +- Not in the catalog, not in this repo. +- The upstream repo IS the skill — maintained, tagged releases, real SKILL.md. +- Verdict: **FIND** — add it as an external reference, keep attribution and + updates upstream. + +## Step 2A — catalog-ref.json + +Placement: `optional` / `research` / `last30days`. Pin the release tag to an +exact commit: + +```sh +gh api repos/mvanhorn/last30days-skill/commits/v3.3.0 --jq .sha +# → daca71f89eb71d0d56d01a43ed7627aa919dba4f +``` + +`catalog/optional/research/last30days/catalog-ref.json` (the only file in the +directory): + +```json +{ + "source": { + "type": "github", + "hostname": "github.com", + "owner": "mvanhorn", + "repo": "last30days-skill", + "ref": "v3.3.0", + "commit": "daca71f89eb71d0d56d01a43ed7627aa919dba4f", + "path": "skills/last30days" + }, + "files": [ + "SKILL.md", + "agents/openai.yaml", + "references/**", + "scripts/briefing.py", + "scripts/compare.sh", + "scripts/last30days.py", + "scripts/lib/**", + "scripts/setup-keychain.sh", + "scripts/store.py", + "scripts/watchlist.py" + ], + "defaultInstall": false, + "recommendedForRoles": ["researcher", "marketer", "product-manager", "analyst"], + "requires": ["node", "python3"], + "tags": ["research", "last-30-days", "social-media", "trends", "citations", "reddit", "x", "youtube"] +} +``` + +Metadata (`recommendedForRoles`, `requires`, `tags`) lives in the JSON because +there is no local SKILL.md to carry it. + +## Steps 4–5 — Manifest, tests, PR + +- `pnpm --filter @paperclipai/skills-catalog build:manifest` fetches the + pinned files from GitHub and inventories them (network required). +- The skill bundles `scripts/`, so trust level derives to + `scripts_executables` → it must also be added to the `scriptBearing` + expectation in `src/shipped-catalog.test.ts`, alongside + `EXPECTED_OPTIONAL_KEYS`. +- PR diff: `catalog-ref.json`, regenerated `generated/catalog.json`, test + expectations. PR body links both the tweet and the upstream repo, and calls + out the elevated trust level so review is deliberate. diff --git a/.agents/skills/create-paperclip-bundled-skill/examples/new-local-skill.md b/.agents/skills/create-paperclip-bundled-skill/examples/new-local-skill.md new file mode 100644 index 00000000000..7a0d1c4aa85 --- /dev/null +++ b/.agents/skills/create-paperclip-bundled-skill/examples/new-local-skill.md @@ -0,0 +1,90 @@ +# Example — MAKE path: idea → new optional catalog skill + +Real run: PAP-15684 → PR #10410 (`feat(skills-catalog): add optional +/simplified-english skill`). + +## Input + +Task from Dotta: "make a skill that has agents write user-facing text in +Simplified English." No tweet this time — the source is a known public +specification (ASD-STE100 Simplified Technical English). + +## Step 1 — FIND + +- `grep -i 'simplified\|plain.english' packages/skills-catalog/generated/catalog.json` + → no hits; nothing in `.agents/skills/` or `skills/` either. +- `gh search code --filename SKILL.md "simplified technical english"` → no + usable published skill (only STE checker tools, no SKILL.md procedure). +- Verdict: **MAKE** a new local skill. + +## Step 2 — Placement + +- kind: `optional` (useful, but not something every company must ship with). +- category: `content` (existing category, fits writing/communication). +- slug: `simplified-english`. +- Path: `packages/skills-catalog/catalog/optional/content/simplified-english/`. + +## Step 2B — Authoring + +One markdown-only `SKILL.md` (trust level stays `markdown_only`): + +```markdown +--- +name: simplified-english +description: Write user-facing comments, plans, and documents in ASD-STE100 Simplified Technical English — short, unambiguous sentences with approved words and one meaning each — so readers understand them the first time. +key: paperclipai/optional/content/simplified-english +recommendedForRoles: + - engineer + - product + - writer + - devrel +tags: + - writing + - communication + - clarity + - style +--- + +# Simplified English + +For user-facing comments, plans, and documents, write using only ASD-STE100 +Simplified Technical English. + +## Core rules + +- Use short sentences (procedures ≤ 20 words, descriptions ≤ 25 words). +- Give one instruction per sentence. +- Use approved words with one meaning each; avoid synonyms and jargon. +... +``` + +Note the frontmatter hits every builder rule: description is 40–300 chars, +`key` matches the placement, roles and tags are non-empty. + +## Steps 3–4 — Examples, manifest, tests + +- Add `examples/rewrite-status-comment.md` showing a jargon-heavy status + comment rewritten under the rules (before/after). +- `pnpm --filter @paperclipai/skills-catalog build:manifest` → regenerates + `generated/catalog.json` with the new entry. +- Add `"paperclipai/optional/content/simplified-english"` to + `EXPECTED_OPTIONAL_KEYS` in `src/shipped-catalog.test.ts` (alphabetical). +- `pnpm --filter @paperclipai/skills-catalog test` → green. + +## Step 5 — PR + +Four-part diff: + +``` +packages/skills-catalog/catalog/optional/content/simplified-english/SKILL.md +packages/skills-catalog/catalog/optional/content/simplified-english/examples/rewrite-status-comment.md +packages/skills-catalog/generated/catalog.json +packages/skills-catalog/src/shipped-catalog.test.ts +``` + +(The historical PR #10410 predates this skill and shipped as a three-part +diff without the `examples/` file; a run that follows this skill includes the +worked example from Step 3 in the same PR.) + +PR body links the ASD-STE100 spec as the source and states trust level +`markdown_only`. diff --git a/.agents/skills/garden-inbox/SKILL.md b/.agents/skills/garden-inbox/SKILL.md new file mode 100644 index 00000000000..10fd653c180 --- /dev/null +++ b/.agents/skills/garden-inbox/SKILL.md @@ -0,0 +1,98 @@ +--- +name: garden-inbox +description: Scan a Paperclip user's Mine inbox, classify reversible archive candidates, request checkbox confirmation, and archive only accepted selections. Use when asked to garden, clean up, prune, or tidy a Paperclip inbox without changing issues, branches, or workspaces. +--- + +# Garden Inbox + +Use the bundled script for every stage. Keep the workflow strictly ordered: `scan` → `confirm` → `apply`. + +## Safety contract + +- Treat `scan` as read-only. It may read inbox, workspace, close-readiness, and local Git metadata only. +- Treat `confirm` as a confirmation-card write only. It must not archive inbox entries or mutate issue fields, branches, or workspaces. +- Run `apply` only after a resolved `request_checkbox_confirmation` interaction. It archives only accepted option IDs that also occur in the originating `candidates.json`. +- Remember that inbox archive state is per-user presentation state. It is reversible and does not change the underlying issue. +- Never substitute issue status changes, branch deletion, workspace cleanup, or issue deletion for inbox archiving. + +## Inputs + +Require `PAPERCLIP_API_URL` and `PAPERCLIP_API_KEY`. The script strips a trailing `/api` from the URL. It resolves the target user from the run JWT payload's `responsible_user_id`; use `--user-id ` only for an explicit target override. + +Use a run-owned output directory when available: + +```bash +RUN_DIR="${PAPERCLIP_RUN_SCRATCH_DIR:-${PAPERCLIP_TASK_SCRATCH_DIR:-.}}/garden-inbox" +mkdir -p "$RUN_DIR" +``` + +## 1. Scan + +```bash +node .agents/skills/garden-inbox/scripts/garden-inbox.mjs scan \ + --output-dir "$RUN_DIR" \ + --stale-days 60 +``` + +Inspect `garden-inbox-report.md` and `candidates.json`. The report groups every inbox row into exactly one bucket: + +The scan reads Mine separately for each included issue status so the endpoint's global 500-row cap does not silently omit older rows. If any single-status query reaches that cap, both outputs mark coverage as possibly truncated; do not treat that scan as complete. + +- A: merged/archived workspace and all linked work terminal; selected by default. +- B: terminal or workspace-gone work idle beyond the threshold; selected by default. +- C: stale work with commits ahead of base; never selected by default. +- D: keep; never offered for archiving. + +Do not manually promote bucket D entries into the candidate file. + +## 2. Confirm + +Post checkbox interactions on the driving issue: + +```bash +node .agents/skills/garden-inbox/scripts/garden-inbox.mjs confirm \ + --issue-id "$PAPERCLIP_TASK_ID" \ + --candidates "$RUN_DIR/candidates.json" +``` + +The script posts sequential cards when a scan has more than 200 candidates. Re-running `confirm` with the same scan file is idempotent. Leave the driving issue in the waiting posture required by the surrounding Paperclip heartbeat workflow. + +For development or payload review, suppress the POST: + +```bash +node .agents/skills/garden-inbox/scripts/garden-inbox.mjs confirm \ + --issue-id "$PAPERCLIP_TASK_ID" \ + --candidates "$RUN_DIR/candidates.json" \ + --dry-run +``` + +## 3. Apply accepted selections + +After an interaction-resolution wake, take the resolved interaction ID from the wake payload and run: + +```bash +node .agents/skills/garden-inbox/scripts/garden-inbox.mjs apply \ + --issue-id "$PAPERCLIP_TASK_ID" \ + --interaction-id "$INTERACTION_ID" \ + --candidates "$RUN_DIR/candidates.json" +``` + +On rejection, expiry, or an accepted empty selection, the script archives nothing. Apply preserves the scan file's target user, including an explicit `--user-id` override. Its summary includes the API undo path and target-user body for every archived row. + +Test `apply` without API writes by supplying a saved interaction response: + +```bash +node .agents/skills/garden-inbox/scripts/garden-inbox.mjs apply \ + --issue-id "$PAPERCLIP_TASK_ID" \ + --interaction-file resolved-interaction.json \ + --candidates "$RUN_DIR/candidates.json" \ + --dry-run +``` + +## Verify the bundled logic + +Run the zero-dependency Node tests after changing classification or selection safety: + +```bash +node --test .agents/skills/garden-inbox/scripts/garden-inbox.test.mjs +``` diff --git a/.agents/skills/garden-inbox/scripts/garden-inbox.mjs b/.agents/skills/garden-inbox/scripts/garden-inbox.mjs new file mode 100755 index 00000000000..d196b71d118 --- /dev/null +++ b/.agents/skills/garden-inbox/scripts/garden-inbox.mjs @@ -0,0 +1,699 @@ +#!/usr/bin/env node + +import { createHash } from "node:crypto"; +import { execFileSync } from "node:child_process"; +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname, resolve } from "node:path"; +import { pathToFileURL } from "node:url"; + +const TERMINAL_STATUSES = new Set(["done", "cancelled"]); +const OFFERED_BUCKETS = new Set(["A", "B", "C"]); +const DEFAULT_STALE_DAYS = 60; +const INTERACTION_LIMIT = 200; +const INBOX_MINE_STATUSES = ["backlog", "todo", "in_progress", "in_review", "blocked", "done"]; +const INBOX_QUERY_CAP = 500; + +class ApiError extends Error { + constructor(status, message, body = null) { + super(`${status}: ${message}`); + this.name = "ApiError"; + this.status = status; + this.body = body; + } +} + +function usage() { + return `Usage: + garden-inbox.mjs [scan] [--user-id UUID] [--stale-days 60] [--output-dir DIR] + garden-inbox.mjs confirm [--issue-id ID] [--candidates FILE] [--dry-run] + garden-inbox.mjs apply [--issue-id ID] (--interaction-id ID | --interaction-file FILE) [--candidates FILE] [--dry-run] + +Common environment: + PAPERCLIP_API_URL, PAPERCLIP_API_KEY, PAPERCLIP_RUN_ID, PAPERCLIP_TASK_ID`; +} + +function parseArgs(argv) { + const args = [...argv]; + let command = "scan"; + if (args[0] && !args[0].startsWith("-")) command = args.shift(); + if (!["scan", "confirm", "apply"].includes(command)) throw new Error(`Unknown command: ${command}`); + + const options = {}; + while (args.length > 0) { + const token = args.shift(); + if (token === "--help" || token === "-h") { + options.help = true; + continue; + } + if (!token.startsWith("--")) throw new Error(`Unexpected argument: ${token}`); + const key = token.slice(2).replaceAll("-", "_"); + if (key === "dry_run") { + options[key] = true; + continue; + } + const value = args.shift(); + if (value === undefined || value.startsWith("--")) throw new Error(`${token} requires a value`); + if (options[key] === undefined) options[key] = value; + else options[key] = Array.isArray(options[key]) ? [...options[key], value] : [options[key], value]; + } + return { command, options }; +} + +function required(value, name) { + if (typeof value !== "string" || value.trim() === "") throw new Error(`${name} is required`); + return value.trim(); +} + +function normalizeApiBase(value) { + let base = required(value, "PAPERCLIP_API_URL").replace(/\/+$/, ""); + if (base.endsWith("/api")) base = base.slice(0, -4); + return base; +} + +function runtime(options, { requireApi = true } = {}) { + const apiUrl = options.api_url ?? process.env.PAPERCLIP_API_URL; + const apiKey = options.api_key ?? process.env.PAPERCLIP_API_KEY; + return { + apiBase: requireApi ? normalizeApiBase(apiUrl) : (apiUrl ? normalizeApiBase(apiUrl) : null), + apiKey: requireApi ? required(apiKey, "PAPERCLIP_API_KEY") : apiKey ?? null, + runId: options.run_id ?? process.env.PAPERCLIP_RUN_ID ?? null, + }; +} + +async function apiRequest(context, path, { method = "GET", body } = {}) { + const headers = { Accept: "application/json", Authorization: `Bearer ${context.apiKey}` }; + if (body !== undefined) headers["Content-Type"] = "application/json"; + if (method !== "GET" && context.runId) headers["X-Paperclip-Run-Id"] = context.runId; + const response = await fetch(`${context.apiBase}/api${path}`, { + method, + headers, + body: body === undefined ? undefined : JSON.stringify(body), + }); + const text = await response.text(); + let parsed = null; + if (text) { + try { parsed = JSON.parse(text); } catch { parsed = text; } + } + if (!response.ok) { + const message = parsed && typeof parsed === "object" + ? parsed.error ?? parsed.message ?? response.statusText + : parsed ?? response.statusText; + throw new ApiError(response.status, String(message), parsed); + } + return parsed; +} + +function decodeJwtPayload(token) { + const parts = required(token, "PAPERCLIP_API_KEY").split("."); + if (parts.length < 2) throw new Error("PAPERCLIP_API_KEY is not a JWT; pass --user-id explicitly"); + try { + return JSON.parse(Buffer.from(parts[1], "base64url").toString("utf8")); + } catch { + throw new Error("Could not decode the PAPERCLIP_API_KEY JWT payload; pass --user-id explicitly"); + } +} + +function resolveUserId(options, apiKey) { + if (options.user_id) return required(options.user_id, "--user-id"); + const userId = decodeJwtPayload(apiKey).responsible_user_id; + if (typeof userId !== "string" || userId.trim() === "") { + throw new Error("Run JWT has no responsible_user_id; pass --user-id explicitly"); + } + return userId.trim(); +} + +function parsePositiveInteger(value, fallback, name) { + if (value === undefined) return fallback; + const parsed = Number(value); + if (!Number.isInteger(parsed) || parsed < 1 || parsed > 3650) { + throw new Error(`${name} must be an integer from 1 to 3650`); + } + return parsed; +} + +function validDate(value) { + if (!value) return null; + const date = new Date(value); + return Number.isNaN(date.getTime()) ? null : date; +} + +function latestDate(...values) { + return values.flat().map(validDate).filter(Boolean).sort((left, right) => right.getTime() - left.getTime())[0] ?? null; +} + +function runGit(args, cwd) { + try { + return execFileSync("git", ["-C", cwd, ...args], { + encoding: "utf8", + stdio: ["ignore", "pipe", "ignore"], + timeout: 5_000, + }).trim(); + } catch { + return ""; + } +} + +function gitLastCommitAt(branchName, roots) { + if (!branchName) return null; + for (const root of [...new Set(roots.filter(Boolean))]) { + const parsed = validDate(runGit(["log", "-1", "--format=%cI", branchName], root)); + if (parsed) return parsed; + } + return null; +} + +function findLocalBranch(identifier, roots) { + if (!identifier) return null; + const needle = identifier.toLowerCase(); + for (const root of [...new Set(roots.filter(Boolean))]) { + const branches = runGit(["for-each-ref", "--format=%(refname:short)", "refs/heads"], root) + .split("\n").map((branch) => branch.trim()).filter(Boolean) + .filter((branch) => branch.toLowerCase().includes(needle)); + if (branches.length === 1) return branches[0]; + } + return null; +} + +async function mapLimit(values, limit, mapper) { + const results = new Array(values.length); + let cursor = 0; + async function worker() { + while (cursor < values.length) { + const index = cursor++; + results[index] = await mapper(values[index], index); + } + } + await Promise.all(Array.from({ length: Math.min(limit, values.length) }, worker)); + return results; +} + +async function fetchMineInboxRows(context, userId) { + const statusResults = await Promise.all(INBOX_MINE_STATUSES.map(async (status) => { + const query = new URLSearchParams({ userId, status }); + const rows = await apiRequest(context, `/agents/me/inbox/mine?${query}`); + if (!Array.isArray(rows)) throw new Error(`Inbox API returned a non-array response for status ${status}`); + return { status, rows }; + })); + const rowsById = new Map(); + const statusCounts = {}; + const cappedStatuses = []; + let duplicateCount = 0; + for (const { status, rows } of statusResults) { + statusCounts[status] = rows.length; + if (rows.length >= INBOX_QUERY_CAP) cappedStatuses.push(status); + for (const issue of rows) { + if (rowsById.has(issue.id)) duplicateCount += 1; + rowsById.set(issue.id, issue); + } + } + return { + rows: [...rowsById.values()], + coverage: { + strategy: "per_status", + queryCap: INBOX_QUERY_CAP, + statusCounts, + cappedStatuses, + duplicateCount, + complete: cappedStatuses.length === 0, + }, + }; +} + +async function inspectWorkspace(context, workspaceId, issueIdentifiers, gitRoot) { + let workspace = null; + let readiness = null; + let gone = false; + let error = null; + try { + workspace = await apiRequest(context, `/execution-workspaces/${encodeURIComponent(workspaceId)}`); + } catch (caught) { + if (caught instanceof ApiError && caught.status === 404) gone = true; + else error = caught.message; + } + if (workspace) { + try { + readiness = await apiRequest(context, `/execution-workspaces/${encodeURIComponent(workspaceId)}/close-readiness`); + } catch (caught) { + error = error ?? caught.message; + } + } + + const roots = [workspace?.cwd, workspace?.providerRef, readiness?.git?.repoRoot, readiness?.git?.workspacePath, gitRoot]; + let branchName = workspace?.branchName ?? readiness?.git?.branchName ?? null; + if (!branchName && gone) { + const matches = issueIdentifiers.map((identifier) => findLocalBranch(identifier, roots)).filter(Boolean); + if (new Set(matches).size === 1) branchName = matches[0]; + } + const branchCommitAt = gitLastCommitAt(branchName, roots); + return { + id: workspaceId, + gone, + error, + status: workspace?.status ?? null, + branchName, + lastUsedAt: workspace?.lastUsedAt ?? null, + branchCommitAt: branchCommitAt?.toISOString() ?? null, + readiness, + }; +} + +function reason(code, message, facts = {}) { + return { code, message, facts }; +} + +function classify(issue, workspaceInfo, staleDays, now) { + const terminal = TERMINAL_STATUSES.has(issue.status); + const workspaceArchived = workspaceInfo?.status === "archived"; + const workspaceGone = Boolean(workspaceInfo?.gone); + const readiness = workspaceInfo?.readiness ?? null; + const linkedIssues = Array.isArray(readiness?.linkedIssues) ? readiness.linkedIssues : []; + const allLinkedIssuesTerminal = Boolean(readiness) && linkedIssues.every((linked) => linked.isTerminal === true); + const mergedIntoBase = readiness?.git?.isMergedIntoBase === true; + const aheadCount = Number.isInteger(readiness?.git?.aheadCount) ? readiness.git.aheadCount : null; + const blockerAttention = issue.blockerAttention?.state; + const hasOpenBlockers = issue.status === "blocked" + || (Array.isArray(issue.blockedBy) && issue.blockedBy.some((blocker) => !TERMINAL_STATUSES.has(blocker.status))) + || (typeof blockerAttention === "string" && blockerAttention !== "none"); + const awaitingUser = issue.blockedInboxAttention?.state === "awaiting_decision" + || (issue.status === "in_review" && Boolean(issue.assigneeUserId)); + const issueActivity = latestDate(issue.lastActivityAt, issue.updatedAt, issue.createdAt); + const branchActivity = latestDate(workspaceInfo?.branchCommitAt, workspaceInfo?.lastUsedAt); + const lastActivity = latestDate(issueActivity, branchActivity) ?? new Date(0); + const cutoff = new Date(now.getTime() - staleDays * 86_400_000); + const stale = lastActivity.getTime() <= cutoff.getTime(); + const facts = { + issueStatus: issue.status, + workspaceStatus: workspaceInfo?.status ?? null, + workspaceGone, + workspaceInspectionError: workspaceInfo?.error ?? null, + mergedIntoBase, + allLinkedIssuesTerminal, + aheadCount, + staleDays, + }; + + if (workspaceInfo?.error) { + return { + bucket: "D", + reason: reason("workspace_inspection_failed", "Keep: workspace archive safety could not be verified.", facts), + lastActivity, + }; + } + if (hasOpenBlockers) return { bucket: "D", reason: reason("open_blockers", "Keep: the issue has unresolved blocker or liveness attention.", facts), lastActivity }; + if (awaitingUser) return { bucket: "D", reason: reason("pending_user_action", "Keep: the issue is awaiting a user decision or review.", facts), lastActivity }; + if (!terminal) { + const code = issue.status === "in_review" ? "pending_interaction_or_review" : "non_terminal_status"; + return { bucket: "D", reason: reason(code, `Keep: issue status ${issue.status} is not terminal.`, facts), lastActivity }; + } + if ((mergedIntoBase || workspaceArchived) && allLinkedIssuesTerminal) { + const code = workspaceArchived ? "workspace_archived_finished" : "merged_finished"; + return { bucket: "A", reason: reason(code, "Archive candidate: work is merged or archived and every linked issue is terminal.", facts), lastActivity }; + } + if (!stale) return { bucket: "D", reason: reason("recent_activity", `Keep: activity is newer than ${staleDays} days.`, facts), lastActivity }; + if (aheadCount !== null && aheadCount > 0) { + return { bucket: "C", reason: reason("stale_unmerged_commits", `Review manually: stale branch is ${aheadCount} commit(s) ahead of base.`, facts), lastActivity }; + } + if (terminal || workspaceGone) { + return { bucket: "B", reason: reason("stale_terminal_or_workspace_gone", `Archive candidate: no issue or branch activity for at least ${staleDays} days and the issue is terminal or workspace is gone.`, facts), lastActivity }; + } + return { bucket: "D", reason: reason("insufficient_archive_evidence", "Keep: archive safety conditions were not met.", facts), lastActivity }; +} + +function candidateFrom(issue, workspace, classification) { + return { + issueId: issue.id, + identifier: issue.identifier ?? null, + title: issue.title, + bucket: classification.bucket, + reason: classification.reason, + executionWorkspaceId: issue.executionWorkspaceId ?? null, + workspaceStatus: workspace?.status ?? null, + workspaceGone: Boolean(workspace?.gone), + branchName: workspace?.branchName ?? null, + lastActivityAt: classification.lastActivity.toISOString(), + }; +} + +function markdownEscape(value) { + return String(value ?? "").replaceAll("|", "\\|").replaceAll("\n", " "); +} + +function renderReport(scanResult) { + const labels = { + A: "A — merged and finished (default checked)", + B: "B — stale (default checked)", + C: "C — stale but unmerged (default unchecked)", + D: "D — keep (not offered)", + }; + const lines = [ + "# Garden inbox scan", "", + `- Generated: ${scanResult.generatedAt}`, + `- Target user: \`${scanResult.userId}\``, + `- Stale threshold: ${scanResult.staleDays} days`, + `- Inbox rows classified: ${scanResult.items.length}`, + `- Archive candidates: ${scanResult.candidates.length}`, + `- Scan coverage: ${scanResult.coverage.complete ? "complete across per-status queries" : "possibly truncated"}`, + ]; + if (!scanResult.coverage.complete) { + lines.push( + "", + "> [!WARNING]", + `> Coverage may be incomplete: ${scanResult.coverage.cappedStatuses.map((status) => `\`${status}\``).join(", ")} returned at least ${scanResult.coverage.queryCap} rows, the Mine endpoint cap.`, + ); + } + for (const bucket of ["A", "B", "C", "D"]) { + const items = scanResult.items.filter((item) => item.bucket === bucket); + lines.push("", `## ${labels[bucket]} (${items.length})`, ""); + if (items.length === 0) { + lines.push("_None._"); + continue; + } + lines.push("| Issue | Title | Reason | Workspace / branch | Last activity |", "|---|---|---|---|---|"); + for (const item of items) { + const issue = item.identifier ? `\`${markdownEscape(item.identifier)}\`` : `\`${item.issueId}\``; + const workspace = [item.workspaceStatus, item.branchName].filter(Boolean).join(" / ") || (item.workspaceGone ? "gone" : "none"); + lines.push(`| ${issue} | ${markdownEscape(item.title)} | \`${item.reason.code}\` — ${markdownEscape(item.reason.message)} | ${markdownEscape(workspace)} | ${item.lastActivityAt} |`); + } + } + lines.push(""); + return lines.join("\n"); +} + +function writeText(path, contents) { + const target = resolve(path); + mkdirSync(dirname(target), { recursive: true }); + writeFileSync(target, contents); + return target; +} + +function readJson(path) { + return JSON.parse(readFileSync(resolve(path), "utf8")); +} + +function stableScanId(scanResult) { + const fingerprint = { + userId: scanResult.userId, + staleDays: scanResult.staleDays, + generatedAt: scanResult.generatedAt, + candidates: scanResult.candidates.map((item) => ({ + issueId: item.issueId, + bucket: item.bucket, + reason: item.reason.code, + lastActivityAt: item.lastActivityAt, + })), + }; + return createHash("sha256").update(JSON.stringify(fingerprint)).digest("hex").slice(0, 24); +} + +async function scan(options) { + const context = runtime(options); + const userId = resolveUserId(options, context.apiKey); + const staleDays = parsePositiveInteger(options.stale_days, DEFAULT_STALE_DAYS, "--stale-days"); + const now = options.now ? validDate(options.now) : new Date(); + if (!now) throw new Error("--now must be an ISO date"); + const outputDir = options.output_dir ?? "."; + const candidatesPath = options.candidates ?? resolve(outputDir, "candidates.json"); + const reportPath = options.report ?? resolve(outputDir, "garden-inbox-report.md"); + + const { rows, coverage } = await fetchMineInboxRows(context, userId); + const issuesByWorkspace = new Map(); + for (const issue of rows) { + if (!issue.executionWorkspaceId) continue; + const identifiers = issuesByWorkspace.get(issue.executionWorkspaceId) ?? []; + if (issue.identifier) identifiers.push(issue.identifier); + issuesByWorkspace.set(issue.executionWorkspaceId, identifiers); + } + const workspaceIds = [...issuesByWorkspace.keys()]; + const inspected = await mapLimit(workspaceIds, 8, (workspaceId) => inspectWorkspace( + context, workspaceId, issuesByWorkspace.get(workspaceId), options.git_root ?? process.cwd(), + )); + const workspaceById = new Map(inspected.map((entry) => [entry.id, entry])); + const items = rows.map((issue) => { + const workspace = issue.executionWorkspaceId ? workspaceById.get(issue.executionWorkspaceId) ?? null : null; + return candidateFrom(issue, workspace, classify(issue, workspace, staleDays, now)); + }); + const scanResult = { + schemaVersion: 1, + generatedAt: now.toISOString(), + userId, + staleDays, + sourceCount: rows.length, + coverage, + items, + candidates: items.filter((item) => OFFERED_BUCKETS.has(item.bucket)), + }; + scanResult.scanId = stableScanId(scanResult); + const output = { + schemaVersion: 1, + scanId: scanResult.scanId, + generatedAt: scanResult.generatedAt, + userId, + staleDays, + sourceCount: rows.length, + coverage, + candidates: scanResult.candidates, + kept: items.filter((item) => item.bucket === "D"), + }; + const candidatesFile = writeText(candidatesPath, `${JSON.stringify(output, null, 2)}\n`); + const report = renderReport(scanResult); + const reportFile = writeText(reportPath, report); + process.stdout.write(`${report}\nCandidates JSON: ${candidatesFile}\nReport: ${reportFile}\n`); + return output; +} + +function truncate(value, limit) { + const text = String(value ?? "").trim(); + if (text.length <= limit) return text; + return `${text.slice(0, Math.max(0, limit - 1)).trimEnd()}…`; +} + +function candidateFile(options) { + const data = readJson(options.candidates ?? "candidates.json"); + if (data?.schemaVersion !== 1 + || typeof data.scanId !== "string" + || typeof data.userId !== "string" + || data.userId.trim() === "" + || !Array.isArray(data.candidates)) { + throw new Error("Candidates file is not a garden-inbox schemaVersion 1 scan"); + } + for (const item of data.candidates) { + if (!item?.issueId || !OFFERED_BUCKETS.has(item.bucket)) { + throw new Error("Candidates file contains an invalid or non-offered item"); + } + } + return data; +} + +function archiveTargetBody(scanData) { + return { userId: scanData.userId }; +} + +function chunk(values, size) { + const chunks = []; + for (let index = 0; index < values.length; index += size) chunks.push(values.slice(index, index + size)); + return chunks; +} + +function confirmationBody(scanData, candidates, index, count) { + const options = candidates.map((candidate) => ({ + id: candidate.issueId, + label: truncate(`${candidate.identifier ?? candidate.issueId} — ${candidate.title}`, 120), + description: truncate(`${candidate.reason.message} Last activity: ${candidate.lastActivityAt}.`, 500), + })); + const part = count > 1 ? ` (${index + 1}/${count})` : ""; + return { + kind: "request_checkbox_confirmation", + idempotencyKey: `garden-inbox:${scanData.scanId}:${index + 1}:${count}`, + title: `Confirm inbox archive candidates${part}`, + summary: `Choose which reversible inbox entries to archive${part}.`, + continuationPolicy: "wake_assignee", + payload: { + version: 1, + prompt: `Select the inbox entries to archive${part}. Unchecked entries will remain visible.`, + options, + defaultSelectedOptionIds: candidates + .filter((candidate) => candidate.bucket === "A" || candidate.bucket === "B") + .map((candidate) => candidate.issueId), + minSelected: 0, + acceptLabel: "Archive selected", + rejectLabel: "Keep everything", + detailsMarkdown: `Scan \`${scanData.scanId}\` used a ${scanData.staleDays}-day stale threshold. Buckets A and B start checked; stale branches with unmerged commits (C) start unchecked. Archiving changes only this user's inbox visibility and is reversible.`, + }, + }; +} + +async function confirm(options) { + const data = candidateFile(options); + const drivingIssueId = options.issue_id ?? process.env.PAPERCLIP_TASK_ID; + if (!options.dry_run) required(drivingIssueId, "--issue-id or PAPERCLIP_TASK_ID"); + const groups = chunk(data.candidates, INTERACTION_LIMIT); + if (groups.length === 0) { + process.stdout.write("No archive candidates; no confirmation interaction created.\n"); + return []; + } + const bodies = groups.map((candidates, index) => confirmationBody(data, candidates, index, groups.length)); + if (options.dry_run) { + process.stdout.write(`${JSON.stringify({ dryRun: true, issueId: drivingIssueId ?? null, interactions: bodies }, null, 2)}\n`); + return bodies; + } + + const context = runtime(options); + const created = []; + for (const body of bodies) { + created.push(await apiRequest(context, `/issues/${encodeURIComponent(drivingIssueId)}/interactions`, { + method: "POST", + body, + })); + } + process.stdout.write(`${JSON.stringify({ + scanId: data.scanId, + createdInteractions: created.map((item) => ({ + id: item.id, + status: item.status, + idempotencyKey: item.idempotencyKey, + })), + }, null, 2)}\n`); + return created; +} + +function flattenInteractionFile(value) { + if (Array.isArray(value)) return value; + if (Array.isArray(value?.interactions)) return value.interactions; + if (value?.interaction && typeof value.interaction === "object") return [value.interaction]; + return value && typeof value === "object" ? [value] : []; +} + +function optionValues(value) { + if (value === undefined) return []; + return Array.isArray(value) ? value : [value]; +} + +async function resolveInteractions(options, context, drivingIssueId) { + const files = optionValues(options.interaction_file); + const ids = optionValues(options.interaction_id); + const fromFiles = files.flatMap((file) => flattenInteractionFile(readJson(file))); + if (ids.length === 0) return fromFiles; + required(drivingIssueId, "--issue-id or PAPERCLIP_TASK_ID"); + const live = await apiRequest(context, `/issues/${encodeURIComponent(drivingIssueId)}/interactions`); + if (!Array.isArray(live)) throw new Error("Interactions API returned a non-array response"); + const byId = new Map(live.map((interaction) => [interaction.id, interaction])); + for (const id of ids) { + if (!byId.has(id)) throw new Error(`Interaction ${id} was not found on the driving issue`); + fromFiles.push(byId.get(id)); + } + return fromFiles; +} + +function acceptedCandidates(interaction, scanData, candidateById) { + if (interaction?.kind !== "request_checkbox_confirmation") { + throw new Error(`Interaction ${interaction?.id ?? "(unknown)"} is not request_checkbox_confirmation`); + } + if (typeof interaction.idempotencyKey !== "string" + || !interaction.idempotencyKey.startsWith(`garden-inbox:${scanData.scanId}:`)) { + throw new Error(`Interaction ${interaction.id ?? "(unknown)"} does not belong to scan ${scanData.scanId}`); + } + if (interaction.status !== "accepted" || interaction.result?.outcome !== "accepted") return []; + const selected = interaction.result?.selectedOptionIds ?? []; + if (!Array.isArray(selected)) throw new Error("Accepted interaction has invalid selectedOptionIds"); + const optionIds = new Set((interaction.payload?.options ?? []).map((option) => option.id)); + const seen = new Set(); + return selected.map((issueId) => { + if (seen.has(issueId)) throw new Error(`Interaction selected duplicate option id ${issueId}`); + seen.add(issueId); + if (!optionIds.has(issueId)) throw new Error(`Selected id ${issueId} was not an option in the interaction`); + const candidate = candidateById.get(issueId); + if (!candidate || !OFFERED_BUCKETS.has(candidate.bucket)) { + throw new Error(`Selected id ${issueId} is not an offered candidate in the originating scan`); + } + return candidate; + }); +} + +function renderApplySummary(results, interactions, dryRun, userId) { + const lines = [ + `# Garden inbox ${dryRun ? "dry-run " : ""}apply summary`, "", + `- Interactions inspected: ${interactions.length}`, + `- Entries ${dryRun ? "that would be archived" : "archived"}: ${results.filter((result) => result.ok).length}`, + `- Failures: ${results.filter((result) => !result.ok).length}`, + ]; + if (results.length === 0) { + lines.push("", "Nothing was archived: the interaction was rejected, unresolved, expired, or accepted with no selections."); + } + for (const result of results) { + lines.push("", `- ${result.ok ? "OK" : "FAILED"}: ${result.identifier ?? result.issueId} — ${result.message}`); + if (result.ok) { + lines.push(` Undo: \`DELETE /api/issues/${result.issueId}/inbox-archive\` with body \`${JSON.stringify({ userId })}\`, or use **Unarchive** in the issue properties pane.`); + } + } + return `${lines.join("\n")}\n`; +} + +async function applyAccepted(options) { + const data = candidateFile(options); + const drivingIssueId = options.issue_id ?? process.env.PAPERCLIP_TASK_ID ?? null; + const needsLiveApi = optionValues(options.interaction_id).length > 0 || !options.dry_run; + const context = runtime(options, { requireApi: needsLiveApi }); + const interactions = await resolveInteractions(options, context, drivingIssueId); + if (interactions.length === 0) throw new Error("Provide --interaction-id or --interaction-file"); + const candidateById = new Map(data.candidates.map((candidate) => [candidate.issueId, candidate])); + const accepted = []; + const seen = new Set(); + for (const interaction of interactions) { + for (const candidate of acceptedCandidates(interaction, data, candidateById)) { + if (!seen.has(candidate.issueId)) accepted.push(candidate); + seen.add(candidate.issueId); + } + } + + const results = []; + for (const candidate of accepted) { + if (options.dry_run) { + results.push({ ...candidate, ok: true, message: "would archive this accepted inbox entry" }); + continue; + } + try { + await apiRequest(context, `/issues/${encodeURIComponent(candidate.issueId)}/inbox-archive`, { + method: "POST", + body: archiveTargetBody(data), + }); + results.push({ ...candidate, ok: true, message: "archived accepted inbox entry" }); + } catch (error) { + results.push({ ...candidate, ok: false, message: error.message }); + } + } + + const summary = renderApplySummary(results, interactions, Boolean(options.dry_run), data.userId); + process.stdout.write(summary); + if (options.summary) writeText(options.summary, summary); + if (results.some((result) => !result.ok)) process.exitCode = 1; + return results; +} + +export { + acceptedCandidates, + archiveTargetBody, + classify, + confirm, + decodeJwtPayload, + fetchMineInboxRows, + normalizeApiBase, + scan, +}; + +async function main() { + const { command, options } = parseArgs(process.argv.slice(2)); + if (options.help) { + process.stdout.write(`${usage()}\n`); + return; + } + if (command === "scan") await scan(options); + else if (command === "confirm") await confirm(options); + else await applyAccepted(options); +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + main().catch((error) => { + process.stderr.write(`garden-inbox: ${error.message}\n`); + process.exitCode = 1; + }); +} diff --git a/.agents/skills/garden-inbox/scripts/garden-inbox.test.mjs b/.agents/skills/garden-inbox/scripts/garden-inbox.test.mjs new file mode 100644 index 00000000000..6ceec11f6db --- /dev/null +++ b/.agents/skills/garden-inbox/scripts/garden-inbox.test.mjs @@ -0,0 +1,157 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { + acceptedCandidates, + archiveTargetBody, + classify, + decodeJwtPayload, + fetchMineInboxRows, + normalizeApiBase, +} from "./garden-inbox.mjs"; + +const now = new Date("2026-07-29T00:00:00.000Z"); + +function issue(overrides = {}) { + return { + id: "issue-1", + identifier: "PAP-1", + title: "Example", + status: "done", + updatedAt: "2025-01-01T00:00:00.000Z", + blockedBy: [], + ...overrides, + }; +} + +function workspace(overrides = {}) { + return { + gone: false, + error: null, + status: "active", + branchCommitAt: "2025-01-01T00:00:00.000Z", + readiness: { + git: { isMergedIntoBase: false, aheadCount: 0 }, + linkedIssues: [{ isTerminal: true }], + }, + ...overrides, + }; +} + +test("classifies each archive and keep condition into one bucket", () => { + const merged = workspace({ + readiness: { + git: { isMergedIntoBase: true, aheadCount: 0 }, + linkedIssues: [{ isTerminal: true }], + }, + }); + assert.equal(classify(issue(), merged, 60, now).bucket, "A"); + assert.equal(classify(issue(), workspace(), 60, now).bucket, "B"); + assert.equal(classify(issue(), workspace({ + readiness: { + git: { isMergedIntoBase: false, aheadCount: 2 }, + linkedIssues: [{ isTerminal: true }], + }, + }), 60, now).bucket, "C"); + assert.equal(classify(issue({ status: "in_progress" }), workspace(), 60, now).bucket, "D"); +}); + +test("keeps candidates when workspace safety inspection fails", () => { + const result = classify(issue(), workspace({ error: "503 Service Unavailable", readiness: null }), 60, now); + assert.equal(result.bucket, "D"); + assert.equal(result.reason.code, "workspace_inspection_failed"); +}); + +test("returns only accepted options from the originating scan", () => { + const candidate = { issueId: "issue-1", bucket: "A" }; + const scan = { scanId: "scan-1" }; + const interaction = { + id: "interaction-1", + kind: "request_checkbox_confirmation", + idempotencyKey: "garden-inbox:scan-1:1:1", + status: "accepted", + payload: { options: [{ id: "issue-1" }] }, + result: { outcome: "accepted", selectedOptionIds: ["issue-1"] }, + }; + assert.deepEqual(acceptedCandidates(interaction, scan, new Map([[candidate.issueId, candidate]])), [candidate]); +}); + +test("preserves an overridden scan user for archive and undo requests", () => { + assert.deepEqual(archiveTargetBody({ userId: "target-user" }), { userId: "target-user" }); +}); + +test("rejects selected ids that were not offered", () => { + const scan = { scanId: "scan-1" }; + const interaction = { + id: "interaction-1", + kind: "request_checkbox_confirmation", + idempotencyKey: "garden-inbox:scan-1:1:1", + status: "accepted", + payload: { options: [{ id: "issue-1" }] }, + result: { outcome: "accepted", selectedOptionIds: ["issue-2"] }, + }; + assert.throws( + () => acceptedCandidates(interaction, scan, new Map()), + /was not an option in the interaction/, + ); +}); + +test("decodes the responsible user and normalizes API URLs locally", () => { + const payload = Buffer.from(JSON.stringify({ responsible_user_id: "user-1" })).toString("base64url"); + assert.equal(decodeJwtPayload(`header.${payload}.signature`).responsible_user_id, "user-1"); + assert.equal(normalizeApiBase("https://paperclip.example/api/"), "https://paperclip.example"); +}); + +test("scans Mine once per status and merges rows by issue id", async (t) => { + const requestedStatuses = []; + t.mock.method(globalThis, "fetch", async (url) => { + const parsed = new URL(url); + const status = parsed.searchParams.get("status"); + requestedStatuses.push(status); + const rows = status === "todo" + ? [issue({ id: "shared", status: "todo" })] + : status === "done" + ? [issue({ id: "shared", status: "done" }), issue({ id: "done-only" })] + : []; + return new Response(JSON.stringify(rows), { status: 200 }); + }); + + const result = await fetchMineInboxRows({ + apiBase: "https://paperclip.example", + apiKey: "secret", + }, "user-1"); + + assert.deepEqual(requestedStatuses, ["backlog", "todo", "in_progress", "in_review", "blocked", "done"]); + assert.equal(result.rows.length, 2); + assert.equal(result.rows.find((row) => row.id === "shared").status, "done"); + assert.equal(result.coverage.duplicateCount, 1); + assert.equal(result.coverage.complete, true); + assert.deepEqual(result.coverage.statusCounts, { + backlog: 0, + todo: 1, + in_progress: 0, + in_review: 0, + blocked: 0, + done: 2, + }); +}); + +test("warns when an individual status query reaches the Mine endpoint cap", async (t) => { + t.mock.method(globalThis, "fetch", async (url) => { + const status = new URL(url).searchParams.get("status"); + const rows = status === "done" + ? Array.from({ length: 500 }, (_, index) => issue({ id: `done-${index}` })) + : []; + return new Response(JSON.stringify(rows), { status: 200 }); + }); + + const result = await fetchMineInboxRows({ + apiBase: "https://paperclip.example", + apiKey: "secret", + }, "user-1"); + + assert.equal(result.rows.length, 500); + assert.equal(result.coverage.complete, false); + assert.deepEqual(result.coverage.cappedStatuses, ["done"]); + assert.equal(result.coverage.queryCap, 500); +}); diff --git a/README.md b/README.md index 2d02971edcb..0d94189718b 100644 --- a/README.md +++ b/README.md @@ -415,7 +415,7 @@ See [doc/DEVELOPING.md](doc/DEVELOPING.md) for the full development guide. - ⚪ Self-Organization - ⚪ Automatic Organizational Learning - ⚪ CEO Chat -- 🟡 Cloud deployments (multi-tenant isolation & local→cloud sync shipped) +- 🟡 Cloud deployments (multi-tenant isolation & company Import/Export shipped) - ⚪ Desktop App - ⚪ Bring-your-own-ticket-system (Asana / Linear / Jira as on-ramps) - ⚪ Connected Apps (one-click integrations, e.g. Vercel) diff --git a/ROADMAP.md b/ROADMAP.md index fd10044308b..fd245a3e88f 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -108,11 +108,11 @@ Paperclip should get better at turning completed work into reusable organization We want a lighter-weight way to talk to leadership agents, but those conversations should still resolve to real work objects like plans, issues, approvals, or decisions. This should improve interaction without changing the core task-and-comments model. -### 🟡 Cloud deployments (multi-tenant isolation & local→cloud sync shipped) +### 🟡 Cloud deployments (multi-tenant isolation & company Import/Export shipped) Local-first remains important, but Paperclip also needs a cleaner shared deployment story. Teams should be able to run the same product in hosted or semi-hosted environments without changing the mental model. -Shipped so far: multi-tenant isolation with per-company JWT keys and company-scoped cloud tenants, local→cloud upstream sync, and cloud-managed instance bootstrap. +Shipped so far: multi-tenant isolation with per-company JWT keys and company-scoped cloud tenants, portable company Import/Export (zip bundles that move a company between instances, local or cloud), and cloud-managed instance bootstrap. Next: a blob-store relay so large instances can move without a hand-carried bundle. ### ⚪ Desktop App diff --git a/cli/README.md b/cli/README.md index 97277ddd2cb..a32a6c5611b 100644 --- a/cli/README.md +++ b/cli/README.md @@ -332,20 +332,17 @@ By default, agents run on scheduled heartbeats and event-based triggers (task as
-## Paperclip Cloud Sync +## Importing & Exporting Companies -Cloud upstream sync is behind the `Cloud Sync` experimental setting. Enable it in Instance Settings before pushing. +Export a company to a portable package and import it into any other instance — local or cloud — from a local path or GitHub: ```bash -paperclipai cloud connect https://your-stack.paperclip.app -paperclipai cloud connect https://your-stack.paperclip.app --no-browser -paperclipai cloud push --company --dry-run -paperclipai cloud push --company +paperclipai company export --out ./my-export +paperclipai company import ./my-export --dry-run +paperclipai company import org/repo --target new ``` -`cloud connect` authorizes the local instance against the target stack and stores the upstream token in the local instance secret store. The default path opens a browser for consent; `--no-browser` uses the device-code flow and prints the verification URL and user code. - -`cloud push --dry-run` exports the selected local company, sends a preview bundle to the connected Cloud stack, and exits with code `2` when conflicts need user resolution. A schema mismatch exits with code `3`. Running without `--dry-run` stages chunks idempotently, applies the run, and prints the final summary and recent progress events. +The board UI has matching Export and Import pages in company settings: the Export page shows a fidelity panel listing what the bundle will not carry, and the Import page starts imported agents and routines paused by default, with a post-import activation step. See the [Importing & Exporting guide](https://github.com/paperclipai/paperclip/blob/master/docs/guides/board-operator/importing-and-exporting.md) for details. ## Development diff --git a/cli/src/__tests__/cloud.test.ts b/cli/src/__tests__/cloud.test.ts deleted file mode 100644 index c66fd33f465..00000000000 --- a/cli/src/__tests__/cloud.test.ts +++ /dev/null @@ -1,243 +0,0 @@ -import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; -import type { CompanyPortabilityExportResult } from "@paperclipai/shared"; -import { - assertDiscoveryCompatible, - buildBundleFromLocalCompany, - cloudCommandExitCodes, - connectCloud, - resolveDeviceCodeExpiresAt, -} from "../commands/client/cloud.js"; -import { - LocalUpstreamPushCoordinator, - normalizedContentHash, - type LocalUpstreamExportBundle, -} from "../commands/client/cloud-transfer.js"; -import { getCloudConnection } from "../commands/client/cloud-store.js"; - -const originalEnv = { ...process.env }; -const originalFetch = globalThis.fetch; - -describe("cloud CLI helpers", () => { - let tempHome: string; - - beforeEach(() => { - tempHome = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-cloud-cli-")); - process.env = { ...originalEnv, PAPERCLIP_HOME: tempHome }; - }); - - afterEach(() => { - process.env = { ...originalEnv }; - globalThis.fetch = originalFetch; - vi.restoreAllMocks(); - fs.rmSync(tempHome, { recursive: true, force: true }); - }); - - it("connects with the device-code flow and stores the resulting cloud connection", async () => { - globalThis.fetch = vi.fn(async (url, init) => { - const requestUrl = String(url); - if (requestUrl.endsWith("/.well-known/paperclip-upstream")) { - return jsonResponse(discovery()); - } - if (requestUrl.endsWith("/api/upstream-sync/device-code")) { - expect(JSON.parse(String(init?.body))).toMatchObject({ - stackId: "stack-1", - scopes: ["upstream_import:preview", "upstream_import:write", "upstream_import:read"], - }); - return jsonResponse({ - deviceCode: "device-1", - userCode: "ABCD-EFGH", - verificationUri: "https://cloud.example.test/api/upstream-sync/device-code/approve", - expiresAt: new Date(Date.now() + 60_000).toISOString(), - intervalSeconds: 0, - }); - } - if (requestUrl.endsWith("/api/upstream-sync/token")) { - return jsonResponse({ - accessToken: "upt_test", - scopes: ["upstream_import:preview"], - token: { - id: "token-1", - companyStackId: "stack-1", - targetOrigin: "https://cloud.example.test", - sourceInstanceId: "paperclip-local-default", - sourceInstanceFingerprint: "sha256:test", - scopes: ["upstream_import:preview"], - expiresAt: new Date(Date.now() + 60_000).toISOString(), - }, - }); - } - return jsonResponse({ error: "not_found" }, 404); - }) as typeof fetch; - - const connection = await connectCloud("https://cloud.example.test", { noBrowser: true, json: true }); - - expect(connection.accessToken).toBe("upt_test"); - expect(getCloudConnection("https://cloud.example.test")?.token.id).toBe("token-1"); - }); - - it("hard-blocks incompatible transfer schema versions with the stable schema exit code", () => { - expect(() => assertDiscoveryCompatible(discovery({ supportedSchemaMajor: 99 }))).toThrow(/schema mismatch/i); - expect(cloudCommandExitCodes.schemaMismatch).toBe(3); - }); - - it("falls back to a bounded device-code expiry when the cloud omits or malforms expiresAt", () => { - const now = Date.UTC(2026, 4, 22, 13, 0, 0); - const validExpiry = "2026-05-22T13:05:00.000Z"; - - expect(resolveDeviceCodeExpiresAt(validExpiry, now)).toBe(Date.parse(validExpiry)); - expect(resolveDeviceCodeExpiresAt(undefined, now)).toBe(now + 15 * 60_000); - expect(resolveDeviceCodeExpiresAt("not-a-date", now)).toBe(now + 15 * 60_000); - }); - - it("builds deterministic chunks with validated payload hashes", async () => { - const bundle = await buildTestBundle(); - - expect(bundle.chunks).toHaveLength(2); - expect(bundle.chunks[0]?.sha256).toBe(normalizedContentHash(bundle.chunks[0]?.payload)); - expect(bundle.manifest.chunks[0]?.manifestHash).toBe(bundle.manifest.manifestHash); - expect(bundle.manifest.idempotencyKey).toBe((await buildTestBundle()).manifest.idempotencyKey); - }); - - it("reuses the same manifest and chunk identity when an interrupted apply is retried", async () => { - const bundle = await buildTestBundle(); - const calls: Array<{ path: string; body: unknown }> = []; - const coordinator = new LocalUpstreamPushCoordinator({ - targetOrigin: "https://cloud.example.test", - paperclipCompanyId: "target-company-1", - fetch: async (url, init) => { - const parsed = new URL(String(url)); - const body = init?.body ? JSON.parse(String(init.body)) as unknown : {}; - calls.push({ path: parsed.pathname, body }); - if (parsed.pathname.endsWith("/runs")) return jsonResponse({ run: { id: "run-1" } }); - return jsonResponse({ run: { id: "run-1" }, summary: { create: 0, update: 0, adopt: 0, skip: 2, conflict: 0, staleMapping: 0 } }); - }, - }); - - await coordinator.apply(bundle); - await coordinator.apply(bundle); - - const runBodies = calls.filter((call) => call.path.endsWith("/runs")).map((call) => call.body as { manifest: { idempotencyKey: string } }); - const chunkBodies = calls.filter((call) => call.path.endsWith("/chunks")).map((call) => call.body as { chunkIndex: number; sha256: string }); - expect(runBodies).toHaveLength(2); - expect(runBodies[0]?.manifest.idempotencyKey).toBe(runBodies[1]?.manifest.idempotencyKey); - expect(chunkBodies[0]).toEqual(chunkBodies[2]); - expect(chunkBodies[1]).toEqual(chunkBodies[3]); - }); -}); - -async function buildTestBundle(): Promise { - return buildBundleFromLocalCompany({ - localCompanyId: "local-company-1", - connection: { - id: "conn-1", - remoteUrl: "https://cloud.example.test", - targetOrigin: "https://cloud.example.test", - targetHost: "cloud.example.test", - stackId: "stack-1", - targetCompanyId: "target-company-1", - accessToken: "upt_test", - token: { - id: "token-1", - companyStackId: "stack-1", - targetOrigin: "https://cloud.example.test", - sourceInstanceId: "paperclip-local-default", - sourceInstanceFingerprint: "sha256:test", - scopes: ["upstream_import:preview"], - expiresAt: new Date(Date.now() + 60_000).toISOString(), - }, - privateKeyPem: "unused", - sourcePublicKey: "unused", - sourceInstanceId: "paperclip-local-default", - sourceInstanceFingerprint: "sha256:test", - scopes: ["upstream_import:preview"], - createdAt: "2026-05-18T00:00:00.000Z", - updatedAt: "2026-05-18T00:00:00.000Z", - }, - discovery: discovery(), - localApi: { - post: async () => portabilityExport() as T, - }, - maxEntitiesPerChunk: 1, - mode: "apply", - }); -} - -function discovery(overrides: Partial<{ supportedSchemaMajor: number }> = {}) { - return { - schema: "paperclip-upstream-discovery-v1", - stack: { - id: "stack-1", - slug: "cloud-test", - displayName: "Cloud Test", - companyId: "target-company-1", - origin: "https://cloud.example.test", - }, - auth: { - deviceCode: { - deviceCodeUrl: "https://cloud.example.test/api/upstream-sync/device-code", - verificationUrl: "https://cloud.example.test/api/upstream-sync/device-code/approve", - tokenUrl: "https://cloud.example.test/api/upstream-sync/token", - }, - scopes: ["upstream_import:preview", "upstream_import:write", "upstream_import:read"], - }, - transfer: { - supportedSchemaMajor: overrides.supportedSchemaMajor ?? 1, - featureFlags: ["cloud_sync"], - }, - }; -} - -function portabilityExport(): CompanyPortabilityExportResult { - return { - rootPath: ".", - paperclipExtensionPath: ".paperclip.yaml", - manifest: { - schemaVersion: 1, - generatedAt: "2026-05-18T00:00:00.000Z", - source: { - companyId: "local-company-1", - companyName: "Local Company", - }, - includes: { - company: true, - agents: true, - projects: true, - issues: true, - skills: true, - }, - company: { - path: "company.json", - name: "Local Company", - description: null, - brandColor: null, - logoPath: null, - attachmentMaxBytes: null, - requireBoardApprovalForNewAgents: false, - feedbackDataSharingEnabled: false, - feedbackDataSharingConsentAt: null, - feedbackDataSharingConsentByUserId: null, - feedbackDataSharingTermsVersion: null, - }, - sidebar: null, - agents: [], - skills: [], - projects: [], - issues: [], - envInputs: [], - }, - files: { - "README.md": "Local Company", - }, - warnings: [], - }; -} - -function jsonResponse(body: unknown, status = 200): Response { - return new Response(JSON.stringify(body), { - status, - headers: { "Content-Type": "application/json" }, - }); -} diff --git a/cli/src/__tests__/company-import-zip.test.ts b/cli/src/__tests__/company-import-zip.test.ts index e2983e9a3a9..db01253c22b 100644 --- a/cli/src/__tests__/company-import-zip.test.ts +++ b/cli/src/__tests__/company-import-zip.test.ts @@ -18,12 +18,14 @@ describe("resolveInlineSourceFromPath", () => { const tempDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-company-import-zip-")); tempDirs.push(tempDir); + const blobBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff]); const archivePath = path.join(tempDir, "paperclip-demo.zip"); const archive = createStoredZipArchive( { "COMPANY.md": "# Company\n", ".paperclip.yaml": "schema: paperclip/v1\n", "agents/ceo/AGENT.md": "# CEO\n", + "blobs/4f2d1c9a": blobBytes, "notes/todo.txt": "ignore me\n", }, "paperclip-demo", @@ -38,6 +40,11 @@ describe("resolveInlineSourceFromPath", () => { "COMPANY.md": "# Company\n", ".paperclip.yaml": "schema: paperclip/v1\n", "agents/ceo/AGENT.md": "# CEO\n", + "blobs/4f2d1c9a": { + encoding: "base64", + data: Buffer.from(blobBytes).toString("base64"), + contentType: "application/octet-stream", + }, }, }); }); diff --git a/cli/src/__tests__/company.test.ts b/cli/src/__tests__/company.test.ts index 3cac3ec72c9..1532047f6f8 100644 --- a/cli/src/__tests__/company.test.ts +++ b/cli/src/__tests__/company.test.ts @@ -509,6 +509,9 @@ describe("renderCompanyImportResult", () => { { slug: "ops", id: "project-2", action: "updated", name: "Operations", reason: "replace strategy" }, { slug: "archive", id: null, action: "skipped", name: "Archive", reason: "skip strategy" }, ], + routines: [ + { slug: "weekly-report", id: "routine-1", action: "created", title: "Weekly report", status: "paused" }, + ], envInputs: [], warnings: ["Review API keys"], }, diff --git a/cli/src/__tests__/helpers/zip.ts b/cli/src/__tests__/helpers/zip.ts index ef79b5beda6..0bfd26e64bd 100644 --- a/cli/src/__tests__/helpers/zip.ts +++ b/cli/src/__tests__/helpers/zip.ts @@ -21,7 +21,7 @@ function crc32(bytes: Uint8Array) { return (crc ^ 0xffffffff) >>> 0; } -export function createStoredZipArchive(files: Record, rootPath: string) { +export function createStoredZipArchive(files: Record, rootPath: string) { const encoder = new TextEncoder(); const localChunks: Uint8Array[] = []; const centralChunks: Uint8Array[] = []; @@ -30,7 +30,7 @@ export function createStoredZipArchive(files: Record, rootPath: for (const [relativePath, content] of Object.entries(files).sort(([left], [right]) => left.localeCompare(right))) { const fileName = encoder.encode(`${rootPath}/${relativePath}`); - const body = encoder.encode(content); + const body = typeof content === "string" ? encoder.encode(content) : content; const checksum = crc32(body); const localHeader = new Uint8Array(30 + fileName.length); diff --git a/cli/src/__tests__/worktree.test.ts b/cli/src/__tests__/worktree.test.ts index b75699caeff..bbd055ea1e7 100644 --- a/cli/src/__tests__/worktree.test.ts +++ b/cli/src/__tests__/worktree.test.ts @@ -276,6 +276,7 @@ describe("worktree helpers", () => { path.resolve("/tmp/paperclip-worktrees", "instances", "feature-worktree-support", "db"), ); expect(config.database.embeddedPostgresPort).toBe(54339); + expect(config.database.backup.enabled).toBe(false); expect(config.server.port).toBe(3110); expect(config.auth.publicBaseUrl).toBe("http://127.0.0.1:3110/"); expect(config.storage.localDisk.baseDir).toBe( @@ -289,6 +290,7 @@ describe("worktree helpers", () => { expect(env.PAPERCLIP_HOME).toBe(path.resolve("/tmp/paperclip-worktrees")); expect(env.PAPERCLIP_INSTANCE_ID).toBe("feature-worktree-support"); expect(env.PAPERCLIP_IN_WORKTREE).toBe("true"); + expect(env.PAPERCLIP_DB_BACKUP_ENABLED).toBe("false"); expect(env.PAPERCLIP_WORKTREE_NAME).toBe("feature-worktree-support"); expect(env.PAPERCLIP_WORKTREE_COLOR).toBe("#3abf7a"); expect(formatShellExports(env)).toContain("export PAPERCLIP_INSTANCE_ID='feature-worktree-support'"); diff --git a/cli/src/__tests__/zip-codec.test.ts b/cli/src/__tests__/zip-codec.test.ts new file mode 100644 index 00000000000..03788e88070 --- /dev/null +++ b/cli/src/__tests__/zip-codec.test.ts @@ -0,0 +1,69 @@ +import { describe, expect, it } from "vitest"; +import { bytesToPortableFileEntry, isBlobStorePath, readZipArchive } from "../commands/client/zip.js"; +import { createStoredZipArchive } from "./helpers/zip.js"; + +describe("isBlobStorePath", () => { + it("matches blobs/ entries at the archive root and under a package root", () => { + expect(isBlobStorePath("blobs/4f2d1c9a")).toBe(true); + expect(isBlobStorePath("paperclip-demo/blobs/4f2d1c9a")).toBe(true); + expect(isBlobStorePath("tasks/pap-1/TASK.md")).toBe(false); + expect(isBlobStorePath("blobs/nested/file")).toBe(false); + }); +}); + +describe("bytesToPortableFileEntry", () => { + it("keeps blobs/ entries as base64 octet streams regardless of extension", () => { + const bytes = new Uint8Array([0x00, 0x01, 0x80, 0xfe, 0xff]); + expect(bytesToPortableFileEntry("blobs/4f2d1c9a", bytes)).toEqual({ + encoding: "base64", + data: Buffer.from(bytes).toString("base64"), + contentType: "application/octet-stream", + }); + }); + + it("falls back to base64 when bytes are not valid UTF-8", () => { + const invalidUtf8 = new Uint8Array([0x68, 0x69, 0xff, 0xfe, 0xc0]); + expect(bytesToPortableFileEntry("tasks/pap-1/raw-notes", invalidUtf8)).toEqual({ + encoding: "base64", + data: Buffer.from(invalidUtf8).toString("base64"), + contentType: "application/octet-stream", + }); + }); + + it("decodes valid UTF-8 entries to text", () => { + const bytes = new TextEncoder().encode("# Notes\n\ncafé ✅\n"); + expect(bytesToPortableFileEntry("tasks/pap-1/TASK.md", bytes)).toBe("# Notes\n\ncafé ✅\n"); + }); +}); + +describe("readZipArchive", () => { + it("round-trips blob and invalid UTF-8 entries byte-exactly", async () => { + const blobBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff]); + const invalidUtf8 = new Uint8Array([0x68, 0x69, 0xff, 0xfe, 0xc0]); + const archive = createStoredZipArchive( + { + "COMPANY.md": "# Company\n", + "blobs/4f2d1c9a": blobBytes, + "notes/raw": invalidUtf8, + }, + "paperclip-demo", + ); + + await expect(readZipArchive(archive)).resolves.toEqual({ + rootPath: "paperclip-demo", + files: { + "COMPANY.md": "# Company\n", + "blobs/4f2d1c9a": { + encoding: "base64", + data: Buffer.from(blobBytes).toString("base64"), + contentType: "application/octet-stream", + }, + "notes/raw": { + encoding: "base64", + data: Buffer.from(invalidUtf8).toString("base64"), + contentType: "application/octet-stream", + }, + }, + }); + }); +}); diff --git a/cli/src/commands/client/cloud-store.ts b/cli/src/commands/client/cloud-store.ts deleted file mode 100644 index fa63c7133b0..00000000000 --- a/cli/src/commands/client/cloud-store.ts +++ /dev/null @@ -1,177 +0,0 @@ -import fs from "node:fs"; -import path from "node:path"; -import { resolvePaperclipInstanceRoot } from "../../config/home.js"; - -export interface CloudConnectionTokenRecord { - id: string; - companyStackId: string; - targetOrigin: string; - sourceInstanceId: string; - sourceInstanceFingerprint: string; - scopes: string[]; - expiresAt: string; - [key: string]: unknown; -} - -export interface CloudConnection { - id: string; - remoteUrl: string; - targetOrigin: string; - targetHost: string; - stackId: string; - stackSlug?: string | null; - stackDisplayName?: string | null; - targetCompanyId: string; - accessToken: string; - token: CloudConnectionTokenRecord; - privateKeyPem: string; - sourcePublicKey: string; - sourceInstanceId: string; - sourceInstanceFingerprint: string; - scopes: string[]; - createdAt: string; - updatedAt: string; -} - -interface CloudConnectionStore { - version: 1; - connections: Record; - currentConnectionId?: string; -} - -function defaultStore(): CloudConnectionStore { - return { - version: 1, - connections: {}, - }; -} - -export function resolveCloudConnectionStorePath(): string { - return path.resolve(resolvePaperclipInstanceRoot(), "secrets", "cloud-upstream-connections.json"); -} - -export function readCloudConnectionStore(storePath = resolveCloudConnectionStorePath()): CloudConnectionStore { - if (!fs.existsSync(storePath)) return defaultStore(); - const raw = JSON.parse(fs.readFileSync(storePath, "utf8")) as Partial | null; - const connections: Record = {}; - if (raw?.connections && typeof raw.connections === "object") { - for (const [id, value] of Object.entries(raw.connections)) { - const normalized = normalizeConnection(value); - if (normalized) connections[id] = normalized; - } - } - const currentConnectionId = - typeof raw?.currentConnectionId === "string" && connections[raw.currentConnectionId] - ? raw.currentConnectionId - : Object.values(connections).sort((left, right) => right.updatedAt.localeCompare(left.updatedAt))[0]?.id; - return { - version: 1, - connections, - currentConnectionId, - }; -} - -export function writeCloudConnectionStore( - store: CloudConnectionStore, - storePath = resolveCloudConnectionStorePath(), -): void { - fs.mkdirSync(path.dirname(storePath), { recursive: true }); - fs.writeFileSync(storePath, `${JSON.stringify(store, null, 2)}\n`, { mode: 0o600 }); -} - -export function upsertCloudConnection( - connection: CloudConnection, - storePath = resolveCloudConnectionStorePath(), -): CloudConnection { - const store = readCloudConnectionStore(storePath); - const existing = store.connections[connection.id]; - const now = new Date().toISOString(); - const next = { - ...connection, - createdAt: existing?.createdAt ?? connection.createdAt ?? now, - updatedAt: now, - }; - store.connections[next.id] = next; - store.currentConnectionId = next.id; - writeCloudConnectionStore(store, storePath); - return next; -} - -export function getCloudConnection( - remoteUrlOrOrigin?: string, - storePath = resolveCloudConnectionStorePath(), -): CloudConnection | null { - const store = readCloudConnectionStore(storePath); - if (remoteUrlOrOrigin?.trim()) { - const needle = normalizeRemoteLookup(remoteUrlOrOrigin); - return Object.values(store.connections).find((connection) => - normalizeRemoteLookup(connection.remoteUrl) === needle || - normalizeRemoteLookup(connection.targetOrigin) === needle - ) ?? null; - } - return store.currentConnectionId ? store.connections[store.currentConnectionId] ?? null : null; -} - -function normalizeRemoteLookup(value: string): string { - try { - const url = new URL(value); - return url.origin.replace(/\/+$/u, ""); - } catch { - return value.trim().replace(/\/+$/u, ""); - } -} - -function normalizeConnection(value: unknown): CloudConnection | null { - if (typeof value !== "object" || value === null || Array.isArray(value)) return null; - const record = value as Record; - const id = stringValue(record.id); - const remoteUrl = stringValue(record.remoteUrl); - const targetOrigin = stringValue(record.targetOrigin); - const targetHost = stringValue(record.targetHost); - const stackId = stringValue(record.stackId); - const targetCompanyId = stringValue(record.targetCompanyId); - const accessToken = stringValue(record.accessToken); - const token = typeof record.token === "object" && record.token !== null && !Array.isArray(record.token) - ? record.token as CloudConnectionTokenRecord - : null; - const privateKeyPem = stringValue(record.privateKeyPem); - const sourcePublicKey = stringValue(record.sourcePublicKey); - const sourceInstanceId = stringValue(record.sourceInstanceId); - const sourceInstanceFingerprint = stringValue(record.sourceInstanceFingerprint); - const createdAt = stringValue(record.createdAt); - const updatedAt = stringValue(record.updatedAt); - if ( - !id || !remoteUrl || !targetOrigin || !targetHost || !stackId || !targetCompanyId || - !accessToken || !token || !privateKeyPem || !sourcePublicKey || !sourceInstanceId || - !sourceInstanceFingerprint || !createdAt || !updatedAt - ) { - return null; - } - return { - id, - remoteUrl, - targetOrigin, - targetHost, - stackId, - stackSlug: stringValue(record.stackSlug), - stackDisplayName: stringValue(record.stackDisplayName), - targetCompanyId, - accessToken, - token, - privateKeyPem, - sourcePublicKey, - sourceInstanceId, - sourceInstanceFingerprint, - scopes: stringArray(record.scopes), - createdAt, - updatedAt, - }; -} - -function stringValue(value: unknown): string | null { - return typeof value === "string" && value.trim().length > 0 ? value.trim() : null; -} - -function stringArray(value: unknown): string[] { - return Array.isArray(value) ? value.filter((entry): entry is string => typeof entry === "string") : []; -} diff --git a/cli/src/commands/client/cloud-transfer.ts b/cli/src/commands/client/cloud-transfer.ts deleted file mode 100644 index 9cd1ccbe74a..00000000000 --- a/cli/src/commands/client/cloud-transfer.ts +++ /dev/null @@ -1,297 +0,0 @@ -import { createHash } from "node:crypto"; - -export const upstreamTransferSchema = { - family: "paperclip-upstream-transfer", - version: "1.0.0", - major: 1, - minor: 0, -} as const; - -export type NormalizedSha256 = `sha256:${string}`; - -export interface SourceEntityKey { - sourceInstanceId: string; - sourceCompanyId: string; - sourceEntityType: string; - sourceEntityId: string; - sourceNaturalKey?: string; -} - -export interface UpstreamTransferWarning { - code: string; - severity: "info" | "warning" | "blocker"; - message: string; - entity?: SourceEntityKey; -} - -export interface UpstreamTransferEntityRecord { - key: SourceEntityKey; - contentHash: NormalizedSha256; - dependencies: SourceEntityKey[]; - warnings: UpstreamTransferWarning[]; -} - -export interface UpstreamTransferManifestSource { - sourceInstanceId: string; - sourceCompanyId: string; - sourceInstanceKeyFingerprint: string; - exporterVersion: string; - sourceSchemaVersion: string; -} - -export interface UpstreamTransferManifestTarget { - targetStackId: string; - targetCompanyId: string; - targetOrigin: string; - supportedSchemaMajor: number; -} - -export interface UpstreamTransferChunk { - chunkIndex: number; - totalChunks: number; - byteLength: number; - sha256: NormalizedSha256; - manifestHash: NormalizedSha256; -} - -export interface UpstreamTransferManifest { - schema: typeof upstreamTransferSchema; - source: UpstreamTransferManifestSource; - target: UpstreamTransferManifestTarget; - runId: string; - idempotencyKey: string; - generatedAt: string; - entityCount: number; - entities: UpstreamTransferEntityRecord[]; - chunks: UpstreamTransferChunk[]; - warnings: UpstreamTransferWarning[]; - featureFlags: string[]; - manifestHash: NormalizedSha256; -} - -export interface LocalUpstreamExportEntityInput { - key: SourceEntityKey; - body: Record; - dependencies?: SourceEntityKey[]; - warnings?: UpstreamTransferWarning[]; - conflictKeys?: string[]; -} - -export interface LocalUpstreamExportEntity { - record: UpstreamTransferEntityRecord; - body: Record; - conflictKeys?: string[]; -} - -export interface LocalUpstreamExportChunk { - chunkIndex: number; - totalChunks: number; - byteLength: number; - sha256: NormalizedSha256; - payload: { - entityKeys: SourceEntityKey[]; - }; -} - -export interface LocalUpstreamExportBundle { - manifest: UpstreamTransferManifest; - entities: LocalUpstreamExportEntity[]; - chunks: LocalUpstreamExportChunk[]; -} - -export interface BuildLocalUpstreamExportBundleInput { - source: UpstreamTransferManifestSource; - target: UpstreamTransferManifestTarget; - runId: string; - idempotencyKey: string; - entities: LocalUpstreamExportEntityInput[]; - warnings?: UpstreamTransferWarning[]; - featureFlags?: string[]; - maxEntitiesPerChunk?: number; -} - -export interface LocalUpstreamPushCoordinatorOptions { - targetOrigin: string; - paperclipCompanyId: string; - fetch?: typeof fetch; - headers?: (input: { method: string; path: string }) => HeadersInit | Promise; -} - -export class UpstreamImportRequestError extends Error { - readonly status: number; - readonly body: unknown; - - constructor(status: number, message: string, body: unknown) { - super(message); - this.status = status; - this.body = body; - } -} - -export class LocalUpstreamPushCoordinator { - readonly #targetOrigin: string; - readonly #paperclipCompanyId: string; - readonly #fetch: typeof fetch; - readonly #headers: NonNullable; - - constructor(options: LocalUpstreamPushCoordinatorOptions) { - this.#targetOrigin = options.targetOrigin.replace(/\/+$/u, ""); - this.#paperclipCompanyId = options.paperclipCompanyId; - this.#fetch = options.fetch ?? fetch; - this.#headers = options.headers ?? (() => ({})); - } - - async preview(bundle: LocalUpstreamExportBundle): Promise { - return this.post(`/api/companies/${encodeURIComponent(this.#paperclipCompanyId)}/upstream-imports/preview`, { - manifest: bundle.manifest, - entities: bundle.entities, - }); - } - - async apply(bundle: LocalUpstreamExportBundle): Promise { - const run = await this.post(`/api/companies/${encodeURIComponent(this.#paperclipCompanyId)}/upstream-imports/runs`, { - mode: "apply", - manifest: bundle.manifest, - entities: bundle.entities, - }) as { run?: { id?: unknown } }; - const runId = typeof run.run?.id === "string" ? run.run.id : undefined; - if (!runId) { - throw new Error("Remote upstream importer did not return a run id"); - } - - for (const chunk of bundle.chunks) { - await this.post(`/api/upstream-import-runs/${encodeURIComponent(runId)}/chunks`, chunk); - } - - return this.post(`/api/upstream-import-runs/${encodeURIComponent(runId)}/apply`, {}); - } - - async events(runId: string): Promise { - return this.get(`/api/upstream-import-runs/${encodeURIComponent(runId)}/events`); - } - - private async get(path: string): Promise { - const response = await this.#fetch(`${this.#targetOrigin}${path}`, { - method: "GET", - headers: await this.#headers({ method: "GET", path }), - }); - return parseCoordinatorResponse(response); - } - - private async post(path: string, body: unknown): Promise { - const response = await this.#fetch(`${this.#targetOrigin}${path}`, { - method: "POST", - headers: { - "Content-Type": "application/json", - ...(await this.#headers({ method: "POST", path })), - }, - body: JSON.stringify(body), - }); - return parseCoordinatorResponse(response); - } -} - -export function buildLocalUpstreamExportBundle( - input: BuildLocalUpstreamExportBundleInput, -): LocalUpstreamExportBundle { - const entities = input.entities.map((entity) => ({ - record: { - key: entity.key, - contentHash: normalizedContentHash(entity.body), - dependencies: entity.dependencies ?? [], - warnings: entity.warnings ?? [], - }, - body: entity.body, - conflictKeys: entity.conflictKeys, - })); - const chunks = buildLocalChunks(entities, input.maxEntitiesPerChunk ?? 100); - const manifestWithoutHash = { - schema: upstreamTransferSchema, - source: input.source, - target: input.target, - runId: input.runId, - idempotencyKey: input.idempotencyKey, - generatedAt: new Date(0).toISOString(), - entityCount: entities.length, - entities: entities.map((entity) => entity.record), - chunks: chunks.map(({ payload: _payload, ...chunk }) => chunk), - warnings: input.warnings ?? [], - featureFlags: (input.featureFlags ?? ["cloud_sync"]).slice().sort(), - }; - const manifestHash = normalizedContentHash(manifestWithoutHash); - return { - manifest: { - ...manifestWithoutHash, - chunks: manifestWithoutHash.chunks.map((chunk) => ({ ...chunk, manifestHash })), - manifestHash, - }, - entities, - chunks, - }; -} - -export function normalizedContentHash(value: unknown): NormalizedSha256 { - return `sha256:${createHash("sha256").update(canonicalJson(value)).digest("hex")}`; -} - -export function canonicalJson(value: unknown): string { - return JSON.stringify(sortJson(value)); -} - -function buildLocalChunks( - entities: LocalUpstreamExportEntity[], - maxEntitiesPerChunk: number, -): LocalUpstreamExportChunk[] { - if (!Number.isInteger(maxEntitiesPerChunk) || maxEntitiesPerChunk < 1) { - throw new Error("maxEntitiesPerChunk must be a positive integer"); - } - if (entities.length === 0) return []; - - const groups: LocalUpstreamExportEntity[][] = []; - for (let index = 0; index < entities.length; index += maxEntitiesPerChunk) { - groups.push(entities.slice(index, index + maxEntitiesPerChunk)); - } - - return groups.map((group, index) => { - const payload = { - entityKeys: group.map((entity) => entity.record.key), - }; - return { - chunkIndex: index, - totalChunks: groups.length, - byteLength: Buffer.byteLength(canonicalJson(payload)), - sha256: normalizedContentHash(payload), - payload, - }; - }); -} - -function sortJson(value: unknown): unknown { - if (Array.isArray(value)) return value.map(sortJson); - if (typeof value !== "object" || value === null) return value; - return Object.fromEntries( - Object.entries(value as Record) - .sort(([left], [right]) => left.localeCompare(right)) - .map(([key, entry]) => [key, sortJson(entry)]), - ); -} - -async function parseCoordinatorResponse(response: Response): Promise { - const text = await response.text(); - const parsed = text.trim() ? safeParseJson(text) : {}; - if (!response.ok) { - const message = typeof parsed === "object" && parsed !== null && "error" in parsed - ? String((parsed as { error: unknown }).error) - : `Upstream importer request failed with ${response.status}`; - throw new UpstreamImportRequestError(response.status, message, parsed); - } - return parsed; -} - -function safeParseJson(text: string): unknown { - try { - return JSON.parse(text); - } catch { - return text; - } -} diff --git a/cli/src/commands/client/cloud.ts b/cli/src/commands/client/cloud.ts deleted file mode 100644 index 1e3046f14c1..00000000000 --- a/cli/src/commands/client/cloud.ts +++ /dev/null @@ -1,722 +0,0 @@ -import { createHash, generateKeyPairSync, randomBytes, randomUUID, sign } from "node:crypto"; -import { createServer, type Server } from "node:http"; -import { URL } from "node:url"; -import { Command } from "commander"; -import pc from "picocolors"; -import type { - CompanyPortabilityExportResult, - CompanyPortabilityFileEntry, - InstanceExperimentalSettings, -} from "@paperclipai/shared"; -import { openUrl } from "../../client/board-auth.js"; -import { resolvePaperclipInstanceId } from "../../config/home.js"; -import { - addCommonClientOptions, - apiPath, - handleCommandError, - printOutput, - resolveCommandContext, - type BaseClientOptions, -} from "./common.js"; -import { - buildLocalUpstreamExportBundle, - LocalUpstreamPushCoordinator, - normalizedContentHash, - upstreamTransferSchema, - UpstreamImportRequestError, - type LocalUpstreamExportBundle, - type LocalUpstreamExportEntityInput, - type SourceEntityKey, - type UpstreamTransferManifestSource, - type UpstreamTransferManifestTarget, - type UpstreamTransferWarning, -} from "./cloud-transfer.js"; -import { - getCloudConnection, - upsertCloudConnection, - type CloudConnection, - type CloudConnectionTokenRecord, -} from "./cloud-store.js"; - -const CLOUD_SYNC_CONFLICT_EXIT_CODE = 2; -const CLOUD_SYNC_SCHEMA_MISMATCH_EXIT_CODE = 3; -const CLOUD_SYNC_SCOPES = ["upstream_import:preview", "upstream_import:write", "upstream_import:read"]; -const DEVICE_CODE_FALLBACK_EXPIRES_MS = 15 * 60_000; - -interface CloudConnectOptions extends BaseClientOptions { - noBrowser?: boolean; -} - -interface CloudPushOptions extends BaseClientOptions { - company?: string; - remoteUrl?: string; - dryRun?: boolean; - maxEntitiesPerChunk?: number; -} - -interface UpstreamDiscovery { - schema: string; - stack: { - id: string; - slug?: string; - displayName?: string; - companyId: string; - origin: string; - }; - auth: { - pkce?: { - authorizeUrl: string; - tokenUrl: string; - codeChallengeMethod: string; - }; - deviceCode?: { - deviceCodeUrl: string; - verificationUrl: string; - tokenUrl: string; - }; - scopes?: string[]; - }; - transfer: { - supportedSchemaMajor: number; - featureFlags?: string[]; - }; -} - -interface TokenResponse { - accessToken: string; - token: CloudConnectionTokenRecord; - scopes?: string[]; - expiresAt?: string; -} - -class CloudAuthRequestError extends Error { - readonly status: number; - readonly body: unknown; - - constructor(status: number, message: string, body: unknown) { - super(message); - this.status = status; - this.body = body; - } -} - -export function registerCloudCommands(program: Command): void { - const cloud = program.command("cloud").description("Paperclip Cloud upstream sync commands"); - - addCommonClientOptions( - cloud - .command("connect") - .description("Authorize this local instance to push into a Paperclip Cloud stack") - .argument("", "Paperclip Cloud stack URL") - .option("--no-browser", "Use the device-code flow instead of opening a browser", false) - .action(async (remoteUrl: string, opts: CloudConnectOptions) => { - try { - await connectCloud(remoteUrl, opts); - } catch (err) { - handleCommandError(err); - } - }), - ); - - addCommonClientOptions( - cloud - .command("push") - .description("Preview or apply a local company push into the connected Paperclip Cloud stack") - .requiredOption("--company ", "Local company ID to export") - .option("--remote-url ", "Use a specific stored cloud connection") - .option("--dry-run", "Preview without applying", false) - .option("--max-entities-per-chunk ", "Chunk size for upstream uploads", (value) => Number(value), 100) - .action(async (opts: CloudPushOptions) => { - try { - await pushCloud(opts); - } catch (err) { - if (isSchemaMismatchError(err)) { - console.error(pc.red(err instanceof Error ? err.message : String(err))); - process.exitCode = CLOUD_SYNC_SCHEMA_MISMATCH_EXIT_CODE; - return; - } - handleCommandError(err); - } - }), - ); -} - -export async function connectCloud(remoteUrl: string, opts: CloudConnectOptions = {}): Promise { - const ctx = resolveCommandContext(opts); - const discovery = await discoverUpstream(remoteUrl); - assertDiscoveryCompatible(discovery); - const source = createSourceIdentity(); - const token = await authorizeConnection(discovery, source, { - noBrowser: Boolean(opts.noBrowser), - }); - const targetOrigin = discovery.stack.origin.replace(/\/+$/u, ""); - const targetHost = new URL(targetOrigin).host; - const now = new Date().toISOString(); - const connection = upsertCloudConnection({ - id: connectionId(targetOrigin), - remoteUrl, - targetOrigin, - targetHost, - stackId: discovery.stack.id, - stackSlug: discovery.stack.slug ?? null, - stackDisplayName: discovery.stack.displayName ?? null, - targetCompanyId: discovery.stack.companyId, - accessToken: token.accessToken, - token: token.token, - privateKeyPem: source.privateKeyPem, - sourcePublicKey: source.sourcePublicKey, - sourceInstanceId: source.sourceInstanceId, - sourceInstanceFingerprint: source.sourceInstanceFingerprint, - scopes: token.scopes ?? token.token.scopes ?? CLOUD_SYNC_SCOPES, - createdAt: now, - updatedAt: now, - }); - - if (ctx.json) { - printOutput(redactConnection(connection), { json: true }); - } else { - console.log(pc.bold("Connected to Paperclip Cloud")); - console.log(`stack=${connection.stackDisplayName ?? connection.stackSlug ?? connection.stackId}`); - console.log(`origin=${connection.targetOrigin}`); - console.log(`company=${connection.targetCompanyId}`); - } - return connection; -} - -export async function pushCloud(opts: CloudPushOptions): Promise { - const ctx = resolveCommandContext(opts, { requireCompany: false }); - const localCompanyId = requiredString(opts.company, "--company"); - await assertCloudSyncEnabled(ctx.api.get("/api/instance/settings/experimental")); - const connection = getCloudConnection(opts.remoteUrl); - if (!connection) { - throw new Error("No cloud connection found. Run `paperclipai cloud connect ` first."); - } - - const discovery = await discoverUpstream(connection.targetOrigin); - assertDiscoveryCompatible(discovery); - const bundle = await buildBundleFromLocalCompany({ - localCompanyId, - connection, - discovery, - localApi: ctx.api, - maxEntitiesPerChunk: opts.maxEntitiesPerChunk, - mode: opts.dryRun ? "preview" : "apply", - }); - const coordinator = new LocalUpstreamPushCoordinator({ - targetOrigin: connection.targetOrigin, - paperclipCompanyId: connection.targetCompanyId, - headers: ({ method, path }) => cloudProofHeaders(connection, method, path), - }); - - const result = opts.dryRun ? await coordinator.preview(bundle) : await coordinator.apply(bundle); - const runId = getRunId(result); - const events = !opts.dryRun && runId ? await coordinator.events(runId).catch(() => null) : null; - const summary = summarizeResult(result); - const conflictCount = summary.conflict + summary.staleMapping; - - if (ctx.json) { - printOutput({ result, events }, { json: true }); - } else { - console.log(pc.bold(opts.dryRun ? "Cloud Push Preview" : "Cloud Push Applied")); - console.log(`run=${runId ?? "-"}`); - console.log(`manifest=${bundle.manifest.manifestHash}`); - console.log( - `create=${summary.create} update=${summary.update} adopt=${summary.adopt} ` + - `skip=${summary.skip} conflict=${summary.conflict} staleMapping=${summary.staleMapping}`, - ); - printWarnings(result); - printConflicts(result); - printEvents(events); - } - - if (conflictCount > 0) { - process.exitCode = CLOUD_SYNC_CONFLICT_EXIT_CODE; - } - return result; -} - -export async function discoverUpstream(remoteUrl: string): Promise { - const base = new URL(remoteUrl); - const discoveryUrl = new URL("/.well-known/paperclip-upstream", base); - return requestCloudJson(discoveryUrl.toString(), { method: "GET" }); -} - -export function assertDiscoveryCompatible(discovery: UpstreamDiscovery): void { - if (discovery.schema !== "paperclip-upstream-discovery-v1") { - throw new Error("Remote URL is not a Paperclip Cloud upstream target."); - } - if (discovery.transfer.supportedSchemaMajor !== upstreamTransferSchema.major) { - throw new Error( - `Cloud upstream schema mismatch: local major ${upstreamTransferSchema.major}, remote supports ${discovery.transfer.supportedSchemaMajor}.`, - ); - } - if (!discovery.transfer.featureFlags?.includes("cloud_sync")) { - throw new Error("Remote Paperclip Cloud stack does not advertise the cloud_sync transfer flag."); - } -} - -export function resolveDeviceCodeExpiresAt(expiresAt: string | undefined, nowMs = Date.now()): number { - const parsed = typeof expiresAt === "string" ? Date.parse(expiresAt) : NaN; - return Number.isFinite(parsed) ? parsed : nowMs + DEVICE_CODE_FALLBACK_EXPIRES_MS; -} - -export async function buildBundleFromLocalCompany(input: { - localCompanyId: string; - connection: CloudConnection; - discovery: UpstreamDiscovery; - localApi: { - post(path: string, body?: unknown): Promise; - }; - maxEntitiesPerChunk?: number; - mode: "preview" | "apply"; -}): Promise { - const exported = await input.localApi.post( - apiPath`/api/companies/${input.localCompanyId}/export`, - { - include: { - company: true, - agents: true, - projects: true, - issues: true, - skills: true, - }, - expandReferencedSkills: true, - }, - ); - if (!exported) throw new Error("Local company export returned no data."); - - const sourceHash = normalizedContentHash({ - manifest: exported.manifest, - files: exported.files, - }); - const source: UpstreamTransferManifestSource = { - sourceInstanceId: input.connection.sourceInstanceId, - sourceCompanyId: input.localCompanyId, - sourceInstanceKeyFingerprint: input.connection.sourceInstanceFingerprint, - exporterVersion: "paperclipai-cli-cloud-v1", - sourceSchemaVersion: "paperclip-local-portability-v1", - }; - const target: UpstreamTransferManifestTarget = { - targetStackId: input.discovery.stack.id, - targetCompanyId: input.discovery.stack.companyId, - targetOrigin: input.discovery.stack.origin, - supportedSchemaMajor: input.discovery.transfer.supportedSchemaMajor, - }; - const entities = buildEntitiesFromPortableExport(input.localCompanyId, input.connection.sourceInstanceId, exported); - const idempotencyKey = [ - input.mode, - input.connection.sourceInstanceId, - input.localCompanyId, - input.discovery.stack.id, - sourceHash, - ].join(":"); - return buildLocalUpstreamExportBundle({ - source, - target, - runId: `local-${input.mode}-${shortHash(idempotencyKey)}`, - idempotencyKey, - entities, - warnings: exported.warnings.map((message): UpstreamTransferWarning => ({ - code: "local_company_export_warning", - severity: "warning", - message, - })), - featureFlags: ["cloud_sync"], - maxEntitiesPerChunk: input.maxEntitiesPerChunk, - }); -} - -async function authorizeConnection( - discovery: UpstreamDiscovery, - source: ReturnType, - opts: { noBrowser: boolean }, -): Promise { - if (!opts.noBrowser && canOpenBrowser() && discovery.auth.pkce) { - try { - return await authorizeWithBrowser(discovery, source); - } catch (error) { - console.error(pc.yellow(`Browser authorization failed; falling back to device-code flow. ${errorMessage(error)}`)); - } - } - if (!discovery.auth.deviceCode) { - throw new Error("Remote Paperclip Cloud stack does not support device-code authorization."); - } - return authorizeWithDeviceCode(discovery, source, { openBrowser: !opts.noBrowser && canOpenBrowser() }); -} - -async function authorizeWithBrowser( - discovery: UpstreamDiscovery, - source: ReturnType, -): Promise { - const pkce = discovery.auth.pkce; - if (!pkce) throw new Error("Remote did not advertise PKCE authorization."); - const callback = await startPkceCallbackServer(); - const verifier = randomBytes(32).toString("base64url"); - const challenge = createHash("sha256").update(verifier).digest("base64url"); - const state = randomUUID(); - const authorizeUrl = new URL(pkce.authorizeUrl); - authorizeUrl.searchParams.set("redirectUri", callback.redirectUri); - authorizeUrl.searchParams.set("state", state); - authorizeUrl.searchParams.set("codeChallenge", challenge); - authorizeUrl.searchParams.set("codeChallengeMethod", "S256"); - authorizeUrl.searchParams.set("sourceInstanceId", source.sourceInstanceId); - authorizeUrl.searchParams.set("sourceInstanceFingerprint", source.sourceInstanceFingerprint); - authorizeUrl.searchParams.set("sourcePublicKey", source.sourcePublicKey); - authorizeUrl.searchParams.set("scopes", CLOUD_SYNC_SCOPES.join(" ")); - - try { - console.error(`Open this URL to approve cloud sync:\n${authorizeUrl.toString()}`); - if (!(await openUrl(authorizeUrl.toString()))) { - throw new Error("Could not open a browser."); - } - const code = await callback.waitForCode(state); - return requestCloudJson(pkce.tokenUrl, { - method: "POST", - body: JSON.stringify({ - grantType: "authorization_code", - code, - redirectUri: callback.redirectUri, - codeVerifier: verifier, - }), - }); - } finally { - await callback.close(); - } -} - -async function authorizeWithDeviceCode( - discovery: UpstreamDiscovery, - source: ReturnType, - opts: { openBrowser: boolean }, -): Promise { - const device = discovery.auth.deviceCode; - if (!device) throw new Error("Remote did not advertise device-code authorization."); - const response = await requestCloudJson<{ - deviceCode: string; - userCode: string; - verificationUri: string; - expiresAt?: string; - intervalSeconds?: number; - }>(device.deviceCodeUrl, { - method: "POST", - body: JSON.stringify({ - stackId: discovery.stack.id, - sourceInstanceId: source.sourceInstanceId, - sourceInstanceFingerprint: source.sourceInstanceFingerprint, - sourcePublicKey: source.sourcePublicKey, - scopes: CLOUD_SYNC_SCOPES, - }), - }); - console.error(pc.bold("Cloud device authorization required")); - console.error(`Open: ${response.verificationUri}`); - console.error(`Code: ${response.userCode}`); - if (opts.openBrowser) await openUrl(response.verificationUri); - - const expiresAt = resolveDeviceCodeExpiresAt(response.expiresAt); - const intervalMs = Math.max(500, (response.intervalSeconds ?? 5) * 1000); - while (Date.now() < expiresAt) { - await sleep(intervalMs); - try { - return await requestCloudJson(device.tokenUrl, { - method: "POST", - body: JSON.stringify({ - grantType: "device_code", - deviceCode: response.deviceCode, - }), - }); - } catch (error) { - if (error instanceof CloudAuthRequestError && error.body && typeof error.body === "object") { - const code = (error.body as { error?: unknown }).error; - if (code === "authorization_pending") continue; - } - throw error; - } - } - throw new Error("Device-code authorization expired before it was approved."); -} - -function buildEntitiesFromPortableExport( - localCompanyId: string, - sourceInstanceId: string, - exported: CompanyPortabilityExportResult, -): LocalUpstreamExportEntityInput[] { - const companyKey: SourceEntityKey = { - sourceInstanceId, - sourceCompanyId: localCompanyId, - sourceEntityType: "company", - sourceEntityId: localCompanyId, - sourceNaturalKey: exported.manifest.company?.name ?? localCompanyId, - }; - const entities: LocalUpstreamExportEntityInput[] = [ - { - key: companyKey, - body: { - kind: "paperclip_company_portability_manifest", - manifest: exported.manifest, - rootPath: exported.rootPath, - paperclipExtensionPath: exported.paperclipExtensionPath, - fileCount: Object.keys(exported.files).length, - }, - conflictKeys: [`company:${companyKey.sourceNaturalKey ?? localCompanyId}`], - }, - ]; - - for (const [filePath, entry] of Object.entries(exported.files).sort(([left], [right]) => left.localeCompare(right))) { - entities.push({ - key: { - sourceInstanceId, - sourceCompanyId: localCompanyId, - sourceEntityType: "company_setting", - sourceEntityId: shortHash(filePath), - sourceNaturalKey: filePath, - }, - body: { - kind: "paperclip_portable_file", - path: filePath, - entry: normalizePortableFileEntry(entry), - }, - dependencies: [companyKey], - conflictKeys: [`portable_file:${filePath}`], - }); - } - return entities; -} - -function normalizePortableFileEntry(entry: CompanyPortabilityFileEntry): Record { - if (typeof entry === "string") { - return { encoding: "utf8", data: entry }; - } - return { ...entry }; -} - -async function assertCloudSyncEnabled(settingsPromise: Promise): Promise { - const settings = await settingsPromise; - if (settings?.enableCloudSync !== true) { - throw new Error( - "Cloud sync is disabled. Enable the cloud sync experimental setting before running `paperclipai cloud push`.", - ); - } -} - -function cloudProofHeaders(connection: CloudConnection, method: string, pathAndSearch: string): Record { - const timestamp = new Date().toISOString(); - const nonce = randomUUID(); - const payload = [ - method, - connection.targetHost.toLowerCase(), - pathAndSearch, - connection.token.id, - connection.sourceInstanceId, - timestamp, - nonce, - ].join("\n"); - return { - Authorization: `Bearer ${connection.accessToken}`, - "X-Paperclip-Upstream-Source-Instance-Id": connection.sourceInstanceId, - "X-Paperclip-Upstream-Proof-Timestamp": timestamp, - "X-Paperclip-Upstream-Proof-Nonce": nonce, - "X-Paperclip-Upstream-Proof-Signature": sign( - null, - Buffer.from(payload, "utf8"), - connection.privateKeyPem, - ).toString("base64url"), - }; -} - -async function requestCloudJson(url: string, init: RequestInit): Promise { - const headers = new Headers(init.headers); - headers.set("accept", "application/json"); - if (init.body !== undefined && !headers.has("content-type")) { - headers.set("content-type", "application/json"); - } - const response = await fetch(url, { ...init, headers }); - const text = await response.text(); - const parsed = text.trim() ? JSON.parse(text) as unknown : {}; - if (!response.ok) { - const message = typeof parsed === "object" && parsed !== null && "error" in parsed - ? String((parsed as { error: unknown }).error) - : `Cloud request failed with ${response.status}`; - throw new CloudAuthRequestError(response.status, message, parsed); - } - return parsed as T; -} - -function createSourceIdentity() { - const { publicKey, privateKey } = generateKeyPairSync("ed25519"); - const sourcePublicKey = publicKey.export({ type: "spki", format: "pem" }).toString(); - const sourceInstanceFingerprint = `sha256:${createHash("sha256") - .update(publicKey.export({ type: "spki", format: "der" })) - .digest("hex")}`; - return { - sourceInstanceId: `paperclip-local-${resolvePaperclipInstanceId()}`, - sourceInstanceFingerprint, - sourcePublicKey, - privateKeyPem: privateKey.export({ type: "pkcs8", format: "pem" }).toString(), - }; -} - -async function startPkceCallbackServer(): Promise<{ - redirectUri: string; - waitForCode: (state: string) => Promise; - close: () => Promise; -}> { - let resolveCode: ((code: string) => void) | null = null; - let rejectCode: ((error: Error) => void) | null = null; - let expectedState = ""; - const codePromise = new Promise((resolve, reject) => { - resolveCode = resolve; - rejectCode = reject; - }); - const server = createServer((req, res) => { - const url = new URL(req.url ?? "/", "http://127.0.0.1"); - const code = url.searchParams.get("code"); - const state = url.searchParams.get("state"); - if (!code || state !== expectedState) { - res.writeHead(400, { "Content-Type": "text/plain" }); - res.end("Paperclip Cloud authorization failed. You can close this tab."); - rejectCode?.(new Error("Authorization callback was missing a valid code or state.")); - return; - } - res.writeHead(200, { "Content-Type": "text/plain" }); - res.end("Paperclip Cloud authorization complete. You can close this tab."); - resolveCode?.(code); - }); - await listenOnLoopback(server); - const address = server.address(); - if (typeof address !== "object" || !address?.port) { - throw new Error("Failed to start local authorization callback server."); - } - return { - redirectUri: `http://127.0.0.1:${address.port}/cloud/callback`, - waitForCode: (state: string) => { - expectedState = state; - return codePromise; - }, - close: () => closeServer(server), - }; -} - -function listenOnLoopback(server: Server): Promise { - return new Promise((resolve, reject) => { - server.once("error", reject); - server.listen(0, "127.0.0.1", () => { - server.off("error", reject); - resolve(); - }); - }); -} - -function closeServer(server: Server): Promise { - return new Promise((resolve, reject) => { - server.close((error) => error ? reject(error) : resolve()); - }); -} - -function canOpenBrowser(): boolean { - if (process.platform === "darwin" || process.platform === "win32") return true; - return Boolean(process.env.DISPLAY || process.env.WAYLAND_DISPLAY); -} - -function summarizeResult(result: unknown): { - create: number; - update: number; - adopt: number; - skip: number; - conflict: number; - staleMapping: number; -} { - const summary = asRecord(asRecord(result)?.summary); - return { - create: numberValue(summary?.create), - update: numberValue(summary?.update), - adopt: numberValue(summary?.adopt), - skip: numberValue(summary?.skip), - conflict: numberValue(summary?.conflict), - staleMapping: numberValue(summary?.staleMapping), - }; -} - -function printWarnings(result: unknown): void { - const warnings = Array.isArray(asRecord(result)?.warnings) ? asRecord(result)?.warnings as unknown[] : []; - for (const warning of warnings) { - const record = asRecord(warning); - console.log(pc.yellow(`warning=${record?.code ?? "warning"} ${record?.message ?? ""}`.trim())); - } -} - -function printConflicts(result: unknown): void { - const conflicts = Array.isArray(asRecord(result)?.conflicts) ? asRecord(result)?.conflicts as unknown[] : []; - for (const conflict of conflicts.slice(0, 10)) { - const record = asRecord(conflict); - console.log(pc.red(`conflict=${record?.conflictKind ?? "target_conflict"} target=${record?.targetEntityId ?? "-"}`)); - } - if (conflicts.length > 10) console.log(pc.red(`conflicts_truncated=${conflicts.length - 10}`)); -} - -function printEvents(events: unknown): void { - const rows = Array.isArray(asRecord(events)?.events) ? asRecord(events)?.events as unknown[] : []; - for (const row of rows.slice(-10)) { - const event = asRecord(row); - console.log(pc.dim(`event=${event?.action ?? "-"} target=${event?.targetEntityId ?? "-"}`)); - } -} - -function getRunId(result: unknown): string | null { - const run = asRecord(asRecord(result)?.run); - return typeof run?.id === "string" ? run.id : null; -} - -function redactConnection(connection: CloudConnection): Record { - return { - id: connection.id, - remoteUrl: connection.remoteUrl, - targetOrigin: connection.targetOrigin, - stackId: connection.stackId, - targetCompanyId: connection.targetCompanyId, - scopes: connection.scopes, - expiresAt: connection.token.expiresAt, - }; -} - -function connectionId(targetOrigin: string): string { - return `cloud-${shortHash(targetOrigin)}`; -} - -function shortHash(value: string): string { - return createHash("sha256").update(value).digest("hex").slice(0, 16); -} - -function requiredString(value: unknown, label: string): string { - if (typeof value === "string" && value.trim()) return value.trim(); - throw new Error(`${label} is required.`); -} - -function numberValue(value: unknown): number { - return typeof value === "number" && Number.isFinite(value) ? value : 0; -} - -function asRecord(value: unknown): Record | null { - return typeof value === "object" && value !== null && !Array.isArray(value) - ? value as Record - : null; -} - -function isSchemaMismatchError(error: unknown): boolean { - if (error instanceof UpstreamImportRequestError) { - return JSON.stringify(error.body).toLowerCase().includes("schema"); - } - return error instanceof Error && error.message.toLowerCase().includes("schema mismatch"); -} - -function errorMessage(error: unknown): string { - return error instanceof Error ? error.message : String(error); -} - -function sleep(ms: number): Promise { - return new Promise((resolve) => setTimeout(resolve, ms)); -} - -export const cloudCommandExitCodes = { - conflict: CLOUD_SYNC_CONFLICT_EXIT_CODE, - schemaMismatch: CLOUD_SYNC_SCHEMA_MISMATCH_EXIT_CODE, -} as const; diff --git a/cli/src/commands/client/company.ts b/cli/src/commands/client/company.ts index e8368ccf9dc..bd0d162a502 100644 --- a/cli/src/commands/client/company.ts +++ b/cli/src/commands/client/company.ts @@ -15,7 +15,7 @@ import type { import { getTelemetryClient, trackCompanyImported } from "../../telemetry.js"; import { ApiRequestError } from "../../client/http.js"; import { openUrl } from "../../client/board-auth.js"; -import { binaryContentTypeByExtension, readZipArchive } from "./zip.js"; +import { binaryContentTypeByExtension, bytesToPortableFileEntry, isBlobStorePath, readZipArchive } from "./zip.js"; import { addCommonClientOptions, apiPath, @@ -140,16 +140,6 @@ type ImportSelectionState = { skills: Set; }; -function readPortableFileEntry(filePath: string, contents: Buffer): CompanyPortabilityFileEntry { - const contentType = binaryContentTypeByExtension[path.extname(filePath).toLowerCase()]; - if (!contentType) return contents.toString("utf8"); - return { - encoding: "base64", - data: contents.toString("base64"), - contentType, - }; -} - function portableFileEntryToWriteValue(entry: CompanyPortabilityFileEntry): string | Uint8Array { if (typeof entry === "string") return entry; return Buffer.from(entry.data, "base64"); @@ -213,7 +203,7 @@ function shouldIncludePortableFile(filePath: string): boolean { const isMarkdown = baseName.endsWith(".md"); const isPaperclipYaml = baseName === ".paperclip.yaml" || baseName === ".paperclip.yml"; const contentType = binaryContentTypeByExtension[path.extname(baseName).toLowerCase()]; - return isMarkdown || isPaperclipYaml || Boolean(contentType); + return isMarkdown || isPaperclipYaml || Boolean(contentType) || isBlobStorePath(filePath); } function findPortableExtensionPath(files: Record): string | null { @@ -932,7 +922,7 @@ async function collectPackageFiles( if (!entry.isFile()) continue; const relativePath = path.relative(root, absolutePath).replace(/\\/g, "/"); if (!shouldIncludePortableFile(relativePath)) continue; - files[relativePath] = readPortableFileEntry(relativePath, await readFile(absolutePath)); + files[relativePath] = bytesToPortableFileEntry(relativePath, await readFile(absolutePath)); } } diff --git a/cli/src/commands/client/zip.ts b/cli/src/commands/client/zip.ts index b75935e9530..8f7b4db4aba 100644 --- a/cli/src/commands/client/zip.ts +++ b/cli/src/commands/client/zip.ts @@ -1,129 +1,10 @@ -import { inflateRawSync } from "node:zlib"; -import path from "node:path"; -import type { CompanyPortabilityFileEntry } from "@paperclipai/shared"; - -const textDecoder = new TextDecoder(); - -export const binaryContentTypeByExtension: Record = { - ".gif": "image/gif", - ".jpeg": "image/jpeg", - ".jpg": "image/jpeg", - ".png": "image/png", - ".svg": "image/svg+xml", - ".webp": "image/webp", -}; - -function normalizeArchivePath(pathValue: string) { - return pathValue - .replace(/\\/g, "/") - .split("/") - .filter(Boolean) - .join("/"); -} - -function readUint16(source: Uint8Array, offset: number) { - return source[offset]! | (source[offset + 1]! << 8); -} - -function readUint32(source: Uint8Array, offset: number) { - return ( - source[offset]! | - (source[offset + 1]! << 8) | - (source[offset + 2]! << 16) | - (source[offset + 3]! << 24) - ) >>> 0; -} - -function sharedArchiveRoot(paths: string[]) { - if (paths.length === 0) return null; - const firstSegments = paths - .map((entry) => normalizeArchivePath(entry).split("/").filter(Boolean)) - .filter((parts) => parts.length > 0); - if (firstSegments.length === 0) return null; - const candidate = firstSegments[0]![0]!; - return firstSegments.every((parts) => parts.length > 1 && parts[0] === candidate) - ? candidate - : null; -} - -function bytesToPortableFileEntry(pathValue: string, bytes: Uint8Array): CompanyPortabilityFileEntry { - const contentType = binaryContentTypeByExtension[path.extname(pathValue).toLowerCase()]; - if (!contentType) return textDecoder.decode(bytes); - return { - encoding: "base64", - data: Buffer.from(bytes).toString("base64"), - contentType, - }; -} - -async function inflateZipEntry(compressionMethod: number, bytes: Uint8Array) { - if (compressionMethod === 0) return bytes; - if (compressionMethod !== 8) { - throw new Error("Unsupported zip archive: only STORE and DEFLATE entries are supported."); - } - return new Uint8Array(inflateRawSync(bytes)); -} - -export async function readZipArchive(source: ArrayBuffer | Uint8Array): Promise<{ - rootPath: string | null; - files: Record; -}> { - const bytes = source instanceof Uint8Array ? source : new Uint8Array(source); - const entries: Array<{ path: string; body: CompanyPortabilityFileEntry }> = []; - let offset = 0; - - while (offset + 4 <= bytes.length) { - const signature = readUint32(bytes, offset); - if (signature === 0x02014b50 || signature === 0x06054b50) break; - if (signature !== 0x04034b50) { - throw new Error("Invalid zip archive: unsupported local file header."); - } - - if (offset + 30 > bytes.length) { - throw new Error("Invalid zip archive: truncated local file header."); - } - - const generalPurposeFlag = readUint16(bytes, offset + 6); - const compressionMethod = readUint16(bytes, offset + 8); - const compressedSize = readUint32(bytes, offset + 18); - const fileNameLength = readUint16(bytes, offset + 26); - const extraFieldLength = readUint16(bytes, offset + 28); - - if ((generalPurposeFlag & 0x0008) !== 0) { - throw new Error("Unsupported zip archive: data descriptors are not supported."); - } - - const nameOffset = offset + 30; - const bodyOffset = nameOffset + fileNameLength + extraFieldLength; - const bodyEnd = bodyOffset + compressedSize; - if (bodyEnd > bytes.length) { - throw new Error("Invalid zip archive: truncated file contents."); - } - - const rawArchivePath = textDecoder.decode(bytes.slice(nameOffset, nameOffset + fileNameLength)); - const archivePath = normalizeArchivePath(rawArchivePath); - const isDirectoryEntry = /\/$/.test(rawArchivePath.replace(/\\/g, "/")); - if (archivePath && !isDirectoryEntry) { - const entryBytes = await inflateZipEntry(compressionMethod, bytes.slice(bodyOffset, bodyEnd)); - entries.push({ - path: archivePath, - body: bytesToPortableFileEntry(archivePath, entryBytes), - }); - } - - offset = bodyEnd; - } - - const rootPath = sharedArchiveRoot(entries.map((entry) => entry.path)); - const files: Record = {}; - for (const entry of entries) { - const normalizedPath = - rootPath && entry.path.startsWith(`${rootPath}/`) - ? entry.path.slice(rootPath.length + 1) - : entry.path; - if (!normalizedPath) continue; - files[normalizedPath] = entry.body; - } - - return { rootPath, files }; -} +// The node-side portability zip reader lives in @paperclipai/shared so the +// server can consume the same codec (a raw uploaded zip is unzipped into the +// exact `{ rootPath, files }` bundle the inline import source carries). This +// module re-exports it to keep the CLI's existing import paths stable. +export { + binaryContentTypeByExtension, + bytesToPortableFileEntry, + isBlobStorePath, + readZipArchive, +} from "@paperclipai/shared/portability-zip"; diff --git a/cli/src/commands/worktree-lib.ts b/cli/src/commands/worktree-lib.ts index 2be4528e507..0d1d60f65ec 100644 --- a/cli/src/commands/worktree-lib.ts +++ b/cli/src/commands/worktree-lib.ts @@ -197,7 +197,7 @@ export function buildWorktreeConfig(input: { embeddedPostgresDataDir: paths.embeddedPostgresDataDir, embeddedPostgresPort: databasePort, backup: { - enabled: source?.database.backup.enabled ?? true, + enabled: false, intervalMinutes: source?.database.backup.intervalMinutes ?? 60, retentionDays: source?.database.backup.retentionDays ?? 30, dir: paths.backupDir, @@ -258,6 +258,7 @@ export function buildWorktreeEnvEntries( PAPERCLIP_CONFIG: paths.configPath, PAPERCLIP_CONTEXT: paths.contextPath, PAPERCLIP_IN_WORKTREE: "true", + PAPERCLIP_DB_BACKUP_ENABLED: "false", ...(branding?.name ? { PAPERCLIP_WORKTREE_NAME: branding.name } : {}), ...(branding?.color ? { PAPERCLIP_WORKTREE_COLOR: branding.color } : {}), }; diff --git a/cli/src/index.ts b/cli/src/index.ts index 2adb131a2b0..c53454636e2 100644 --- a/cli/src/index.ts +++ b/cli/src/index.ts @@ -22,7 +22,6 @@ import { registerRoutineCommands } from "./commands/routines.js"; import { registerPipelineCommands } from "./commands/pipelines.js"; import { registerFeedbackCommands } from "./commands/client/feedback.js"; import { registerSecretCommands } from "./commands/client/secrets.js"; -import { registerCloudCommands } from "./commands/client/cloud.js"; import { registerSkillsCommands } from "./commands/client/skills.js"; import { registerTeamCommands } from "./commands/client/teams.js"; import { applyDataDirOverride, type DataDirOptionLike } from "./config/data-dir.js"; @@ -181,7 +180,6 @@ registerRoutineCommands(program); registerPipelineCommands(program); registerFeedbackCommands(program); registerSecretCommands(program); -registerCloudCommands(program); registerSkillsCommands(program); registerTeamCommands(program); registerWorktreeCommands(program); diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index c9290369154..0e0fddf80f9 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -355,6 +355,7 @@ This command: - creates an isolated instance under `~/.paperclip-worktrees/instances//` - when run inside a linked git worktree, mirrors the effective git hooks into that worktree's private git dir - picks a free app port and embedded PostgreSQL port +- disables automatic database backups for the isolated instance - by default seeds the isolated DB in `minimal` mode from the current effective Paperclip instance/config (repo-local worktree config when present, otherwise the default instance) via a logical SQL snapshot Seed modes: @@ -374,6 +375,7 @@ Provisioned git worktrees also pause seeded routines that still have enabled sch That repo-local env also sets: - `PAPERCLIP_IN_WORKTREE=true` +- `PAPERCLIP_DB_BACKUP_ENABLED=false` - `PAPERCLIP_WORKTREE_NAME=` - `PAPERCLIP_WORKTREE_COLOR=` @@ -655,6 +657,11 @@ schemas. Defaults: - retain 30 days - backup dir: `~/.paperclip/instances/default/data/backups` +Automatic backups are disabled for isolated worktree instances created with +`paperclipai worktree init` or `paperclipai worktree:make`. Existing worktree +configs are migrated to the disabled setting when their server next starts. The +main/default instance keeps the normal enabled-by-default behavior. + Configure these in: ```sh diff --git a/docker/agent-runtime/README.md b/docker/agent-runtime/README.md index b2d35a26b22..9783495173a 100644 --- a/docker/agent-runtime/README.md +++ b/docker/agent-runtime/README.md @@ -21,6 +21,14 @@ Container images for running coding-agent harnesses in sandboxed environments (f - tini (PID-1 init, ensures signal propagation) - Non-root user `paperclip` (uid/gid 1000) +The NodeSource install puts `node` on the default `PATH`. The agent shim in this +image runs the harness directly with that `PATH`. The shim does not source a +login profile, and the runtime never writes a profile or an rc file. Some +sandbox providers instead wrap each command in a login shell. That shell sources +`/etc/profile` and the user profile files to read an owner-supplied `PATH`. No +exec path sources `nvm`. For the full exec-path contract, see +`packages/plugins/sandbox-providers/SANDBOX-REQUIREMENTS.md`. + **Paperclip Binaries:** - `/usr/local/bin/paperclip-agent-shim`: Go binary compiled from `tools/agent-shim/`. Reads `/run/paperclip/runtime-command.json` and `syscall.Exec`s the harness CLI. diff --git a/docs/api/issues.md b/docs/api/issues.md index a0bff74004e..617b17f5578 100644 --- a/docs/api/issues.md +++ b/docs/api/issues.md @@ -62,7 +62,7 @@ Headers: X-Paperclip-Run-Id: {runId} } ``` -The optional `comment` field adds a comment in the same call. +The optional `comment` field adds a comment in the same call. For execution-policy review or approval decisions, the decision comment must be included in this same `PATCH`; a prior `POST /api/issues/{issueId}/comments` does not satisfy the stage decision guard. Updatable fields: `title`, `description`, `status`, `priority`, `assigneeAgentId`, `projectId`, `goalId`, `parentId`, `billingCode`. diff --git a/docs/guides/agent-developer/comments-and-communication.md b/docs/guides/agent-developer/comments-and-communication.md index 23a77d455a5..fe44cb78d89 100644 --- a/docs/guides/agent-developer/comments-and-communication.md +++ b/docs/guides/agent-developer/comments-and-communication.md @@ -19,6 +19,8 @@ PATCH /api/issues/{issueId} { "status": "done", "comment": "Implemented login endpoint with JWT auth." } ``` +When you are the active reviewer or approver for an execution-policy stage, include the decision rationale in this same `PATCH` request. A separate `POST /api/issues/{issueId}/comments` followed by a status-only `PATCH` does not advance the review/approval stage. + ## Comment Style Use concise markdown with: diff --git a/docs/guides/board-operator/importing-and-exporting.md b/docs/guides/board-operator/importing-and-exporting.md index 02c8cc132f8..1d8e40d241e 100644 --- a/docs/guides/board-operator/importing-and-exporting.md +++ b/docs/guides/board-operator/importing-and-exporting.md @@ -29,6 +29,14 @@ my-company/ - **SKILL.md** files are compatible with the Agent Skills ecosystem. - **.paperclip.yaml** holds Paperclip-specific config (adapter types, env inputs, budgets) as an optional sidecar. +## Export & Import in the App + +Both flows are also available in the web UI as company settings pages: **Export** and **Import** appear in the company settings navigation. + +The **Export** page lets you pick exactly which files go into the bundle before downloading it. Above the file tree it shows a **"Not included in this export"** panel — the export fidelity report — listing data the bundle will not carry (for example attachments, approvals, cost history, or activity log entries), with blocking issues highlighted. + +The **Import** page previews the package, lets you resolve name collisions and adapter assignments, and applies the import. A **"Start imported agents and routines paused"** checkbox (on by default) makes imported agents and routines land paused instead of live. After the import finishes, an **"Activate imported agents and routines"** panel lists everything that was imported paused so you can resume the agents and activate the routines you select — nothing starts running until you say so. + ## Exporting a Company Export a company into a portable folder: @@ -142,6 +150,8 @@ The preview shows: Imported agents always land with timer heartbeats disabled. Assignment/on-demand wake behavior from the package is preserved, but scheduled runs stay off until a board operator re-enables them. +Imports can additionally request `pauseAutomations` (the default in the app's Import page) so imported agents and routines land fully paused. Use the post-import activation panel — or resume the agents and activate the routines individually — when you are ready for them to run. + ### Common Workflows **Clone a company template from GitHub:** @@ -184,11 +194,14 @@ The CLI commands use these API endpoints under the hood: | Action | Endpoint | |--------|----------| | Export company | `POST /api/companies/{companyId}/export` | +| Export fidelity report | `GET /api/companies/{companyId}/export/fidelity` | | Preview import (existing company) | `POST /api/companies/{companyId}/imports/preview` | | Apply import (existing company) | `POST /api/companies/{companyId}/imports/apply` | | Preview import (new company) | `POST /api/companies/import/preview` | | Apply import (new company) | `POST /api/companies/import` | +Import apply requests accept `pauseAutomations: true` to create imported agents and routines in a paused state. + CEO agents can also use the safe import routes (`/imports/preview` and `/imports/apply`) which enforce non-destructive rules: `replace` is rejected, collisions resolve with `rename` or `skip`, and issues are always created as new. ## GitHub Sources diff --git a/docs/guides/execution-policy.md b/docs/guides/execution-policy.md index f12af1cb13c..030798b7118 100644 --- a/docs/guides/execution-policy.md +++ b/docs/guides/execution-policy.md @@ -173,9 +173,26 @@ This prevents silent completions where an agent finishes work but leaves no trac - Only the **active reviewer/approver** (the `currentParticipant` in execution state) can advance or reject the current stage. - Non-participants who attempt to transition the issue receive a `422 Unprocessable Entity` error. - Both approvals and change requests **require a comment** — empty or whitespace-only comments are rejected. +- The decision comment must be included in the same `PATCH /api/issues/{issueId}` request that changes the status. A prior `POST /api/issues/{issueId}/comments` entry remains normal discussion and does not satisfy the review/approval decision guard. ## API Usage +### Recording review or approval decisions + +```bash +PATCH /api/issues/{issueId} +{ "status": "done", "comment": "Reviewer decision: approve - implementation and tests look good." } +``` + +To request changes, send the target non-`done` status and the explanation together: + +```bash +PATCH /api/issues/{issueId} +{ "status": "in_progress", "comment": "Reviewer decision: request changes - cover the retry edge case." } +``` + +Do not split a decision into `POST /comments` followed by a status-only `PATCH`; the runtime only records the decision from the `comment` field on the same status update. + ### Setting an execution policy on issue creation ```bash diff --git a/packages/adapter-utils/src/acpx-engine/execute.test.ts b/packages/adapter-utils/src/acpx-engine/execute.test.ts index c1fe7c2c2d6..12ea70891d9 100644 --- a/packages/adapter-utils/src/acpx-engine/execute.test.ts +++ b/packages/adapter-utils/src/acpx-engine/execute.test.ts @@ -3,7 +3,7 @@ import os from "node:os"; import path from "node:path"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { AcpRuntimeOptions } from "acpx/runtime"; -import type { AdapterRuntimeMcpAccess } from "@paperclipai/adapter-utils"; +import type { AdapterExecutionContext, AdapterRuntimeMcpAccess } from "@paperclipai/adapter-utils"; import { DEFAULT_REMOTE_SANDBOX_ADAPTER_TIMEOUT_SEC, prepareAdapterExecutionTargetRuntime, @@ -167,6 +167,7 @@ async function runExecutor( executionTarget?: Record; runtimeMcp?: AdapterRuntimeMcpAccess; prepareRemoteManagedHome?: AcpxEngineExecutorOptions["prepareRemoteManagedHome"]; + startupTraceContext?: AdapterExecutionContext["startupTraceContext"]; } = {}, ) { const runtimeOptions: Record[] = []; @@ -201,6 +202,7 @@ async function runExecutor( authToken: options.authToken, executionTarget: options.executionTarget, runtimeMcp: options.runtimeMcp, + startupTraceContext: options.startupTraceContext, onLog: async (stream: "stdout" | "stderr", text: string) => { logs.push({ stream, text }); }, @@ -216,6 +218,77 @@ async function runExecutor( return { logs, meta, events, runtimeOptions, configOptions, sessionInputs, result }; } +// A recording span, used only in tests. It captures the span name, the parent +// span (resolved from the explicit parent-context token), the attribute map, +// the terminal status, and whether the span ended. The engine treats it purely +// through the structural `StartupSpan` contract. +interface RecordingSpan { + name: string; + attributes: Record; + parent: RecordingSpan | null; + status: { code: number } | null; + ended: boolean; + setAttribute(key: string, value: string | number | boolean): void; + setStatus(status: { code: number; message?: string }): void; + end(): void; +} + +// Build an in-memory startup trace context that records every span. It models +// the real OTel parenting contract: `startSpan(name, options, context)` reads +// the parent from the explicit `context` token that `contextWithSpan` produced, +// so a test asserts the exact parent of each child without an OTel package or +// ambient async-context propagation. +function createRecordingStartupTrace() { + const spans: RecordingSpan[] = []; + const traceContext = { + tracer: { + startSpan( + name: string, + options?: { attributes?: Record }, + context?: unknown, + ) { + const parent = + context && typeof context === "object" && "span" in context + ? ((context as { span: RecordingSpan }).span ?? null) + : null; + const span: RecordingSpan = { + name, + attributes: { ...(options?.attributes ?? {}) }, + parent, + status: null, + ended: false, + setAttribute(key: string, value: string | number | boolean) { + span.attributes[key] = value; + }, + setStatus(status: { code: number }) { + span.status = { code: status.code }; + }, + end() { + span.ended = true; + }, + }; + spans.push(span); + return span; + }, + }, + contextWithSpan(span: unknown) { + return { span }; + }, + } satisfies AdapterExecutionContext["startupTraceContext"]; + return { traceContext, spans }; +} + +// The closed span-attribute allowlist for a sandbox-start span (Phase 2 + 3). +// A test asserts every recorded attribute key is in this set, so a command, +// path, id, or error-text key can never ride a span. +const ALLOWED_STARTUP_SPAN_ATTRIBUTE_KEYS = new Set([ + "step", + "provider", + "roundTrips", + "providerExecMs", + "providerGetMs", +]); + describe("shared ACPX engine runtime behavior", () => { it("sets Codex model, effort, and fast mode through CODEX_CONFIG without session config calls", async () => { const { configOptions, meta } = await runExecutor({ @@ -798,6 +871,155 @@ describe("shared ACPX engine runtime behavior", () => { expect(fp(rotatedKey)).not.toBe(fp(withKey)); }); + it("busts the session fingerprint when the referenced-project set or pinned checkout changes", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const cwd = path.join(root, "workspace"); + const baseConfig = { agentCommand: "node ./fake-acp.js", stateDir }; + + // A compatible resume reuses the already-staged referenced-project trees, so + // the fingerprint must fold in the referenced-project set. Otherwise a resume + // reuses a stale tree after the set or a project's pinned checkout changes. + const withReferenced = (additional: unknown[]) => ({ + context: { + taskId: "issue-1", + wakeReason: "issue_assigned", + paperclipWorkspace: { cwd, realization: { additional } }, + }, + }); + const projectA = { + path: "/host/project-a", + projectId: "a", + projectWorkspaceId: "ws-a", + repoUrl: "https://example.test/a.git", + repoRef: "ref-a-1", + }; + const projectB = { + path: "/host/project-b", + projectId: "b", + projectWorkspaceId: "ws-b", + repoUrl: "https://example.test/b.git", + repoRef: "ref-b-1", + }; + + const one = await runExecutor(baseConfig, withReferenced([projectA])); + const two = await runExecutor(baseConfig, withReferenced([projectA, projectB])); + const reordered = await runExecutor(baseConfig, withReferenced([projectB, projectA])); + const repointed = await runExecutor( + baseConfig, + withReferenced([projectA, { ...projectB, repoRef: "ref-b-2" }]), + ); + + const fp = (r: { result: { sessionParams?: unknown } }) => + (r.result.sessionParams as { configFingerprint?: string } | undefined)?.configFingerprint; + + // Adding a referenced project changes the set identity, so the fingerprint + // busts and the next launch stages the current set. + expect(fp(one)).toBeDefined(); + expect(fp(two)).not.toBe(fp(one)); + // The identity depends on the set, not the order the records arrive in. + expect(fp(reordered)).toBe(fp(two)); + // Re-pinning one project's checkout ref busts the fingerprint, so the resume + // re-stages instead of reusing a stale tree. + expect(fp(repointed)).not.toBe(fp(two)); + }); + + it("busts the session fingerprint when referenced-project files change at the same host path", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const cwd = path.join(root, "workspace"); + const projectDir = path.join(root, "project-a"); + await fs.mkdir(path.join(projectDir, "src"), { recursive: true }); + await fs.writeFile(path.join(projectDir, "src", "index.ts"), "export const value = 1;\n"); + const baseConfig = { agentCommand: "node ./fake-acp.js", stateDir }; + + // Same host path and same metadata across runs. Only the file bytes change, + // which mirrors a branch that moved or a re-checkout in place. The metadata + // identity stays constant, so without the content signature a compatible + // resume would reuse a stale staged tree. + const referencedProject = { + path: projectDir, + projectId: "a", + projectWorkspaceId: "ws-a", + repoUrl: "https://example.test/a.git", + repoRef: "main", + }; + const withProject = () => ({ + context: { + taskId: "issue-1", + wakeReason: "issue_assigned", + paperclipWorkspace: { cwd, realization: { additional: [referencedProject] } }, + }, + }); + const fp = (r: { result: { sessionParams?: unknown } }) => + (r.result.sessionParams as { configFingerprint?: string } | undefined)?.configFingerprint; + + const before = await runExecutor(baseConfig, withProject()); + // Change the tree content and its size while the host path and metadata stay + // identical. + await fs.writeFile(path.join(projectDir, "src", "index.ts"), "export const value = 2; // changed\n"); + const afterEdit = await runExecutor(baseConfig, withProject()); + // Add a new file to the tree. + await fs.writeFile(path.join(projectDir, "src", "extra.ts"), "export const extra = true;\n"); + const afterAdd = await runExecutor(baseConfig, withProject()); + + expect(fp(before)).toBeDefined(); + // A content-only edit busts the fingerprint, so the resume re-stages. + expect(fp(afterEdit)).not.toBe(fp(before)); + // A new file in the tree busts the fingerprint too. + expect(fp(afterAdd)).not.toBe(fp(afterEdit)); + }); + + it("busts the session fingerprint when referenced-project bytes change but size and mtime stay equal", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const cwd = path.join(root, "workspace"); + const projectDir = path.join(root, "project-a"); + const filePath = path.join(projectDir, "src", "index.ts"); + await fs.mkdir(path.join(projectDir, "src"), { recursive: true }); + // The two contents have the same byte length, so a stat-only signature (size + // and modification time) would collide once the modification time also matches. + const firstContent = "export const value = 1;\n"; + const secondContent = "export const value = 2;\n"; + expect(secondContent.length).toBe(firstContent.length); + await fs.writeFile(filePath, firstContent); + + // A re-checkout can restore the same modification time, so pin it explicitly + // and reapply it after the edit. Only the file bytes change between runs. + const pinnedTime = new Date("2026-01-01T00:00:00.000Z"); + await fs.utimes(filePath, pinnedTime, pinnedTime); + + const referencedProject = { + path: projectDir, + projectId: "a", + projectWorkspaceId: "ws-a", + repoUrl: "https://example.test/a.git", + repoRef: "main", + }; + const withProject = () => ({ + context: { + taskId: "issue-1", + wakeReason: "issue_assigned", + paperclipWorkspace: { cwd, realization: { additional: [referencedProject] } }, + }, + }); + const fp = (r: { result: { sessionParams?: unknown } }) => + (r.result.sessionParams as { configFingerprint?: string } | undefined)?.configFingerprint; + + const baseConfig = { agentCommand: "node ./fake-acp.js", stateDir }; + const before = await runExecutor(baseConfig, withProject()); + // Replace the bytes with an equal-length string and restore the same + // modification time, so path, size, and mtime all stay identical across the two runs. + await fs.writeFile(filePath, secondContent); + await fs.utimes(filePath, pinnedTime, pinnedTime); + const after = await runExecutor(baseConfig, withProject()); + + expect(fp(before)).toBeDefined(); + // The content signature reads bytes, so an equal-size, equal-mtime edit busts + // the fingerprint and the resume re-stages instead of reusing a stale tree. + expect(fp(after)).not.toBe(fp(before)); + }); + it("shapes ACPX session env for remote execution identities", async () => { const root = await makeTempRoot(); const localCwd = path.join(root, "local"); @@ -2780,6 +3002,226 @@ describe("ACPX engine remote session-lifecycle re-staging (PR 3: stage once / re }); }); +describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + async function remoteSandboxTarget(root: string) { + const remoteCwd = path.join(root, "remote-workspace"); + await fs.mkdir(remoteCwd, { recursive: true }); + return { + kind: "remote", + transport: "sandbox", + // A plugin-backed key (not a built-in family). It must never ride a span + // as a raw attribute. + providerKey: "fake-plugin", + remoteCwd, + runner: createLocalSandboxRunner(), + }; + } + + it("emits one root span with a child span per bring-up boundary, each parented to the root", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const localCwd = path.join(root, "worktree"); + const codexHome = path.join(root, "codex-home"); + await fs.mkdir(localCwd, { recursive: true }); + await fs.mkdir(codexHome, { recursive: true }); + const executionTarget = await remoteSandboxTarget(root); + const { traceContext, spans } = createRecordingStartupTrace(); + + await runExecutor( + { + agent: "codex", + agentCommand: "node ./fake-acp.js", + stateDir, + cwd: localCwd, + env: { CODEX_HOME: codexHome }, + }, + { authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext }, + ); + + // Exactly one root span, and it is the bring-up root. + const roots = spans.filter((span) => span.parent === null); + expect(roots).toHaveLength(1); + const rootSpan = roots[0]!; + expect(rootSpan.name).toBe("sandbox.startup"); + expect(rootSpan.ended).toBe(true); + + // A codex bring-up over the remote sandbox lane crosses all 7 boundaries. + const childNames = spans.filter((span) => span !== rootSpan).map((span) => span.name).sort(); + expect(childNames).toEqual( + [ + "acp.handshake", + "bridge.paperclip", + "bridge.process-session", + "codex-home.seed", + "skills.reconcile", + "stage.sync", + "workspace.resolve", + ], + ); + + // Every child parents to the one root and ends. + for (const span of spans) { + if (span === rootSpan) continue; + expect(span.parent, `span "${span.name}" must parent to the root`).toBe(rootSpan); + expect(span.ended, `span "${span.name}" must end`).toBe(true); + } + }); + + it("parents both concurrent bridge spans to the root (neither orphans)", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const localCwd = path.join(root, "worktree"); + await fs.mkdir(localCwd, { recursive: true }); + const executionTarget = await remoteSandboxTarget(root); + const { traceContext, spans } = createRecordingStartupTrace(); + + await runExecutor( + { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd }, + { authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext }, + ); + + const rootSpan = spans.find((span) => span.name === "sandbox.startup" && span.parent === null); + expect(rootSpan).toBeTruthy(); + const paperclip = spans.find((span) => span.name === "bridge.paperclip"); + const processSession = spans.find((span) => span.name === "bridge.process-session"); + expect(paperclip?.parent).toBe(rootSpan); + expect(processSession?.parent).toBe(rootSpan); + }); + + it("keeps every span attribute inside the closed allowlist (no command/path/id keys)", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const localCwd = path.join(root, "worktree"); + await fs.mkdir(localCwd, { recursive: true }); + const executionTarget = await remoteSandboxTarget(root); + const { traceContext, spans } = createRecordingStartupTrace(); + + await runExecutor( + { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd }, + { authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext }, + ); + + expect(spans.length).toBeGreaterThan(0); + for (const span of spans) { + for (const [key, value] of Object.entries(span.attributes)) { + expect( + ALLOWED_STARTUP_SPAN_ATTRIBUTE_KEYS.has(key), + `span "${span.name}" set a non-allowlisted attribute "${key}"`, + ).toBe(true); + // No non-finite numeric attribute (no NaN, no Infinity). + if (typeof value === "number") { + expect(Number.isFinite(value), `attribute "${key}" must be finite`).toBe(true); + } + } + } + }); + + it("closes the root span with error status when the bring-up handshake fails", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const localCwd = path.join(root, "worktree"); + await fs.mkdir(localCwd, { recursive: true }); + const executionTarget = await remoteSandboxTarget(root); + const { traceContext, spans } = createRecordingStartupTrace(); + + const execute = createAcpxEngineExecutor({ + createRuntime: () => + ({ + ensureSession: async () => { + throw new Error("handshake boom"); + }, + startTurn: () => ({ + events: (async function* () {})(), + result: Promise.resolve({ status: "failed" }), + cancel: async () => {}, + }), + setConfigOption: async () => {}, + close: async () => {}, + }) as never, + }); + + const result = await execute({ + runId: "run-handshake-fail", + agent: { id: "agent-1", companyId: "company-1" }, + runtime: {}, + config: { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd }, + context: {}, + authToken: "real-run-jwt", + executionTarget, + startupTraceContext: traceContext, + onLog: async () => {}, + onMeta: async () => {}, + onEvent: async () => {}, + } as never); + + expect(result.exitCode).toBe(1); + const rootSpan = spans.find((span) => span.name === "sandbox.startup" && span.parent === null); + expect(rootSpan).toBeTruthy(); + expect(rootSpan!.ended).toBe(true); + // `2` is `SpanStatusCode.ERROR`. + expect(rootSpan!.status?.code).toBe(2); + }); + + it("opens no span and does not throw when no trace context is injected", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const localCwd = path.join(root, "worktree"); + await fs.mkdir(localCwd, { recursive: true }); + const executionTarget = await remoteSandboxTarget(root); + + // runExecutor asserts exitCode 0 internally; the run must complete with no + // injected trace context (the default no-op path). + const { result } = await runExecutor( + { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd }, + { authToken: "real-run-jwt", executionTarget }, + ); + expect(result.exitCode).toBe(0); + }); + + it("opens no span for a local run, even when a trace context is injected", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const localCwd = path.join(root, "worktree"); + await fs.mkdir(localCwd, { recursive: true }); + const { traceContext, spans } = createRecordingStartupTrace(); + + // A local run has no execution target. The `sandbox.startup` span names a + // sandbox bring-up, so a local run must stay out of sandbox telemetry. + const { result } = await runExecutor( + { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd }, + { authToken: "real-run-jwt", startupTraceContext: traceContext }, + ); + expect(result.exitCode).toBe(0); + expect(spans).toHaveLength(0); + }); + + it("opens no span for an SSH run, even when a trace context is injected", async () => { + const root = await makeTempRoot(); + const stateDir = path.join(root, "state"); + const localCwd = path.join(root, "worktree"); + const remoteCwd = path.join(root, "remote-workspace"); + await fs.mkdir(localCwd, { recursive: true }); + const { traceContext, spans } = createRecordingStartupTrace(); + + // An SSH run is remote but is not a sandbox, so it also stays out of sandbox + // telemetry. + const { result } = await runExecutor( + { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd }, + { + authToken: "real-run-jwt", + executionTarget: { kind: "remote", transport: "ssh", remoteCwd }, + startupTraceContext: traceContext, + }, + ); + expect(result.exitCode).toBe(0); + expect(spans).toHaveLength(0); + }); +}); + describe("ACPX engine per-step startup timing (run.startup.step events)", () => { beforeEach(() => { vi.clearAllMocks(); diff --git a/packages/adapter-utils/src/acpx-engine/execute.ts b/packages/adapter-utils/src/acpx-engine/execute.ts index 67c4aae5a2c..e3321c02ede 100644 --- a/packages/adapter-utils/src/acpx-engine/execute.ts +++ b/packages/adapter-utils/src/acpx-engine/execute.ts @@ -28,6 +28,7 @@ import { type AdapterExecutionTargetTimeoutResolution, type AdapterManagedRuntimeAsset, type PreparedAdapterExecutionTargetRuntime, + type SandboxAdditionalSource, } from "@paperclipai/adapter-utils/execution-target"; import { DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE, @@ -82,7 +83,15 @@ import { DEFAULT_ACP_ENGINE_TIMEOUT_SEC, DEFAULT_ACP_ENGINE_WARM_HANDLE_IDLE_MS, } from "./constants.js"; -import { measureStartupStep, type StartupStepMeasureOptions } from "./startup-timing.js"; +import { + measureStartupStep, + NOOP_STARTUP_SPAN, + NOOP_STARTUP_TRACE_CONTEXT, + type StartupSpan, + type StartupSpanContext, + type StartupStepMeasureOptions, + type StartupTraceContext, +} from "./startup-timing.js"; import type { CommandManagedRuntimeRunner } from "../command-managed-runtime.js"; const defaultModuleDir = path.dirname(fileURLToPath(import.meta.url)); @@ -418,6 +427,83 @@ function shortHash(value: unknown): string { return createHash("sha256").update(stableJson(value)).digest("hex").slice(0, 16); } +// Directory names the staging path never ships for a referenced project (heavy +// build/cache output and git history). The content signature skips them so it +// reflects only the staged tree and never reads their bytes. Keep this set equal +// to the staging excludes in the sandbox and remote runtimes. +const REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS = new Set([ + "node_modules", + "vendor", + "dist", + "build", + "out", + "coverage", + ".next", + ".turbo", + ".cache", + ".git", +]); + +/** + * Content signature of a referenced-project host tree for the session fingerprint. + * + * The staged-runtime cache reuses an already-staged referenced-project tree on a + * compatible resume and does not re-sync it. Referenced-project metadata (id, host + * path, workspace id, repo url, pinned ref) can stay identical while the files at + * that host path change: a branch moved to a new commit, a re-checkout in place, or + * a dirty worktree. So the metadata identity alone lets a resume serve a stale tree. + * This signature folds the tree's own content state into the identity. + * + * The walk reads each file's relative path and bytes and folds them into the hash. + * It reads bytes, not only file stats. A stat-only signature (size and modification + * time) collides when an edit keeps the byte length and the modification time — a + * re-checkout that restores the same size and timestamp. The byte hash busts on any + * content change, so the fingerprint busts and the next launch stages the current + * tree. The walk skips the heavy build, cache, and git directories the staging path + * never ships, and records a symlink by its target text without following it. On a + * read error the function returns a stable marker, so the fingerprint does not churn + * while staging surfaces the real error. The walk runs only when the run carries + * referenced projects (the multi-project sync path). + */ +async function referencedSourceContentSignature(localPath: string): Promise { + const hash = createHash("sha256"); + const walk = async (relative: string): Promise => { + const current = relative ? path.join(localPath, relative) : localPath; + const dirents = await fs.readdir(current, { withFileTypes: true }); + dirents.sort((left, right) => (left.name < right.name ? -1 : left.name > right.name ? 1 : 0)); + for (const dirent of dirents) { + const next = relative ? path.posix.join(relative, dirent.name) : dirent.name; + if (dirent.isDirectory()) { + if (REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS.has(dirent.name)) { + continue; + } + await walk(next); + continue; + } + const absolute = path.join(localPath, next); + const stats = await fs.lstat(absolute); + if (stats.isSymbolicLink()) { + const target = await fs.readlink(absolute); + hash.update(`symlink:${next}:${target}\n`); + continue; + } + if (!stats.isFile()) { + hash.update(`other:${next}:${stats.mode}\n`); + continue; + } + hash.update(`file:${next}:${stats.size}\n`); + hash.update(await fs.readFile(absolute)); + hash.update("\n"); + } + }; + try { + await walk(""); + } catch (error) { + return `unreadable:${String(error)}`; + } + return hash.digest("hex").slice(0, 16); +} + function defaultPaperclipInstanceDir(): string { const home = process.env.PAPERCLIP_HOME?.trim() || path.join(os.homedir(), ".paperclip"); const instanceId = process.env.PAPERCLIP_INSTANCE_ID?.trim() || "default"; @@ -1209,6 +1295,11 @@ async function stageAcpRemoteRuntime(input: { workspaceRemoteDir?: string; timeoutSec: number; assets?: AdapterManagedRuntimeAsset[]; + // Referenced (additional) projects to stage into the sandbox as plain, + // read-only trees alongside the anchor workspace. Empty unless run prep + // resolved referenced projects (gated upstream), so the anchor-only path is + // unchanged. + additionalSources?: SandboxAdditionalSource[]; onLog: AdapterExecutionContext["onLog"]; onRuntimeProgress: AdapterExecutionContext["onRuntimeProgress"]; }): Promise { @@ -1224,6 +1315,9 @@ async function stageAcpRemoteRuntime(input: { workspaceLocalDir: input.workspaceLocalDir, ...(input.workspaceRemoteDir ? { workspaceRemoteDir: input.workspaceRemoteDir } : {}), ...(input.assets && input.assets.length > 0 ? { assets: input.assets } : {}), + ...(input.additionalSources && input.additionalSources.length > 0 + ? { additionalSources: input.additionalSources } + : {}), onProgress: (line) => input.onLog("stdout", line), onRuntimeProgress: input.onRuntimeProgress, }); @@ -1249,6 +1343,10 @@ async function buildRuntime(input: { ctx: AdapterExecutionContext; engine: AcpxEngineSettings; deps: AcpxEngineExecutorOptions; + // The injected tracer plus the root-span parent-context token. Merged into + // every startup-step option set, so each boundary span parents to the one + // root span (`sandbox.startup`) that the executor opens. + spanParent: Pick; }): Promise { const { runId, agent, config, context, authToken } = input.ctx; // Injectable monotonic clock for per-step startup timing. Hoisted above the @@ -1271,6 +1369,43 @@ async function buildRuntime(input: { const useConfiguredInsteadOfAgentHome = workspaceSource === "agent_home" && configuredCwd.length > 0; const effectiveWorkspaceCwd = useConfiguredInsteadOfAgentHome ? "" : workspaceCwd; const cwd = effectiveWorkspaceCwd || configuredCwd || process.cwd(); + // Referenced (additional) projects to stage into the sandbox alongside the + // anchor workspace, read from the workspace realization record. The list is + // empty unless run prep resolved referenced projects — gated upstream by the + // multi-project workspace-sync kill-switch — so the anchor-only path is + // unchanged. + const realizationContext = parseObject(workspaceContext.realization); + const additionalSourceRecords = ( + Array.isArray(realizationContext.additional) ? realizationContext.additional : [] + ).map((entry) => parseObject(entry)); + const additionalSources: SandboxAdditionalSource[] = additionalSourceRecords + .map((entry) => ({ localPath: asString(entry.path, ""), projectId: asString(entry.projectId, "") })) + .filter((entry) => entry.localPath.length > 0 && entry.projectId.length > 0); + // Stable identity of the referenced-project set for the session fingerprint. + // The staged-runtime cache reuses already-staged referenced-project trees on a + // compatible resume, so the fingerprint must change when the set OR a project's + // pinned checkout changes. Without this, a resume reuses a stale staged tree. + // Fold in each project's id, host path, workspace id, and pinned ref; sort by + // projectId so the identity depends on the set, not the record order. + const additionalSourcesIdentityBase = additionalSourceRecords + .map((entry) => ({ + projectId: asString(entry.projectId, ""), + localPath: asString(entry.path, ""), + projectWorkspaceId: asString(entry.projectWorkspaceId, ""), + repoUrl: asString(entry.repoUrl, ""), + repoRef: asString(entry.repoRef, ""), + })) + .filter((entry) => entry.localPath.length > 0 && entry.projectId.length > 0) + .sort((a, b) => (a.projectId < b.projectId ? -1 : a.projectId > b.projectId ? 1 : 0)); + // Metadata alone does not change on a content-only checkout change (same host + // path and pinned ref, new file bytes). Fold in each tree's content signature so + // a file add, remove, or edit busts the fingerprint and the resume re-stages. + const additionalSourcesIdentity = await Promise.all( + additionalSourcesIdentityBase.map(async (entry) => ({ + ...entry, + contentSignature: await referencedSourceContentSignature(entry.localPath), + })), + ); const executionTarget = readAdapterExecutionTarget({ executionTarget: input.ctx.executionTarget, legacyRemoteExecution: input.ctx.executionTransport?.remoteExecution, @@ -1287,17 +1422,23 @@ async function buildRuntime(input: { // and emits the per-step delta. Empty when there is no runner (local runs, // the runner-less ACP→CLI fallback, or an SSH runner that does not // instrument the seam), so those steps simply omit the fields. - const stepMetrics = buildStartupStepMetrics( - executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" - ? executionTarget.runner - : undefined, - ); + // Merge the injected tracer + root parent-context into every step option set, + // so each boundary span parents to the root span. With no injected trace + // context both fields are no-ops and the span path stays inert. + const stepMetrics: StartupStepMeasureOptions = { + ...buildStartupStepMetrics( + executionTarget?.kind === "remote" && executionTarget.transport === "sandbox" + ? executionTarget.runner + : undefined, + ), + ...input.spanParent, + }; // The two bridge-start steps intentionally overlap, so their runner counters // would double-count each other if we sampled them here. Keep the shared // counter attribution on the sequential startup phases only; the concurrent - // bridge steps still emit duration telemetry, just not misleading per-step - // round-trip/provider deltas. - const concurrentBridgeStepMetrics: StartupStepMeasureOptions = {}; + // bridge steps still emit duration telemetry (and a span), just not + // misleading per-step round-trip/provider deltas. + const concurrentBridgeStepMetrics: StartupStepMeasureOptions = { ...input.spanParent }; const shapedWorkspaceEnv = shapePaperclipWorkspaceEnvForExecution({ workspaceCwd: effectiveWorkspaceCwd, workspaceWorktreePath, @@ -1553,6 +1694,11 @@ async function buildRuntime(input: { requestedThinkingEffort, fastMode, remoteExecutionIdentity, + // Referenced-project set + pinned-checkout identity. A change here (a project + // added, removed, or re-pinned) invalidates a warm/resumable session so the + // next launch stages the current referenced-project trees instead of reusing + // a stale staged tree. + additionalSourcesIdentity, skillsIdentity, skillPromptInstructions, paperclipClaudeSettings: paperclipClaudeSettings @@ -1681,6 +1827,7 @@ async function buildRuntime(input: { workspaceRemoteDir: sessionCwd, timeoutSec, assets, + additionalSources, onLog: input.ctx.onLog, onRuntimeProgress: input.ctx.onRuntimeProgress, }); @@ -2658,6 +2805,54 @@ function warmHandleMatches( return entry !== undefined && entry.runtime === runtime && entry.handle === handle; } +/** The stable name of the one root span for a sandbox bring-up. It is a fixed + * low-cardinality constant, never derived from run/user data. */ +const STARTUP_ROOT_SPAN_NAME = "sandbox.startup"; + +/** + * Open the one root span for a sandbox bring-up and return its parent-context + * token plus a guarded `end`. The span parents every startup boundary span: + * the engine forwards `parentContext` to each `measureStartupStep` call. The + * `end` closure runs at most once (bring-up complete OR a bring-up failure) and + * swallows every tracer error, so observability never changes startup control + * flow. With no injected trace context, the tracer is a no-op and the span is + * a no-op. + */ +function openStartupRootSpan(tracing: StartupTraceContext): { + parentContext: StartupSpanContext; + end: (failed: boolean) => void; +} { + let span: StartupSpan; + try { + span = tracing.tracer.startSpan(STARTUP_ROOT_SPAN_NAME); + } catch { + span = NOOP_STARTUP_SPAN; + } + let parentContext: StartupSpanContext; + try { + parentContext = tracing.contextWithSpan(span); + } catch { + parentContext = undefined; + } + let ended = false; + return { + parentContext, + end: (failed: boolean) => { + if (ended) return; + ended = true; + try { + // `2` is `SpanStatusCode.ERROR`. `adapter-utils` stays OTel-free, so it + // uses the numeric value that a real injected span reads as the error + // status. + if (failed) span.setStatus({ code: 2 }); + span.end(); + } catch { + // Observability must not change startup control flow. + } + }, + }; +} + export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { const createRuntime = deps.createRuntime ?? createAcpRuntime; const now = deps.now ?? (() => Date.now()); @@ -2688,7 +2883,39 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { now, idleMs: warmIdleMs, }); - const prepared = await buildRuntime({ ctx, engine, deps }); + // The `sandbox.startup` span names a sandbox bring-up. It must not cover a + // local or SSH run: those runs have no sandbox, so they stay out of sandbox + // telemetry. Open the real root span only when the target is a remote + // sandbox; every other target forces the no-op trace context, so the whole + // startup span path stays inert regardless of the injected context. + const startupExecutionTarget = readAdapterExecutionTarget({ + executionTarget: ctx.executionTarget, + legacyRemoteExecution: ctx.executionTransport?.remoteExecution, + }); + const targetsRemoteSandbox = + startupExecutionTarget?.kind === "remote" && startupExecutionTarget.transport === "sandbox"; + // Open the one root span for this bring-up. It spans `buildRuntime` through + // `acp.handshake`, so every startup boundary span parents to it. `spanParent` + // carries the injected tracer + the root parent-context token into each + // `measureStartupStep` call. With no injected trace context the whole path + // is a no-op. `endRootSpan` runs exactly once — at bring-up completion or on + // a bring-up failure. + const tracing = + targetsRemoteSandbox && ctx.startupTraceContext + ? ctx.startupTraceContext + : NOOP_STARTUP_TRACE_CONTEXT; + const rootSpan = openStartupRootSpan(tracing); + const spanParent: Pick = { + tracer: tracing.tracer, + parentContext: rootSpan.parentContext, + }; + let prepared: AcpxPreparedRuntime; + try { + prepared = await buildRuntime({ ctx, engine, deps, spanParent }); + } catch (err) { + rootSpan.end(true); + throw err; + } // State the effective wall-clock timeout and its source up front so a // later timeout is diagnosable from the run log alone. Goes to stderr: // the acpx stdout log stream carries JSON acpx.* event payloads and must @@ -2816,6 +3043,8 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { } } } catch (err) { + // Bring-up failed at the handshake — close the root span with error status. + rootSpan.end(true); const { classified, message } = await emitAcpxFailure({ ctx, prepared, @@ -2839,6 +3068,8 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { } if (!handle) { + // Bring-up produced no session handle — close the root span with error status. + rootSpan.end(true); await discardStagedRuntime({ handles: stagedRuntimes, prepared }); await cleanupRemoteBridges(prepared); return { @@ -2853,6 +3084,10 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) { summary: "ACPX did not return a runtime session handle.", }; } + // Bring-up is complete: the session handle is established. Close the root + // span here, so it covers `buildRuntime` through `acp.handshake` and no + // further. The agent turn runs after and is out of the startup root's scope. + rootSpan.end(false); const sessionHandle = handle; try { await applySessionConfigOptions({ diff --git a/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts b/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts index 70a51899a7a..650da20739c 100644 --- a/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts +++ b/packages/adapter-utils/src/acpx-engine/startup-timing.test.ts @@ -1,6 +1,49 @@ import { describe, expect, it, vi } from "vitest"; import type { AdapterRuntimeEvent } from "../types.js"; -import { measureStartupStep } from "./startup-timing.js"; +import type { StartupSpan, StartupTracer } from "./startup-timing.js"; +import { measureStartupStep, normalizeProviderFamily } from "./startup-timing.js"; + +/** + * A recording span for the mock tracer. It captures the attribute set, the + * status, and the end count so a test can assert the emitted span shape. + */ +class MockSpan implements StartupSpan { + readonly attributes: Record = {}; + status: { code: number; message?: string } | undefined; + endCount = 0; + + constructor( + readonly name: string, + initial: Record | undefined, + ) { + if (initial) Object.assign(this.attributes, initial); + } + + setAttribute(key: string, value: string | number | boolean): void { + this.attributes[key] = value; + } + + setStatus(status: { code: number; message?: string }): void { + this.status = status; + } + + end(): void { + this.endCount += 1; + } +} + +/** A recording tracer that keeps every span it opens for assertions. */ +function makeMockTracer(): { tracer: StartupTracer; spans: MockSpan[] } { + const spans: MockSpan[] = []; + const tracer: StartupTracer = { + startSpan(name, options) { + const span = new MockSpan(name, options?.attributes); + spans.push(span); + return span; + }, + }; + return { tracer, spans }; +} describe("measureStartupStep", () => { it("emits one run.startup.step event with the step name and measured durationMs", async () => { @@ -193,4 +236,165 @@ describe("measureStartupStep", () => { }), ).rejects.toBe(boom); }); + + it("opens one span and ends it once for a normal step", async () => { + const { tracer, spans } = makeMockTracer(); + const onEvent = vi.fn(async () => {}); + + await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", { + tracer, + }); + + expect(spans).toHaveLength(1); + expect(spans[0]!.name).toBe("stage.sync"); + expect(spans[0]!.attributes.step).toBe("stage.sync"); + expect(spans[0]!.endCount).toBe(1); + }); + + it("ends the span and sets an error status when fn throws, then re-throws", async () => { + const { tracer, spans } = makeMockTracer(); + const onEvent = vi.fn(async () => {}); + const boom = new Error("step failed"); + + await expect( + measureStartupStep({ onEvent }, () => 0, "acp.handshake", async () => { + throw boom; + }, { tracer }), + ).rejects.toBe(boom); + + expect(spans).toHaveLength(1); + expect(spans[0]!.endCount).toBe(1); + // SpanStatusCode.ERROR === 2 in @opentelemetry/api. + expect(spans[0]!.status?.code).toBe(2); + }); + + it("sets the same roundTrips / providerExecMs / providerGetMs deltas on the payload and the span", async () => { + let t = 0; + const now = () => t; + let execCount = 5; + let execMs = 100; + let getMs = 40; + const events: AdapterRuntimeEvent[] = []; + const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => { + events.push(event); + }); + const { tracer, spans } = makeMockTracer(); + + await measureStartupStep({ onEvent }, now, "stage.sync", async () => { + t = 90; + execCount += 3; + execMs += 600; + getMs += 15; + return "ok"; + }, { + tracer, + roundTrips: () => execCount, + providerExecMs: () => execMs, + providerGetMs: () => getMs, + }); + + const payload = events[0]!.payload as Record; + expect(payload.roundTrips).toBe(3); + expect(payload.providerExecMs).toBe(600); + expect(payload.providerGetMs).toBe(15); + // The span carries the identical deltas — one build block feeds both. + expect(spans[0]!.attributes.roundTrips).toBe(3); + expect(spans[0]!.attributes.providerExecMs).toBe(600); + expect(spans[0]!.attributes.providerGetMs).toBe(15); + }); + + it("sets no span attribute (and no payload field) when a reader returns undefined", async () => { + const events: AdapterRuntimeEvent[] = []; + const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => { + events.push(event); + }); + const { tracer, spans } = makeMockTracer(); + + await measureStartupStep({ onEvent }, () => 0, "workspace.resolve", async () => "ok", { + tracer, + // A reader may return undefined when the counter is unavailable. The guard + // must omit the attribute rather than emit NaN or 0. + roundTrips: () => undefined as unknown as number, + providerExecMs: () => undefined as unknown as number, + }); + + expect(spans[0]!.attributes).not.toHaveProperty("roundTrips"); + expect(spans[0]!.attributes).not.toHaveProperty("providerExecMs"); + expect(Object.values(spans[0]!.attributes).some((v) => Number.isNaN(v))).toBe(false); + const payload = events[0]!.payload as Record; + expect(payload).not.toHaveProperty("roundTrips"); + expect(payload).not.toHaveProperty("providerExecMs"); + }); + + it("normalizes a plugin-backed provider key to plugin and keeps a built-in family as-is", async () => { + const onEvent = vi.fn(async () => {}); + + const custom = makeMockTracer(); + await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", { + tracer: custom.tracer, + provider: "acme-cloud-runner", + }); + expect(custom.spans[0]!.attributes.provider).toBe("plugin"); + + const builtIn = makeMockTracer(); + await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", { + tracer: builtIn.tracer, + provider: "daytona", + }); + expect(builtIn.spans[0]!.attributes.provider).toBe("daytona"); + }); + + it("normalizeProviderFamily maps every non-built-in key to plugin", () => { + for (const key of ["daytona", "kubernetes", "e2b", "cloudflare", "exe-dev", "modal", "novita"]) { + expect(normalizeProviderFamily(key)).toBe(key); + } + for (const key of ["acme", "my-plugin", "", "DAYTONA", undefined]) { + expect(normalizeProviderFamily(key)).toBe("plugin"); + } + }); + + it("emits exactly the allowlisted span-attribute key set and no command / path / ID / error-text key", async () => { + const onEvent = vi.fn(async () => {}); + const { tracer, spans } = makeMockTracer(); + + await measureStartupStep({ onEvent }, () => 0, "acp.handshake", async () => "ok", { + tracer, + provider: "daytona", + roundTrips: () => 3, + providerExecMs: () => 600, + providerGetMs: () => 15, + // extra() carries caller-measured numbers into the EVENT payload only. + // It must never widen the span-attribute set. + extra: () => ({ createRuntimeMs: 12, ensureSessionMs: 6988 }), + }); + + expect(Object.keys(spans[0]!.attributes).sort()).toEqual( + ["provider", "providerExecMs", "providerGetMs", "roundTrips", "step"], + ); + // extra() keys stay off the span. + expect(spans[0]!.attributes).not.toHaveProperty("createRuntimeMs"); + expect(spans[0]!.attributes).not.toHaveProperty("ensureSessionMs"); + // No free-form identifier / command / path key leaks in. The pattern uses + // no `i` flag, so the camelCase `Id` matches `runId` / `userId` but not the + // "id" inside the allowlisted `provider`. + for (const key of Object.keys(spans[0]!.attributes)) { + expect(key).not.toMatch(/command|args|env|stdout|stderr|path|url|repo|ref|branch|Id|_id|error|message/); + } + }); + + it("uses a no-op tracer by default so a call without a tracer changes nothing", async () => { + const events: AdapterRuntimeEvent[] = []; + const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => { + events.push(event); + }); + + // No tracer supplied. The helper must still emit the event and return the + // value without throwing. + const result = await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", { + roundTrips: () => 3, + }); + + expect(result).toBe("ok"); + expect(events[0]!.payload).toMatchObject({ step: "stage.sync" }); + }); }); diff --git a/packages/adapter-utils/src/acpx-engine/startup-timing.ts b/packages/adapter-utils/src/acpx-engine/startup-timing.ts index 7e3f55941dd..434e607cd84 100644 --- a/packages/adapter-utils/src/acpx-engine/startup-timing.ts +++ b/packages/adapter-utils/src/acpx-engine/startup-timing.ts @@ -9,6 +9,156 @@ import type { AdapterExecutionContext, AdapterRuntimeEvent } from "../types.js"; */ export const RUN_STARTUP_STEP_EVENT_TYPE = "run.startup.step"; +/** + * The public built-in sandbox provider families. A key in this set is safe to + * emit as a low-cardinality span attribute. Any other key is operator-defined + * (plugin-backed) and unbounded, so `normalizeProviderFamily` maps it to the + * generic value `plugin`. Keep this list closed and small. + */ +const BUILT_IN_PROVIDER_FAMILIES: ReadonlySet = new Set([ + "daytona", + "kubernetes", + "e2b", + "cloudflare", + "exe-dev", + "modal", + "novita", +]); + +/** The generic family for any provider key outside the built-in list. */ +const PLUGIN_PROVIDER_FAMILY = "plugin"; + +/** + * Map a raw provider key to a low-cardinality public family. Return the key + * unchanged when it is a built-in family. Return `plugin` for every other + * value, so an operator-defined plugin key never becomes an unbounded span + * attribute. A missing or empty key also maps to `plugin`. + */ +export function normalizeProviderFamily(key: string | undefined): string { + if (key && BUILT_IN_PROVIDER_FAMILIES.has(key)) return key; + return PLUGIN_PROVIDER_FAMILY; +} + +/** + * The value of `SpanStatusCode.ERROR` in `@opentelemetry/api`. `adapter-utils` + * stays OTel-free, so the timing helper uses the numeric value directly. A real + * injected OTel span reads it as the error status. + */ +const SPAN_STATUS_CODE_ERROR = 2; + +/** + * A minimal, OTel-free span contract. The server injects a real + * `@opentelemetry/api` span, which satisfies this shape structurally. The + * default is a no-op span, so a step with no injected tracer changes nothing. + */ +export interface StartupSpan { + setAttribute(key: string, value: string | number | boolean): void; + setStatus(status: { code: number; message?: string }): void; + end(): void; +} + +/** + * An opaque parent-context token. The server builds it from the OTel + * `@opentelemetry/api` `context` / `trace` helpers. `adapter-utils` never reads + * it; it only forwards it to `startSpan`, so this package stays OTel-free. A + * child span opened with this token parents to the span the token carries. + */ +export type StartupSpanContext = unknown; + +/** + * A minimal, OTel-free tracer contract. The server injects a real + * `@opentelemetry/api` tracer, which satisfies this shape structurally. The + * `startSpan` signature is a subset of the OTel one, so a real tracer is + * assignable here. The optional third argument is the explicit parent context: + * a real OTel `startSpan(name, options, context)` parents the new span to the + * span that `context` carries. `adapter-utils` passes it through as an opaque + * token, so parenting never depends on ambient async-context propagation. + */ +export interface StartupTracer { + startSpan( + name: string, + options?: { attributes?: Record }, + context?: StartupSpanContext, + ): StartupSpan; +} + +/** + * The injected tracer plus the one context helper the engine needs to build a + * parent-context token from the root span. The server binds these to + * `@opentelemetry/api` (`trace.getTracer`, `trace.setSpan` over + * `context.active()`). The default is a no-op, so the whole span path stays a + * no-op until the server injects a real implementation. + */ +export interface StartupTraceContext { + readonly tracer: StartupTracer; + /** + * Return a parent-context token whose active span is `span`. A child span + * opened with this token parents to `span`. The token is opaque to + * `adapter-utils`. + */ + contextWithSpan(span: StartupSpan): StartupSpanContext; +} + +/** A shared no-op span. It implements the structural span contract and does + * nothing, so a caller with no injected tracer changes no behavior. */ +export const NOOP_STARTUP_SPAN: StartupSpan = { + setAttribute() {}, + setStatus() {}, + end() {}, +}; + +const NOOP_SPAN = NOOP_STARTUP_SPAN; + +/** + * The default tracer. It opens no real span, so `measureStartupStep` behaves + * exactly as before when the caller injects no tracer. + */ +const NOOP_TRACER: StartupTracer = { + startSpan: () => NOOP_SPAN, +}; + +/** + * The default trace context. Its tracer is a no-op and it produces no parent + * token, so the engine emits no spans until the server injects a real + * implementation. + */ +export const NOOP_STARTUP_TRACE_CONTEXT: StartupTraceContext = { + tracer: NOOP_TRACER, + contextWithSpan: () => undefined, +}; + +/** + * Set a numeric span attribute only when the value is a finite number. A reader + * that returns `undefined` (the counter is unavailable) yields no attribute, + * never `NaN` and never a misleading `0`. This mirrors the host counter guard + * at `environment-execution-target.ts`. + */ +function setFiniteNumberAttr( + span: StartupSpan, + key: string, + value: number | undefined, +): void { + if (typeof value === "number" && Number.isFinite(value)) { + span.setAttribute(key, value); + } +} + +/** + * Compute a counter delta from a reader. Return `undefined` when the reader is + * absent, or when either the start or the end snapshot is not a finite number. + * A `undefined` result yields no payload field and no span attribute. + */ +function finiteDelta( + read: (() => number) | undefined, + start: number | undefined, +): number | undefined { + if (!read) return undefined; + const end = read(); + if (typeof end !== "number" || !Number.isFinite(end)) return undefined; + const base = typeof start === "number" && Number.isFinite(start) ? start : 0; + return end - base; +} + /** * Optional per-step attribution attached to a `run.startup.step` event, all * additive to the free-form jsonb payload (no schema change). Each reader is a @@ -26,12 +176,28 @@ export const RUN_STARTUP_STEP_EVENT_TYPE = "run.startup.step"; * any additional numeric fields to merge into the payload; used by * `acp.handshake` to carry its `createRuntimeMs` / `ensureSessionMs` sub-split * (Open Q2), which are measured by the caller rather than read from a counter. + * The `extra` map feeds the EVENT payload only. Its keys never become span + * attributes, so a free-form key cannot widen the closed span allowlist. + * - `tracer` — an injected structural tracer. It defaults to a no-op, so the + * span path changes no runtime behavior until the server injects a real + * tracer. The span carries only the closed attribute allowlist (`step`, the + * normalized `provider`, and the finite counter deltas). + * - `parentContext` — an opaque parent-context token from the root span. When + * set, the step's span parents to that root. `measureStartupStep` forwards it + * to `startSpan` and never inspects it, so parenting stays explicit and does + * not depend on ambient async-context propagation. + * - `provider` — the raw provider key for the step. `measureStartupStep` + * normalizes it through `normalizeProviderFamily` before it sets the + * low-cardinality `provider` span attribute. It never sets the raw key. */ export interface StartupStepMeasureOptions { roundTrips?: () => number; providerExecMs?: () => number; providerGetMs?: () => number; extra?: () => Record; + tracer?: StartupTracer; + parentContext?: StartupSpanContext; + provider?: string; } function buildStepEvent(payload: Record): AdapterRuntimeEvent { @@ -55,6 +221,16 @@ function buildStepEvent(payload: Record): AdapterRuntimeEvent { * `ctx.onEvent` is optional — a missing sink is a no-op that neither throws nor * swallows `fn`'s return value or error. A step skipped by a warm cache never * calls this helper, so it emits no event (never a zero). + * + * When `options.tracer` is injected, the helper also opens one span at `start` + * and ends it in the `finally`. The span carries a closed attribute allowlist: + * `step`, the normalized `provider`, and the finite counter deltas + * (`roundTrips` / `providerExecMs` / `providerGetMs`). A throwing `fn` sets the + * span error status before the span ends. The span build reuses the same delta + * values as the event payload, so the two paths never drift. The tracer + * defaults to a no-op, so a caller with no tracer changes nothing. Every span + * call sits inside the same error swallow as the event sink, so a throwing + * tracer never changes startup control flow. */ export async function measureStartupStep( ctx: Pick, @@ -67,23 +243,58 @@ export async function measureStartupStep( const roundTripsStart = options.roundTrips?.(); const providerExecStart = options.providerExecMs?.(); const providerGetStart = options.providerGetMs?.(); + + // Open the span with only the low-cardinality allowlisted attributes known at + // the start: the step name and the normalized provider family. + const tracer = options.tracer ?? NOOP_TRACER; + const startAttributes: Record = { step }; + if (options.provider !== undefined) { + startAttributes.provider = normalizeProviderFamily(options.provider); + } + let span: StartupSpan; + try { + span = tracer.startSpan(step, { attributes: startAttributes }, options.parentContext); + } catch { + // A throwing tracer must not change startup control flow. + span = NOOP_SPAN; + } + + let stepFailed = false; try { return await fn(); + } catch (err) { + stepFailed = true; + throw err; } finally { const durationMs = now() - start; + + // One attribute-build block feeds both the event payload and the span, so + // the two paths never drift. `undefined` deltas produce neither a payload + // field nor a span attribute (fail open — never `NaN`, never `0`). + const roundTrips = finiteDelta(options.roundTrips, roundTripsStart); + const providerExecMs = finiteDelta(options.providerExecMs, providerExecStart); + const providerGetMs = finiteDelta(options.providerGetMs, providerGetStart); + const payload: Record = { step, durationMs }; - if (options.roundTrips) { - payload.roundTrips = options.roundTrips() - (roundTripsStart ?? 0); - } - if (options.providerExecMs) { - payload.providerExecMs = options.providerExecMs() - (providerExecStart ?? 0); - } - if (options.providerGetMs) { - payload.providerGetMs = options.providerGetMs() - (providerGetStart ?? 0); - } + if (roundTrips !== undefined) payload.roundTrips = roundTrips; + if (providerExecMs !== undefined) payload.providerExecMs = providerExecMs; + if (providerGetMs !== undefined) payload.providerGetMs = providerGetMs; if (options.extra) { + // `extra` feeds the EVENT payload only. Its keys never become span + // attributes, so it cannot widen the closed span allowlist. Object.assign(payload, options.extra()); } + + try { + if (stepFailed) span.setStatus({ code: SPAN_STATUS_CODE_ERROR }); + setFiniteNumberAttr(span, "roundTrips", roundTrips); + setFiniteNumberAttr(span, "providerExecMs", providerExecMs); + setFiniteNumberAttr(span, "providerGetMs", providerGetMs); + span.end(); + } catch { + // Observability must not change startup control flow. + } + try { await ctx.onEvent?.(buildStepEvent(payload)); } catch { diff --git a/packages/adapter-utils/src/command-managed-runtime.test.ts b/packages/adapter-utils/src/command-managed-runtime.test.ts index 3e494212c30..859ea127582 100644 --- a/packages/adapter-utils/src/command-managed-runtime.test.ts +++ b/packages/adapter-utils/src/command-managed-runtime.test.ts @@ -17,7 +17,7 @@ const execFile = promisify(execFileCallback); interface SpawnRunnerHandle { runner: CommandManagedRuntimeRunner; - calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string; noProfile?: boolean }>; + calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string }>; } // A runner that actually executes the shell scripts (piping stdin through a real @@ -27,7 +27,7 @@ function makeSpawnRunner(options: { supportsSingleStreamStdinProgress?: boolean; maxStdoutBytes?: number; } = {}): SpawnRunnerHandle { - const calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string; noProfile?: boolean }> = []; + const calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string }> = []; const runner: CommandManagedRuntimeRunner = { supportsSingleStreamStdinProgress: options.supportsSingleStreamStdinProgress, execute: async (input) => @@ -37,7 +37,6 @@ function makeSpawnRunner(options: { args: input.args, cwd: input.cwd, stdin: input.stdin, - noProfile: input.noProfile, }); const startedAt = new Date().toISOString(); const command = @@ -153,7 +152,6 @@ describe("command managed runtime", () => { env?: Record; stdin?: string; timeoutMs?: number; - noProfile?: boolean; }> = []; const runner = { execute: async (input: { @@ -163,7 +161,6 @@ describe("command managed runtime", () => { env?: Record; stdin?: string; timeoutMs?: number; - noProfile?: boolean; }): Promise => { calls.push({ ...input }); const startedAt = new Date().toISOString(); @@ -236,7 +233,6 @@ describe("command managed runtime", () => { // The single-stream upload pipes the tarball through exactly one stdin-backed // process (the speed fix); nothing else streams stdin. expect(calls.filter((call) => call.stdin != null).length).toBe(1); - expect(calls.some((call) => call.noProfile === true)).toBe(true); await mkdir(path.join(remoteWorkspaceDir, ".paperclip-runtime"), { recursive: true }); await writeFile(path.join(remoteWorkspaceDir, "README.md"), "remote workspace\n", "utf8"); @@ -251,7 +247,6 @@ describe("command managed runtime", () => { // Restore streams the download through `base64`/onLog (no stdin), so the only // stdin-backed call remains the single upload from prepare. expect(calls.filter((call) => call.stdin != null).length).toBe(1); - expect(calls.some((call) => call.noProfile === true)).toBe(true); }); it("stages runtime assets without replacing or restoring an in-place workspace", async () => { @@ -308,6 +303,59 @@ describe("command managed runtime", () => { ); }); + it("stages each additional project into an isolated dir on the base64/tar transport, one failure skipped", async () => { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-runtime-additional-")); + cleanupDirs.push(rootDir); + + const localWorkspaceDir = path.join(rootDir, "local-workspace"); + const remoteWorkspaceDir = path.join(rootDir, "remote-workspace"); + await mkdir(localWorkspaceDir, { recursive: true }); + await mkdir(remoteWorkspaceDir, { recursive: true }); + await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor\n", "utf8"); + + const goodOne = path.join(rootDir, "src-one"); + const goodTwo = path.join(rootDir, "src-two"); + await mkdir(goodOne, { recursive: true }); + await mkdir(path.join(goodTwo, "nested"), { recursive: true }); + await writeFile(path.join(goodOne, "one.txt"), "one body\n", "utf8"); + await writeFile(path.join(goodTwo, "nested", "two.txt"), "two body\n", "utf8"); + + // The `makeSpawnRunner` runner exposes no native syncIn, so staging rides the + // base64/tar fallback. The middle source points at a missing directory, so + // its tar build fails; failure isolation skips only it. + const { runner } = makeSpawnRunner(); + const prepared = await prepareCommandManagedRuntime({ + runner, + spec: { + remoteCwd: remoteWorkspaceDir, + timeoutMs: 30_000, + }, + adapterKey: "claude", + workspaceLocalDir: localWorkspaceDir, + additionalSources: [ + { localPath: goodOne, projectId: "one" }, + { localPath: path.join(rootDir, "missing"), projectId: "broken" }, + { localPath: goodTwo, projectId: "two" }, + ], + }); + + const runtimeRootDir = path.posix.join(remoteWorkspaceDir, ".paperclip-runtime", "claude"); + expect(Object.keys(prepared.additionalSourceDirs).sort()).toEqual(["one", "two"]); + expect(prepared.additionalSourceDirs.one).toBe(path.posix.join(runtimeRootDir, "project-one")); + expect(prepared.additionalSourceDirs.two).toBe(path.posix.join(runtimeRootDir, "project-two")); + expect(prepared.additionalSourceDirs.broken).toBeUndefined(); + + // Each healthy project's tree materialized in its OWN dir (nested files kept). + await expect(readFile(path.join(prepared.additionalSourceDirs.one, "one.txt"), "utf8")).resolves.toBe("one body\n"); + await expect(readFile(path.join(prepared.additionalSourceDirs.two, "nested", "two.txt"), "utf8")).resolves.toBe( + "two body\n", + ); + // The broken project's dir was never created. + await expect(readFile(path.join(runtimeRootDir, "project-broken"), "utf8")).rejects.toMatchObject({ + code: "ENOENT", + }); + }); + it("keeps adapter detection on the profile-backed shell path", async () => { const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-runtime-detect-")); cleanupDirs.push(rootDir); @@ -330,10 +378,8 @@ describe("command managed runtime", () => { detectCommand: "sh", }); - // The detection probe must be the first shell invocation and stay on the - // default profile-sourcing path (noProfile !== true) so a CLI provided by - // the login profile is discoverable before we decide whether to install. - expect(calls[0]?.noProfile).not.toBe(true); + // The detection probe must be the first shell invocation, so a CLI on the + // sandbox default PATH is discoverable before we decide whether to install. expect(calls[0]?.args?.join(" ")).toContain("command -v 'sh'"); // Detection succeeds here, so the install command must be skipped entirely; // the remaining calls are workspace staging, never the install command. @@ -588,15 +634,6 @@ describe("command managed runtime", () => { expect(untarIdx).toBeGreaterThan(uploadIdx); expect(cmd1Idx).toBeGreaterThan(untarIdx); expect(cmd2Idx).toBeGreaterThan(cmd1Idx); - - // Fast path: the fixed internal transport helpers (tar upload + untar) ride - // the no-profile shell — they are trusted, fixed commands that never need a - // login-shell profile. The opaque post-upload commands stay profile-backed - // (noProfile !== true) so any env a caller-supplied command relies on is present. - expect(calls[uploadIdx]?.noProfile).toBe(true); - expect(calls[untarIdx]?.noProfile).toBe(true); - expect(calls[cmd1Idx]?.noProfile).not.toBe(true); - expect(calls[cmd2Idx]?.noProfile).not.toBe(true); }); it("fallback syncIn runs a post-upload command under its own timeout, not the sync-client default", async () => { @@ -663,11 +700,6 @@ describe("command managed runtime", () => { expect(scripts[3]).toContain(targetFile); expect(scripts[4]).toContain("rm -rf"); expect(scripts[4]).toContain(targetFile + ".paperclip-syncin."); - - // Fast path: the staged-write helpers (chmod + rename) are fixed internal - // commands, so they ride the no-profile shell alongside the upload/staging. - expect(calls[2]?.noProfile).toBe(true); // chmod - expect(calls[3]?.noProfile).toBe(true); // mv (rename into place) }); it("fallback syncIn cleans up a staged file when chmod fails before rename", async () => { diff --git a/packages/adapter-utils/src/command-managed-runtime.ts b/packages/adapter-utils/src/command-managed-runtime.ts index 3d5a2f86f1d..fc9b0f0cc85 100644 --- a/packages/adapter-utils/src/command-managed-runtime.ts +++ b/packages/adapter-utils/src/command-managed-runtime.ts @@ -6,6 +6,7 @@ import { createTarballFromDirectory, prepareSandboxManagedRuntime, type PreparedSandboxManagedRuntime, + type SandboxAdditionalSource, type SandboxManagedRuntimeAsset, type SandboxManagedRuntimeClient, type SandboxRemoteExecutionSpec, @@ -50,7 +51,6 @@ export interface CommandManagedRuntimeRunner { env?: Record; stdin?: string; timeoutMs?: number; - noProfile?: boolean; onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise; onSpawn?: (meta: { pid: number; startedAt: string }) => Promise; }): Promise; @@ -206,7 +206,6 @@ export function createCommandManagedRuntimeClient(input: { opts: { stdin?: string; timeoutMs?: number; - noProfile?: boolean; onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise; } = {}, ) => { @@ -216,7 +215,6 @@ export function createCommandManagedRuntimeClient(input: { cwd: input.commandCwd, stdin: opts.stdin, timeoutMs: opts.timeoutMs ?? input.timeoutMs, - noProfile: opts.noProfile === true, onLog: opts.onLog, }); requireSuccessfulResult(result, script); @@ -225,7 +223,7 @@ export function createCommandManagedRuntimeClient(input: { const client: SandboxManagedRuntimeClient = { makeDir: async (remotePath) => { - await runShell(`mkdir -p ${shellQuote(remotePath)}`, { noProfile: true }); + await runShell(`mkdir -p ${shellQuote(remotePath)}`); }, writeFile: async (remotePath, bytes, options) => { const buffer = toBuffer(bytes); @@ -252,7 +250,7 @@ export function createCommandManagedRuntimeClient(input: { `mkdir -p ${shellQuote(remoteDir)} && ` + `base64 -d > ${shellQuote(remoteTempPath)} && ` + `mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`, - { stdin: body, noProfile: true }, + { stdin: body }, ); await options?.onProgress?.(total, total); return; @@ -266,15 +264,14 @@ export function createCommandManagedRuntimeClient(input: { await runShell( `mkdir -p ${shellQuote(remoteDir)} && ` + `rm -f ${shellQuote(remoteTempPath)} && : > ${shellQuote(remoteTempPath)}`, - { noProfile: true }, ); for (let offset = 0; offset < total; offset += REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE) { const end = Math.min(total, offset + REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE); const chunk = buffer.subarray(offset, end).toString("base64"); - await runShell(`base64 -d >> ${shellQuote(remoteTempPath)}`, { stdin: chunk, noProfile: true }); + await runShell(`base64 -d >> ${shellQuote(remoteTempPath)}`, { stdin: chunk }); await options?.onProgress?.(end, total); } - await runShell(`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`, { noProfile: true }); + await runShell(`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`); await options?.onProgress?.(total, total); } finally { await bestEffortRemoveRemotePath(client, remoteTempPath); @@ -284,7 +281,7 @@ export function createCommandManagedRuntimeClient(input: { // Chunked reads intentionally query the remote size first, even without // a progress sink, so each sandbox RPC stays bounded and truncation is // detected without materializing the whole file as one stdout string. - const sizeResult = await runShell(`wc -c < ${shellQuote(remotePath)}`, { noProfile: true }); + const sizeResult = await runShell(`wc -c < ${shellQuote(remotePath)}`); const totalBytes = Number.parseInt(sizeResult.stdout.trim(), 10); if (!Number.isFinite(totalBytes) || totalBytes < 0) { throw new Error(`Could not determine remote file size for ${remotePath}`); @@ -303,7 +300,6 @@ export function createCommandManagedRuntimeClient(input: { for (let chunkIndex = 0; decodedSoFar < totalBytes; chunkIndex++) { const result = await runShell( `dd if=${shellQuote(remotePath)} bs=${REMOTE_READ_CHUNK_BYTES} skip=${chunkIndex} count=1 2>/dev/null | base64`, - { noProfile: true }, ); const chunk = Buffer.from(result.stdout.replace(/\s+/g, ""), "base64"); if (chunk.byteLength === 0) break; @@ -326,7 +322,6 @@ export function createCommandManagedRuntimeClient(input: { `basename "$entry"; ` + `done; ` + `fi`, - { noProfile: true }, ); return result.stdout .split(/\r?\n/) @@ -340,7 +335,6 @@ export function createCommandManagedRuntimeClient(input: { args: shellCommandArgs(`rm -rf ${shellQuote(remotePath)}`), cwd: input.commandCwd, timeoutMs: input.timeoutMs, - noProfile: true, }); requireSuccessfulResult(result, `remove ${remotePath}`); }, @@ -350,7 +344,6 @@ export function createCommandManagedRuntimeClient(input: { args: shellCommandArgs(command), cwd: input.commandCwd, timeoutMs: options.timeoutMs, - noProfile: options.noProfile === true, }); requireSuccessfulResult(result, command); }, @@ -390,7 +383,7 @@ export function createCommandManagedRuntimeClient(input: { await client.writeFile(remoteTarPath, bufferToArrayBuffer(tarBytes)); await client.run( buildSyncInExtractDirectoryCommand({ remoteTarPath, targetDir: mapping.targetPath }), - { timeoutMs: input.timeoutMs, noProfile: true }, + { timeoutMs: input.timeoutMs }, ); bytesTransferred += tarBytes.byteLength; } else { @@ -403,11 +396,11 @@ export function createCommandManagedRuntimeClient(input: { if (mapping.mode != null) { await client.run( buildSyncInChmodCommand({ mode: mapping.mode, targetPath: targetPathForWrite }), - { timeoutMs: input.timeoutMs, noProfile: true }, + { timeoutMs: input.timeoutMs }, ); await client.run( buildSyncInRenameCommand({ sourcePath: targetPathForWrite, targetPath: mapping.targetPath }), - { timeoutMs: input.timeoutMs, noProfile: true }, + { timeoutMs: input.timeoutMs }, ); } bytesTransferred += fileBytes.byteLength; @@ -475,6 +468,8 @@ export async function prepareCommandManagedRuntime(input: { workspaceExclude?: string[]; preserveAbsentOnRestore?: string[]; assets?: CommandManagedRuntimeAsset[]; + /** Referenced (additional) projects to stage into the sandbox as plain, read-only trees. */ + additionalSources?: SandboxAdditionalSource[]; installCommand?: string | null; /** When provided alongside `installCommand`, skip the install if `command -v ` succeeds. */ detectCommand?: string | null; @@ -530,6 +525,7 @@ export async function prepareCommandManagedRuntime(input: { workspaceExclude: mergeRuntimeExcludes(input.workspaceExclude), preserveAbsentOnRestore: input.preserveAbsentOnRestore, assets: input.assets, + additionalSources: input.additionalSources, onProgress: input.onProgress, onRuntimeProgress: input.onRuntimeProgress, }); @@ -567,6 +563,7 @@ export async function prepareCommandManagedRuntime(input: { workspaceExclude: mergeRuntimeExcludes(input.workspaceExclude), preserveAbsentOnRestore: input.preserveAbsentOnRestore, assets: input.assets, + additionalSources: input.additionalSources, onProgress: input.onProgress, onRuntimeProgress: input.onRuntimeProgress, }); diff --git a/packages/adapter-utils/src/execution-target-sandbox.test.ts b/packages/adapter-utils/src/execution-target-sandbox.test.ts index 657a506557a..8f939c65402 100644 --- a/packages/adapter-utils/src/execution-target-sandbox.test.ts +++ b/packages/adapter-utils/src/execution-target-sandbox.test.ts @@ -194,6 +194,60 @@ describe("sandbox adapter execution targets", () => { }); }); + it("creates the process session directories only in the launch exec, not in upfront makeDir execs", async () => { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-process-session-makedir-")); + cleanupDirs.push(rootDir); + const childPath = path.join(rootDir, "noop-acp-child.mjs"); + await writeFile(childPath, "process.stdin.on('data', () => {});\n", "utf8"); + + const delegate = createLocalSandboxRunner(); + const execScripts: string[] = []; + const runner = { + execute: vi.fn(async (input: Parameters[0]) => { + execScripts.push(input.args?.[1] ?? ""); + return delegate.execute(input); + }), + }; + const target: AdapterSandboxExecutionTarget = { + kind: "remote", + transport: "sandbox", + providerKey: "local-test", + remoteCwd: rootDir, + timeoutMs: 30_000, + runner, + }; + + const bridge = await startAdapterExecutionTargetProcessSessionBridge({ + runId: "run-process-session-makedir", + target, + runtimeRootDir: path.posix.join(rootDir, ".paperclip-runtime", "acpx"), + adapterKey: "acpx", + command: process.execPath, + args: [childPath], + cwd: rootDir, + env: {}, + timeoutSec: 5, + onLog: async () => {}, + }); + expect(bridge).not.toBeNull(); + + try { + // No standalone `mkdir -p '/stdin'` or `.../events` exec runs before launch. + const standaloneSessionDirExecs = execScripts.filter((script) => + /^mkdir -p '[^']*\/(stdin|events)'\s*$/.test(script), + ); + expect(standaloneSessionDirExecs).toEqual([]); + + // The launch exec creates both directories in one `mkdir -p` line. + const launchExecs = execScripts.filter( + (script) => script.includes("nohup") && /mkdir -p [^\n]*\/stdin[^\n]*\/events/.test(script), + ); + expect(launchExecs.length).toBe(1); + } finally { + await bridge?.stop(); + } + }); + it("bridges bidirectional sandbox process sessions through a local ACPX-spawnable proxy", async () => { const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-process-session-")); cleanupDirs.push(rootDir); diff --git a/packages/adapter-utils/src/execution-target.ts b/packages/adapter-utils/src/execution-target.ts index 25777e57321..9652fd2cb32 100644 --- a/packages/adapter-utils/src/execution-target.ts +++ b/packages/adapter-utils/src/execution-target.ts @@ -14,6 +14,8 @@ import { prepareRemoteManagedRuntime, remoteExecutionSessionMatches, } from "./remote-managed-runtime.js"; +import type { SandboxAdditionalSource } from "./sandbox-managed-runtime.js"; +export type { SandboxAdditionalSource } from "./sandbox-managed-runtime.js"; import { createCommandManagedSandboxCallbackBridgeQueueClient, createSandboxCallbackBridgeAsset, @@ -114,6 +116,12 @@ export interface PreparedAdapterExecutionTargetRuntime { workspaceRemoteDir: string | null; runtimeRootDir: string | null; assetDirs: Record; + /** + * Remote directory of each additional (referenced) project that staged + * successfully, keyed by `projectId`. Empty for a local target or when no + * additional sources were requested. + */ + additionalSourceDirs: Record; restoreWorkspace(onProgress?: RuntimeProgressSink): Promise; } @@ -1107,6 +1115,8 @@ export async function prepareAdapterExecutionTargetRuntime(input: { workspaceExclude?: string[]; preserveAbsentOnRestore?: string[]; assets?: AdapterManagedRuntimeAsset[]; + /** Referenced (additional) projects to stage into the sandbox as plain, read-only trees. */ + additionalSources?: SandboxAdditionalSource[]; installCommand?: string | null; /** When provided alongside `installCommand`, skip the install if the binary is already on PATH. */ detectCommand?: string | null; @@ -1124,6 +1134,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: { workspaceRemoteDir: null, runtimeRootDir: null, assetDirs: {}, + additionalSourceDirs: {}, restoreWorkspace: async () => {}, }; } @@ -1137,6 +1148,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: { workspaceRemoteDir: input.workspaceRemoteDir, syncWorkspace: input.syncWorkspace, assets: input.assets, + additionalSources: input.additionalSources, onProgress: input.onProgress, }); return { @@ -1144,6 +1156,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: { workspaceRemoteDir: prepared.workspaceRemoteDir, runtimeRootDir: prepared.runtimeRootDir, assetDirs: prepared.assetDirs, + additionalSourceDirs: prepared.additionalSourceDirs, restoreWorkspace: prepared.restoreWorkspace, }; } @@ -1167,6 +1180,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: { workspaceExclude: input.workspaceExclude, preserveAbsentOnRestore: input.preserveAbsentOnRestore, assets: input.assets, + additionalSources: input.additionalSources, installCommand: input.installCommand, detectCommand: input.detectCommand, onProgress: input.onProgress, @@ -1177,6 +1191,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: { workspaceRemoteDir: prepared.workspaceRemoteDir, runtimeRootDir: prepared.runtimeRootDir, assetDirs: prepared.assetDirs, + additionalSourceDirs: prepared.additionalSourceDirs, restoreWorkspace: prepared.restoreWorkspace, }; } @@ -1363,8 +1378,9 @@ export async function startAdapterExecutionTargetProcessSessionBridge(input: { shellCommand, }); - await client.makeDir(stdinDir); - await client.makeDir(eventsDir); + // The launch exec below re-creates stdinDir and eventsDir with one `mkdir -p`, + // and the remote script also creates them on start. No reader touches the two + // directories before the launch exec runs, so upfront makeDir execs are redundant. await syncProcessSessionRemoteScript({ runner, remoteCwd: target.remoteCwd, diff --git a/packages/adapter-utils/src/remote-managed-runtime.test.ts b/packages/adapter-utils/src/remote-managed-runtime.test.ts index 0d0c98fb0eb..330b1cfcd63 100644 --- a/packages/adapter-utils/src/remote-managed-runtime.test.ts +++ b/packages/adapter-utils/src/remote-managed-runtime.test.ts @@ -15,7 +15,7 @@ const { stdout: Buffer.from('{"token":"remote"}\n').toString("base64"), stderr: "", })), - syncDirectoryToSsh: vi.fn(async () => undefined), + syncDirectoryToSsh: vi.fn(async (_input: { localDir: string }) => undefined), })); vi.mock("./ssh.js", () => ({ @@ -92,4 +92,95 @@ describe("remote managed runtime", () => { ); expect(restoredAuth).toBe('{"token":"remote"}\n'); }); + + it("stages each additional project into its own isolated SSH dir, isolating one failure", async () => { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-remote-runtime-additional-")); + cleanupDirs.push(rootDir); + const workspaceDir = path.join(rootDir, "workspace"); + const firstDir = path.join(rootDir, "referenced-first"); + const secondDir = path.join(rootDir, "referenced-second"); + const brokenDir = path.join(rootDir, "referenced-broken"); + await mkdir(workspaceDir, { recursive: true }); + + // The transfer rejects only for the broken project's directory. + syncDirectoryToSsh.mockImplementation(async (input: { localDir: string }) => { + if (input.localDir === brokenDir) throw new Error("ssh transfer failed"); + return undefined; + }); + + const prepared = await prepareRemoteManagedRuntime({ + spec: { + host: "127.0.0.1", + port: 2222, + username: "fixture", + remoteWorkspacePath: "/app", + remoteCwd: "/app", + privateKey: "PRIVATE KEY", + knownHosts: "KNOWN HOSTS", + strictHostKeyChecking: true, + }, + runId: "run-additional", + adapterKey: "codex", + workspaceLocalDir: workspaceDir, + workspaceRemoteDir: "/app", + syncWorkspace: false, + additionalSources: [ + { localPath: firstDir, projectId: "first" }, + { localPath: brokenDir, projectId: "broken" }, + { localPath: secondDir, projectId: "second" }, + ], + }); + + // Each healthy project staged into its OWN isolated dir under the runtime + // root; the broken one is skipped, not fatal. + expect(Object.keys(prepared.additionalSourceDirs).sort()).toEqual(["first", "second"]); + expect(prepared.additionalSourceDirs.first).toBe("/app/.paperclip-runtime/codex/project-first"); + expect(prepared.additionalSourceDirs.second).toBe("/app/.paperclip-runtime/codex/project-second"); + expect(prepared.additionalSourceDirs.broken).toBeUndefined(); + expect(syncDirectoryToSsh).toHaveBeenCalledWith(expect.objectContaining({ + localDir: firstDir, + remoteDir: "/app/.paperclip-runtime/codex/project-first", + })); + expect(syncDirectoryToSsh).toHaveBeenCalledWith(expect.objectContaining({ + localDir: secondDir, + remoteDir: "/app/.paperclip-runtime/codex/project-second", + })); + }); + + it("skips an additional project whose localPath is not absolute", async () => { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-remote-runtime-relative-")); + cleanupDirs.push(rootDir); + const workspaceDir = path.join(rootDir, "workspace"); + const healthyDir = path.join(rootDir, "referenced-healthy"); + await mkdir(workspaceDir, { recursive: true }); + + const prepared = await prepareRemoteManagedRuntime({ + spec: { + host: "127.0.0.1", + port: 2222, + username: "fixture", + remoteWorkspacePath: "/app", + remoteCwd: "/app", + privateKey: "PRIVATE KEY", + knownHosts: "KNOWN HOSTS", + strictHostKeyChecking: true, + }, + runId: "run-relative", + adapterKey: "codex", + workspaceLocalDir: workspaceDir, + workspaceRemoteDir: "/app", + syncWorkspace: false, + additionalSources: [ + { localPath: "relative/referenced", projectId: "relative" }, + { localPath: healthyDir, projectId: "healthy" }, + ], + }); + + // The relative-path project never reaches the transfer and is skipped; the + // absolute-path project still stages. + expect(Object.keys(prepared.additionalSourceDirs)).toEqual(["healthy"]); + expect(syncDirectoryToSsh).not.toHaveBeenCalledWith(expect.objectContaining({ + localDir: "relative/referenced", + })); + }); }); diff --git a/packages/adapter-utils/src/remote-managed-runtime.ts b/packages/adapter-utils/src/remote-managed-runtime.ts index 6d8cd3aa4bf..0902e17a9cc 100644 --- a/packages/adapter-utils/src/remote-managed-runtime.ts +++ b/packages/adapter-utils/src/remote-managed-runtime.ts @@ -7,10 +7,26 @@ import { restoreWorkspaceFromSshExecution, syncDirectoryToSsh, } from "./ssh.js"; -import type { SandboxManagedRuntimeAssetRestoreContext } from "./sandbox-managed-runtime.js"; +import type { + SandboxAdditionalSource, + SandboxManagedRuntimeAssetRestoreContext, +} from "./sandbox-managed-runtime.js"; import { captureDirectorySnapshot } from "./workspace-restore-merge.js"; import type { RuntimeProgressSink } from "./runtime-progress.js"; +const REMOTE_ADDITIONAL_SOURCE_HEAVY_DIR_EXCLUDES = [ + "node_modules", + "vendor", + "dist", + "build", + "out", + "coverage", + ".next", + ".turbo", + ".cache", + ".git", +].flatMap((entry) => [entry, `${entry}/*`, `*/${entry}`, `*/${entry}/*`]); + export interface RemoteManagedRuntimeAsset { key: string; localDir: string; @@ -25,6 +41,12 @@ export interface PreparedRemoteManagedRuntime { workspaceRemoteDir: string; runtimeRootDir: string; assetDirs: Record; + /** + * Remote directory of each additional (referenced) project that staged + * successfully, keyed by `projectId`. A project whose staging failed is + * absent (per-project failure isolation). + */ + additionalSourceDirs: Record; restoreWorkspace(onProgress?: RuntimeProgressSink): Promise; } @@ -86,6 +108,8 @@ export async function prepareRemoteManagedRuntime(input: { workspaceRemoteDir?: string; syncWorkspace?: boolean; assets?: RemoteManagedRuntimeAsset[]; + /** Referenced (additional) projects to stage as plain, read-only trees. */ + additionalSources?: SandboxAdditionalSource[]; // Upload progress sink. Threaded for the byte-counting transport rewrite; the // child task wires it into the workspace/asset transfers. onProgress?: RuntimeProgressSink; @@ -148,12 +172,50 @@ export async function prepareRemoteManagedRuntime(input: { throw error; } + // Stage each referenced (additional) project as a plain, read-only tree in its + // OWN isolated remote directory (`project-`). Additional sources + // never get the anchor's git-history/overlay semantics. Per-project failure + // isolation: one project's failure logs a warning and is skipped; the run and + // the other projects continue (no workspace restore, unlike an asset failure). + const additionalSourceDirs: Record = {}; + for (const source of input.additionalSources ?? []) { + const { localPath, projectId } = source; + try { + if (!path.posix.isAbsolute(localPath)) { + throw new Error(`additional source localPath is not an absolute path: ${localPath}`); + } + if ( + projectId.length === 0 || + projectId.includes("/") || + projectId.includes("\\") || + projectId.includes("..") + ) { + throw new Error(`additional source projectId is not a simple path segment: ${projectId}`); + } + const remoteDir = path.posix.join(runtimeRootDir, `project-${projectId}`); + await syncDirectoryToSsh({ + spec: input.spec, + localDir: localPath, + remoteDir, + exclude: REMOTE_ADDITIONAL_SOURCE_HEAVY_DIR_EXCLUDES, + onProgress: input.onProgress, + progressLabel: `project-${projectId}`, + }); + additionalSourceDirs[projectId] = remoteDir; + } catch (error) { + console.warn( + `[paperclip] Failed to stage referenced project ${projectId}; skipping it. ${String(error)}`, + ); + } + } + return { spec: input.spec, workspaceLocalDir: input.workspaceLocalDir, workspaceRemoteDir, runtimeRootDir, assetDirs, + additionalSourceDirs, restoreWorkspace: async (onProgress?: RuntimeProgressSink) => { if (preparedWorkspace && baselineSnapshot) { await restoreWorkspaceFromSshExecution({ diff --git a/packages/adapter-utils/src/sandbox-callback-bridge.test.ts b/packages/adapter-utils/src/sandbox-callback-bridge.test.ts index af7ed6c96d0..73fc3b472a4 100644 --- a/packages/adapter-utils/src/sandbox-callback-bridge.test.ts +++ b/packages/adapter-utils/src/sandbox-callback-bridge.test.ts @@ -438,6 +438,7 @@ describe("sandbox callback bridge", () => { const worker = await startSandboxCallbackBridgeWorker({ client: { makeDir: async () => {}, + makeDirs: async () => {}, listJsonFiles: async () => { throw new Error( "list /remote/.paperclip-runtime/gemini/paperclip-bridge/queue/requests failed with exit code 255: kex_exchange_identification: read: Connection reset by peer", @@ -1114,4 +1115,110 @@ describe("sandbox callback bridge", () => { }, })); }); + + it("creates the bridge queue directories in one directory-creation exec", async () => { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-bridge-makedirs-")); + cleanupDirs.push(rootDir); + + const queueDir = path.posix.join(rootDir, "queue"); + const directories = sandboxCallbackBridgeDirectories(queueDir); + const makeDir = vi.fn(async () => {}); + const makeDirs = vi.fn(async () => {}); + + const worker = await startSandboxCallbackBridgeWorker({ + client: { + makeDir, + makeDirs, + listJsonFiles: async () => [], + readTextFile: async () => { + throw new Error("unexpected readTextFile"); + }, + writeTextFile: async () => {}, + rename: async () => {}, + remove: async () => {}, + }, + queueDir, + authorizeRequest: async () => null, + handleRequest: async () => ({ status: 200, body: "ok" }), + }); + + await worker.stop(); + + expect(makeDir).not.toHaveBeenCalled(); + expect(makeDirs).toHaveBeenCalledTimes(1); + expect(makeDirs).toHaveBeenCalledWith([ + directories.rootDir, + directories.requestsDir, + directories.responsesDir, + directories.logsDir, + ]); + }); + + it("falls back to sequential makeDir when the queue client omits makeDirs", async () => { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-bridge-makedir-fallback-")); + cleanupDirs.push(rootDir); + + const queueDir = path.posix.join(rootDir, "queue"); + const directories = sandboxCallbackBridgeDirectories(queueDir); + const makeDir = vi.fn(async (_remotePath: string) => {}); + + // A queue client that predates the batched makeDirs method. The worker + // must still create every queue directory through sequential makeDir. + const worker = await startSandboxCallbackBridgeWorker({ + client: { + makeDir, + listJsonFiles: async () => [], + readTextFile: async () => { + throw new Error("unexpected readTextFile"); + }, + writeTextFile: async () => {}, + rename: async () => {}, + remove: async () => {}, + }, + queueDir, + authorizeRequest: async () => null, + handleRequest: async () => ({ status: 200, body: "ok" }), + }); + + await worker.stop(); + + expect(makeDir.mock.calls.map((call) => call[0])).toEqual([ + directories.rootDir, + directories.requestsDir, + directories.responsesDir, + directories.logsDir, + ]); + }); + + it("runs one mkdir -p exec for makeDirs on the command-managed queue client", async () => { + const runner = { + execute: vi.fn(async (_input: { args?: string[] }) => ({ + exitCode: 0, + signal: null, + timedOut: false, + stdout: "", + stderr: "", + pid: null, + startedAt: new Date().toISOString(), + })), + }; + + const client = createCommandManagedSandboxCallbackBridgeQueueClient({ + runner, + remoteCwd: "/workspace", + timeoutMs: 30_000, + }); + + // The command-managed client always provides the batched makeDirs method. + expect(client.makeDirs).toBeDefined(); + await client.makeDirs?.(["/workspace/a", "/workspace/b", "/workspace/c"]); + + expect(runner.execute).toHaveBeenCalledTimes(1); + const call = runner.execute.mock.calls[0][0]; + const script = call.args?.[call.args.length - 1] ?? ""; + expect(script).toContain("mkdir -p"); + expect(script).toContain("/workspace/a"); + expect(script).toContain("/workspace/b"); + expect(script).toContain("/workspace/c"); + }); }); diff --git a/packages/adapter-utils/src/sandbox-callback-bridge.ts b/packages/adapter-utils/src/sandbox-callback-bridge.ts index 8ba56bc805c..d990adbef3b 100644 --- a/packages/adapter-utils/src/sandbox-callback-bridge.ts +++ b/packages/adapter-utils/src/sandbox-callback-bridge.ts @@ -149,6 +149,11 @@ export interface SandboxCallbackBridgeDirectories { export interface SandboxCallbackBridgeQueueClient { makeDir(remotePath: string): Promise; + // Optional batched directory create. The built-in clients create every + // queue directory in one remote exec. A client that predates this method + // omits it; the worker falls back to sequential `makeDir` calls, so an + // external implementation stays compatible without a change. + makeDirs?(remotePaths: string[]): Promise; listJsonFiles(remotePath: string): Promise; readTextFile(remotePath: string): Promise; writeTextFile(remotePath: string, body: string): Promise; @@ -359,6 +364,11 @@ export function createFileSystemSandboxCallbackBridgeQueueClient(): SandboxCallb makeDir: async (remotePath) => { await fs.mkdir(remotePath, { recursive: true }); }, + makeDirs: async (remotePaths) => { + for (const remotePath of remotePaths) { + await fs.mkdir(remotePath, { recursive: true }); + } + }, listJsonFiles: async (remotePath) => { const entries = await fs.readdir(remotePath, { withFileTypes: true }).catch(() => []); return entries @@ -470,6 +480,13 @@ export function createCommandManagedSandboxCallbackBridgeQueueClient(input: { makeDir: async (remotePath) => { await runChecked(`mkdir ${remotePath}`, `mkdir -p ${shellQuote(remotePath)}`); }, + makeDirs: async (remotePaths) => { + if (remotePaths.length === 0) { + return; + } + const quoted = remotePaths.map((remotePath) => shellQuote(remotePath)); + await runChecked(`mkdir ${remotePaths.join(" ")}`, `mkdir -p ${quoted.join(" ")}`); + }, listJsonFiles: async (remotePath) => { const result = await runShell( input.runner, @@ -606,10 +623,21 @@ export async function startSandboxCallbackBridgeWorker(input: { const pollIntervalMs = normalizeTimeoutMs(input.pollIntervalMs, DEFAULT_BRIDGE_POLL_INTERVAL_MS); const maxBodyBytes = normalizeTimeoutMs(input.maxBodyBytes, DEFAULT_BRIDGE_MAX_BODY_BYTES); const directories = sandboxCallbackBridgeDirectories(input.queueDir); - await input.client.makeDir(directories.rootDir); - await input.client.makeDir(directories.requestsDir); - await input.client.makeDir(directories.responsesDir); - await input.client.makeDir(directories.logsDir); + const queueDirectories = [ + directories.rootDir, + directories.requestsDir, + directories.responsesDir, + directories.logsDir, + ]; + if (input.client.makeDirs) { + await input.client.makeDirs(queueDirectories); + } else { + // Backward-compatible fallback for a queue client that omits the batched + // makeDirs method. Create each queue directory with a single makeDir. + for (const directory of queueDirectories) { + await input.client.makeDir(directory); + } + } let stopping = false; let inFlight = 0; diff --git a/packages/adapter-utils/src/sandbox-file-sync.test.ts b/packages/adapter-utils/src/sandbox-file-sync.test.ts index de3071c1c92..aeb642a645f 100644 --- a/packages/adapter-utils/src/sandbox-file-sync.test.ts +++ b/packages/adapter-utils/src/sandbox-file-sync.test.ts @@ -231,6 +231,108 @@ describe("sandbox native file sync", () => { expect(await readFile(path.join(prepared.assetDirs.creds, "cred.txt"), "utf8")).toBe("secret\n"); }); + it("stages each additional project into its own isolated dir via a native directory syncIn", async () => { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-additional-")); + cleanupDirs.push(rootDir); + const localWorkspaceDir = path.join(rootDir, "local-workspace"); + const remoteWorkspaceDir = path.join(rootDir, "remote-workspace"); + await mkdir(localWorkspaceDir, { recursive: true }); + await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor\n", "utf8"); + + // Three referenced projects, each with a distinctive file, staged as plain + // read-only trees. + const projects = [ + { projectId: "alpha", localDir: path.join(rootDir, "src-alpha"), file: "alpha.txt", body: "alpha body\n" }, + { projectId: "bravo", localDir: path.join(rootDir, "src-bravo"), file: "bravo.txt", body: "bravo body\n" }, + { projectId: "charlie", localDir: path.join(rootDir, "src-charlie"), file: "charlie.txt", body: "charlie body\n" }, + ]; + for (const project of projects) { + await mkdir(project.localDir, { recursive: true }); + await writeFile(path.join(project.localDir, project.file), project.body, "utf8"); + } + + const { client, syncInOps } = makeNativeClient(); + const prepared = await prepareSandboxManagedRuntime({ + spec: { transport: "sandbox", provider: "test", sandboxId: "s1", remoteCwd: remoteWorkspaceDir, timeoutMs: 30_000, apiKey: null }, + adapterKey: "test-adapter", + client, + workspaceLocalDir: localWorkspaceDir, + additionalSources: projects.map((project) => ({ localPath: project.localDir, projectId: project.projectId })), + }); + + // Each project lands in its OWN `project-` directory under the + // runtime root, and its file materializes there. + const projectDirs = projects.map((project) => { + const dir = prepared.additionalSourceDirs[project.projectId]; + expect(dir).toBe(path.posix.join(prepared.runtimeRootDir, `project-${project.projectId}`)); + return dir; + }); + for (const [index, project] of projects.entries()) { + expect(await readFile(path.join(projectDirs[index], project.file), "utf8")).toBe(project.body); + } + + // The target dirs are pairwise distinct and never nested inside one another. + for (const outer of projectDirs) { + for (const inner of projectDirs) { + if (outer === inner) continue; + expect(inner.startsWith(`${outer}/`)).toBe(false); + } + } + + // Each project rides its own `syncIn` operation as a single `directory` + // mapping, source = the host checkout dir, target = the isolated project dir. + const inboundOps = syncInOps.flat(); + for (const [index, project] of projects.entries()) { + const op = inboundOps.find((candidate) => + candidate.files.some((mapping) => mapping.targetPath === projectDirs[index]), + ); + expect(op).toBeDefined(); + expect(op!.operationId).toMatch(/^sync-op-\d+$/); + expect(op!.operationId).not.toContain(project.projectId); + expect(op!.files).toHaveLength(1); + expect(op!.files[0]).toMatchObject({ + sourcePath: project.localDir, + targetPath: projectDirs[index], + kind: "directory", + }); + // Plain read-only tree — no post-upload extract/wipe/merge command. + expect(op!.postUploadCommands ?? []).toHaveLength(0); + } + }); + + it("isolates one additional project's sync failure and stages the rest", async () => { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-additional-fail-")); + cleanupDirs.push(rootDir); + const localWorkspaceDir = path.join(rootDir, "local-workspace"); + const remoteWorkspaceDir = path.join(rootDir, "remote-workspace"); + const goodDir = path.join(rootDir, "src-good"); + await mkdir(localWorkspaceDir, { recursive: true }); + await mkdir(goodDir, { recursive: true }); + await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor\n", "utf8"); + await writeFile(path.join(goodDir, "good.txt"), "good body\n", "utf8"); + + // The middle source points at a directory that does not exist, so its native + // transfer fails. Failure isolation must skip only it and stage the rest. + const { client } = makeNativeClient(); + const prepared = await prepareSandboxManagedRuntime({ + spec: { transport: "sandbox", provider: "test", sandboxId: "s1", remoteCwd: remoteWorkspaceDir, timeoutMs: 30_000, apiKey: null }, + adapterKey: "test-adapter", + client, + workspaceLocalDir: localWorkspaceDir, + additionalSources: [ + { localPath: goodDir, projectId: "good-a" }, + { localPath: path.join(rootDir, "does-not-exist"), projectId: "broken" }, + { localPath: goodDir, projectId: "good-b" }, + ], + }); + + // Both healthy projects staged; the broken one is absent, not fatal. + expect(Object.keys(prepared.additionalSourceDirs).sort()).toEqual(["good-a", "good-b"]); + expect(prepared.additionalSourceDirs.broken).toBeUndefined(); + expect(await readFile(path.join(prepared.additionalSourceDirs["good-a"], "good.txt"), "utf8")).toBe("good body\n"); + expect(await readFile(path.join(prepared.additionalSourceDirs["good-b"], "good.txt"), "utf8")).toBe("good body\n"); + }); + it("dereferences symlinks only when followSymlinks is true (native honors the flag)", async () => { const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-symlink-")); cleanupDirs.push(rootDir); diff --git a/packages/adapter-utils/src/sandbox-managed-runtime.test.ts b/packages/adapter-utils/src/sandbox-managed-runtime.test.ts index 8444f059156..db796761f28 100644 --- a/packages/adapter-utils/src/sandbox-managed-runtime.test.ts +++ b/packages/adapter-utils/src/sandbox-managed-runtime.test.ts @@ -2,18 +2,24 @@ import { promises as fsPromises } from "node:fs"; import { lstat, mkdir, mkdtemp, readFile, readdir, rm, symlink, writeFile } from "node:fs/promises"; import os from "node:os"; import path from "node:path"; -import { execFile as execFileCallback } from "node:child_process"; +import { execFile as execFileCallback, spawn } from "node:child_process"; import { promisify } from "node:util"; import { afterEach, describe, expect, it, vi } from "vitest"; import { resetLocalGitIndexToHead } from "./git-workspace-sync.js"; import { + assertSyncOperationsConfined, mirrorDirectory, prepareSandboxManagedRuntime, type SandboxManagedRuntimeClient, type SandboxSyncOperation, type SandboxSyncResult, } from "./sandbox-managed-runtime.js"; +import { + prepareCommandManagedRuntime, + type CommandManagedRuntimeRunner, +} from "./command-managed-runtime.js"; +import type { RunProcessResult } from "./server-utils.js"; function toArrayBuffer(bytes: Buffer): ArrayBuffer { return bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength) as ArrayBuffer; @@ -87,6 +93,41 @@ function attachNativeRecordingSyncIn( }; } +// A capturing `syncIn` that records every operation for assertion and +// materializes file AND directory mappings. A directory mapping (a referenced +// project) uses `mirrorDirectory`, so a test can assert the advisory `access` +// intent on directory mappings as well as file mappings. +function attachCapturingSyncIn( + client: SandboxManagedRuntimeClient, + captured: SandboxSyncOperation[], +): void { + client.syncIn = async (operations: SandboxSyncOperation[]): Promise => { + const resultOperations: SandboxSyncResult["operations"] = []; + for (const operation of operations) { + captured.push(operation); + let filesTransferred = 0; + let bytesTransferred = 0; + for (const mapping of operation.files) { + await mkdir(path.posix.dirname(mapping.targetPath), { recursive: true }); + if (mapping.kind === "directory") { + await mirrorDirectory(mapping.sourcePath, mapping.targetPath); + } else { + const bytes = await readFile(mapping.sourcePath); + await writeFile(mapping.targetPath, bytes); + if (mapping.mode != null) await fsPromises.chmod(mapping.targetPath, mapping.mode); + bytesTransferred += bytes.byteLength; + } + filesTransferred += 1; + } + for (const command of operation.postUploadCommands ?? []) { + await execFile("sh", ["-c", command.command], { maxBuffer: 32 * 1024 * 1024 }); + } + resultOperations.push({ operationId: operation.operationId, filesTransferred, bytesTransferred }); + } + return { operations: resultOperations }; + }; +} + vi.mock("node:fs", async (importOriginal) => { const actual = await importOriginal(); return { @@ -438,9 +479,13 @@ describe("sandbox managed runtime", () => { "restore", "finalize", ])); + // Git history and workspace overlay sync as ONE merged operation, so a single + // transfer-progress event rides the config_sync (workspace) phase. The git_sync + // phase still emits its plain status message (asserted by the arrayContaining + // check above). expect(runtimeStatuses.some((status) => ( - status.phase === "git_sync" && - /^Syncing git history to sandbox: 100% \(\d+\.\d\/\d+\.\d MB\)$/.test(status.message) + status.phase === "config_sync" && + /^Syncing workspace to sandbox: 100% \(\d+\.\d\/\d+\.\d MB\)$/.test(status.message) ))).toBe(true); expect(runtimeStatuses.some((status) => ( status.phase === "export" && @@ -1272,6 +1317,18 @@ describe("sandbox managed runtime", () => { `rm -rf ${q(path.posix.join(prepared.runtimeRootDir, "widget"))} && mkdir -p ${q(path.posix.join(prepared.runtimeRootDir, "widget"))}`, ); + // The asset tar carries the asset directory as its read-write destination, + // because the extract command fills that directory, not the staging archive. + const assetTarMapping = assetOp!.files.find((mapping) => mapping.targetPath.endsWith("widget-upload.tar")); + expect(assetTarMapping?.access).toBe("rw"); + expect(assetTarMapping?.writablePath).toBe(prepared.assetDirs.widget); + + // The staged helper file is a read-only input that the command consumes, so + // it is `access: "ro"` and never joins the writable set. + const stageMapping = assetOp!.files.find((mapping) => mapping.targetPath.endsWith("widget-helper.sh")); + expect(stageMapping?.access).toBe("ro"); + expect(stageMapping?.writablePath).toBeUndefined(); + // The asset actually materialized through the native seam. await expect(readFile(path.join(prepared.assetDirs.widget, "seed.txt"), "utf8")).resolves.toBe("seed\n"); }); @@ -1337,21 +1394,18 @@ describe("sandbox managed runtime", () => { expect(directWrites).toEqual([]); expect(directRuns).toEqual([]); - // Two operations: git-workspace then workspace overlay. Each uploads a single - // tar as a `file` mapping and carries its extract as an ordered post-command. - expect(captured.length).toBeGreaterThanOrEqual(2); - const gitOp = captured.find((op) => - op.files.some((mapping) => mapping.targetPath.endsWith("git-workspace-upload.tar")), - ); - const workspaceOp = captured.find((op) => - op.files.some((mapping) => mapping.targetPath.endsWith("workspace-upload.tar")), - ); - expect(gitOp).toBeDefined(); - expect(workspaceOp).toBeDefined(); - expect(gitOp!.files.every((mapping) => mapping.kind === "file")).toBe(true); - // The git operation's post-upload command preserves `.paperclip-runtime` while - // replacing the rest of the tree (wipe-except-preserved), then untars. - const gitCommand = gitOp!.postUploadCommands![0].command; + // One merged operation carries BOTH the git-history and workspace-overlay tars + // as two `file` mappings, each with its extract as an ordered post-command. + expect(captured).toHaveLength(1); + const op = captured[0]; + const byBase = (base: string) => + op.files.find((mapping) => path.posix.basename(mapping.targetPath) === base); + expect(byBase("git-workspace-upload.tar")).toBeDefined(); + expect(byBase("workspace-upload.tar")).toBeDefined(); + expect(op.files.every((mapping) => mapping.kind === "file")).toBe(true); + // The first post-upload command extracts the git history and preserves + // `.paperclip-runtime` while replacing the rest of the tree (wipe-except-preserved). + const gitCommand = op.postUploadCommands![0].command; expect(gitCommand).toContain(".paperclip-runtime"); expect(gitCommand).toContain("tar -xf"); @@ -1363,16 +1417,168 @@ describe("sandbox managed runtime", () => { expect(prepared.workspaceRemoteDir).toBe(remoteWorkspaceDir); }); - // Regression lock: a representative `codex_local` start stages its inbound - // bytes — git history, workspace overlay, and the managed Codex `home` asset - // (auth.json merge) — as EXACTLY ONE `syncIn` operation each. Every inbound step - // is routed through `client.syncIn` (one native `uploadFiles` round-trip per - // operation, with the extract/merge carried as provider-executed - // `postUploadCommands`), with no separate custom-provision diversion. Assert - // the collapsed round-trip count so a future change that re-inlines a - // `writeFile`+`run` sequence — or fans one staging step across multiple - // operations — fails loudly here instead of silently regressing the start path. - it("collapses a representative codex_local start to one syncIn round-trip per inbound staging step", async () => { + it("issues one merged syncIn operation for a git-backed workspace stage-sync with two ordered extract commands", async () => { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-merged-git-")); + cleanupDirs.push(rootDir); + const sourceRepoDir = path.join(rootDir, "source-repo"); + const localWorkspaceDir = path.join(rootDir, "local-workspace"); + const remoteWorkspaceDir = path.join(rootDir, "remote-workspace"); + await mkdir(sourceRepoDir, { recursive: true }); + await git(sourceRepoDir, ["init"]); + await git(sourceRepoDir, ["checkout", "-b", "main"]); + await git(sourceRepoDir, ["config", "user.name", "Paperclip Test"]); + await git(sourceRepoDir, ["config", "user.email", "test@paperclip.dev"]); + await writeFile(path.join(sourceRepoDir, "tracked.txt"), "tracked\n", "utf8"); + await git(sourceRepoDir, ["add", "tracked.txt"]); + await git(sourceRepoDir, ["commit", "-m", "base"]); + await git(sourceRepoDir, ["worktree", "add", "-b", "work", localWorkspaceDir, "HEAD"]); + // Pre-seed the sandbox with a `.paperclip-runtime` dir that MUST survive. + await mkdir(path.join(remoteWorkspaceDir, ".paperclip-runtime"), { recursive: true }); + await writeFile(path.join(remoteWorkspaceDir, ".paperclip-runtime", "keep.txt"), "keep\n", "utf8"); + + const client: SandboxManagedRuntimeClient = { + makeDir: async (remotePath) => { + await mkdir(remotePath, { recursive: true }); + }, + writeFile: async (remotePath, bytes) => { + await mkdir(path.dirname(remotePath), { recursive: true }); + await writeFile(remotePath, Buffer.from(bytes)); + }, + readFile: async (remotePath) => await readFile(remotePath), + listFiles: async () => [], + remove: async (remotePath) => { + await rm(remotePath, { recursive: true, force: true }); + }, + run: async (command) => { + await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 }); + }, + }; + const captured: SandboxSyncOperation[] = []; + attachNativeRecordingSyncIn(client, captured); + // Count how many times `syncIn` is invoked so the merge collapses the two + // workspace staging steps into a single native round trip. + let syncInCallCount = 0; + const recordingSyncIn = client.syncIn!; + client.syncIn = async (operations) => { + syncInCallCount += 1; + return recordingSyncIn(operations); + }; + + await prepareSandboxManagedRuntime({ + spec: { + transport: "sandbox", + provider: "test", + sandboxId: "sandbox-1", + remoteCwd: remoteWorkspaceDir, + timeoutMs: 30_000, + apiKey: null, + }, + adapterKey: "test-adapter", + client, + workspaceLocalDir: localWorkspaceDir, + }); + + // One `syncIn` call carrying exactly one operation for the whole workspace. + expect(syncInCallCount).toBe(1); + expect(captured).toHaveLength(1); + const op = captured[0]; + + // Both host tars ride the one operation as two `file` mappings. + const byBase = (base: string) => + op.files.find((mapping) => path.posix.basename(mapping.targetPath) === base); + const gitMapping = byBase("git-workspace-upload.tar"); + const overlayMapping = byBase("workspace-upload.tar"); + expect(gitMapping).toBeDefined(); + expect(overlayMapping).toBeDefined(); + expect(op.files).toHaveLength(2); + expect(op.files.every((mapping) => mapping.kind === "file")).toBe(true); + + // Both tar targets live under `.paperclip-runtime`, so the git extract's wipe + // (which preserves `.paperclip-runtime`) cannot delete the overlay tar before + // the overlay extract runs. + for (const mapping of op.files) { + expect(mapping.targetPath).toContain("/.paperclip-runtime/"); + } + + // Two ordered extract commands: git history first (wipe-except-preserved), + // overlay second (merge, no wipe). No deleted paths in this clean worktree. + const commands = op.postUploadCommands ?? []; + expect(commands).toHaveLength(2); + expect(commands[0].command).toContain("git-workspace-upload.tar"); + expect(commands[0].command).toContain(".paperclip-runtime"); + expect(commands[0].command).toContain("find "); + expect(commands[1].command).toContain("workspace-upload.tar"); + expect(commands[1].command).not.toContain("git-workspace-upload.tar"); + expect(commands[1].command).not.toContain("find "); + + // The pre-seeded runtime dir survived and the workspace overlay applied. + await expect( + readFile(path.join(remoteWorkspaceDir, ".paperclip-runtime", "keep.txt"), "utf8"), + ).resolves.toBe("keep\n"); + await expect(readFile(path.join(remoteWorkspaceDir, "tracked.txt"), "utf8")).resolves.toBe("tracked\n"); + }); + + it("the merged workspace confine guard covers both tar mappings (escape in either trips it)", () => { + const runtimeRoot = "/home/daytona/paperclip-workspace/.paperclip-runtime/test-adapter"; + const tempRoot = "/tmp/paperclip-sandbox-sync-abc"; + const gitMapping = { + sourcePath: `${tempRoot}/git-workspace.tar`, + targetPath: `${runtimeRoot}/git-workspace-upload.tar`, + kind: "file" as const, + }; + const overlayMapping = { + sourcePath: `${tempRoot}/workspace.tar`, + targetPath: `${runtimeRoot}/workspace-upload.tar`, + kind: "file" as const, + }; + const roots = { sourceRoots: [tempRoot], targetRoots: [runtimeRoot] }; + + // A confined merged operation with both tar mappings passes the guard. + expect(() => + assertSyncOperationsConfined( + [{ operationId: "merged", files: [gitMapping, overlayMapping] }], + roots, + ), + ).not.toThrow(); + + // A `..` target escape in the overlay mapping trips the guard, so the whole + // merged operation is rejected before any transfer. + expect(() => + assertSyncOperationsConfined( + [{ + operationId: "merged", + files: [ + gitMapping, + { ...overlayMapping, targetPath: `${runtimeRoot}/../../etc/workspace-upload.tar` }, + ], + }], + roots, + ), + ).toThrow(/escapes its confinement root|not a confined absolute path/); + + // An absolute-path source escape in the git mapping trips the guard too. + expect(() => + assertSyncOperationsConfined( + [{ + operationId: "merged", + files: [{ ...gitMapping, sourcePath: "/etc/passwd" }, overlayMapping], + }], + roots, + ), + ).toThrow(/escapes its confinement root|not a confined absolute path/); + }); + + // Regression lock: a representative `codex_local` start stages its inbound bytes + // as TWO `syncIn` operations. The git-history and workspace-overlay tars share + // ONE merged operation (one native `uploadFiles` round-trip that carries both + // tars, with the two extract commands as ordered `postUploadCommands`); the + // managed Codex `home` asset (auth.json merge) is the second operation. Every + // inbound step is routed through `client.syncIn`, with no separate + // custom-provision diversion. Assert the collapsed round-trip count so a future + // change that re-inlines a `writeFile`+`run` sequence — or splits the merged + // workspace operation back into two — fails loudly here instead of silently + // regressing the start path. + it("collapses a representative codex_local start to two syncIn round-trips: one merged workspace op plus the asset op", async () => { const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-codex-roundtrip-")); cleanupDirs.push(rootDir); const sourceRepoDir = path.join(rootDir, "source-repo"); @@ -1380,7 +1586,7 @@ describe("sandbox managed runtime", () => { const remoteWorkspaceDir = path.join(rootDir, "remote-workspace"); const homeDir = path.join(rootDir, "codex-home"); - // Git-backed workspace → git history + workspace overlay are two staging steps. + // Git-backed workspace → git history + workspace overlay share one merged op. await mkdir(sourceRepoDir, { recursive: true }); await git(sourceRepoDir, ["init"]); await git(sourceRepoDir, ["checkout", "-b", "main"]); @@ -1453,36 +1659,168 @@ describe("sandbox managed runtime", () => { expect(directWrites).toEqual([]); expect(directRuns).toEqual([]); - // The collapsed count: exactly three inbound round-trips — git, workspace, home. - expect(captured).toHaveLength(3); - const gitOp = captured.find((op) => - op.files.some((mapping) => mapping.targetPath.endsWith("git-workspace-upload.tar")), - ); - const workspaceOp = captured.find((op) => - op.files.some((mapping) => mapping.targetPath.endsWith("workspace-upload.tar")), - ); - const homeOp = captured.find((op) => - op.files.some((mapping) => mapping.targetPath.endsWith("home-upload.tar")), - ); - expect(gitOp).toBeDefined(); + // The collapsed count: exactly two inbound round-trips — the merged workspace + // op (git history + overlay) and the home asset op. + expect(captured).toHaveLength(2); + const hasBase = (op: SandboxSyncOperation, base: string) => + op.files.some((mapping) => path.posix.basename(mapping.targetPath) === base); + const workspaceOp = captured.find((op) => hasBase(op, "workspace-upload.tar")); + const homeOp = captured.find((op) => hasBase(op, "home-upload.tar")); expect(workspaceOp).toBeDefined(); expect(homeOp).toBeDefined(); + // The merged workspace op carries BOTH the git-history and overlay tars, with + // both extract commands as ordered post-upload commands (git first, overlay + // second). The two tars ride one native uploadFiles round-trip. + expect(hasBase(workspaceOp!, "git-workspace-upload.tar")).toBe(true); + expect(workspaceOp!.files).toHaveLength(2); + expect((workspaceOp!.postUploadCommands ?? []).length).toBeGreaterThanOrEqual(2); - // Every operation is a single native uploadFiles (all `file` mappings) whose + // Every operation is a native uploadFiles (all `file` mappings) whose // extract/merge rides as an ordered provider-executed post-upload command. for (const op of captured) { expect(op.files.length).toBeGreaterThanOrEqual(1); expect(op.files.every((mapping) => mapping.kind === "file")).toBe(true); expect(op.postUploadCommands ?? []).not.toHaveLength(0); } - // Operation ids are distinct, so "3 operations" is 3 real round-trips. - expect(new Set(captured.map((op) => op.operationId)).size).toBe(3); + // Operation ids are distinct, so "2 operations" is 2 real round-trips. + expect(new Set(captured.map((op) => op.operationId)).size).toBe(2); // The credential asset actually materialized through the native seam. await expect(readFile(path.join(prepared.assetDirs.home, "auth.json"), "utf8")) .resolves.toBe("{\"OPENAI_API_KEY\":\"sk-test\"}\n"); }); + it("authors the advisory access intent rw on workspace, git, and asset inbound mappings", async () => { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-access-rw-")); + cleanupDirs.push(rootDir); + const sourceRepoDir = path.join(rootDir, "source-repo"); + const localWorkspaceDir = path.join(rootDir, "local-workspace"); + const remoteWorkspaceDir = path.join(rootDir, "remote-workspace"); + const assetDir = path.join(rootDir, "asset-home"); + + // A git-backed workspace produces both a git-history tar and an overlay tar. + await mkdir(sourceRepoDir, { recursive: true }); + await git(sourceRepoDir, ["init"]); + await git(sourceRepoDir, ["checkout", "-b", "main"]); + await git(sourceRepoDir, ["config", "user.name", "Paperclip Test"]); + await git(sourceRepoDir, ["config", "user.email", "test@paperclip.dev"]); + await writeFile(path.join(sourceRepoDir, "tracked.txt"), "tracked\n", "utf8"); + await git(sourceRepoDir, ["add", "tracked.txt"]); + await git(sourceRepoDir, ["commit", "-m", "base"]); + await git(sourceRepoDir, ["worktree", "add", "-b", "work", localWorkspaceDir, "HEAD"]); + + await mkdir(assetDir, { recursive: true }); + await writeFile(path.join(assetDir, "config.toml"), "model = \"gpt\"\n", "utf8"); + + const client: SandboxManagedRuntimeClient = { + makeDir: async (remotePath) => { await mkdir(remotePath, { recursive: true }); }, + writeFile: async (remotePath, bytes) => { + await mkdir(path.dirname(remotePath), { recursive: true }); + await writeFile(remotePath, Buffer.from(bytes)); + }, + readFile: async (remotePath) => await readFile(remotePath), + listFiles: async () => [], + remove: async (remotePath) => { await rm(remotePath, { recursive: true, force: true }); }, + run: async (command) => { await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 }); }, + }; + const captured: SandboxSyncOperation[] = []; + attachCapturingSyncIn(client, captured); + + await prepareSandboxManagedRuntime({ + spec: { + transport: "sandbox", + provider: "test", + sandboxId: "sandbox-1", + remoteCwd: remoteWorkspaceDir, + timeoutMs: 30_000, + apiKey: null, + }, + adapterKey: "test-adapter", + client, + workspaceLocalDir: localWorkspaceDir, + assets: [{ key: "home", localDir: assetDir }], + }); + + const findMapping = (base: string) => + captured + .flatMap((op) => op.files) + .find((mapping) => path.posix.basename(mapping.targetPath) === base); + + // The workspace, git-history, and asset destinations receive read-write bytes, + // so the author marks each mapping `access: "rw"`. + expect(findMapping("workspace-upload.tar")?.access).toBe("rw"); + expect(findMapping("git-workspace-upload.tar")?.access).toBe("rw"); + expect(findMapping("home-upload.tar")?.access).toBe("rw"); + + // Each tar mapping uploads a staging archive under the runtime root, so its + // `targetPath` is not the read-write destination. `writablePath` names the + // directory that the post-upload extract command fills: the workspace + // directory for the workspace and git tars, and the asset directory for the + // asset tar. + const remoteAssetDir = path.posix.join(remoteWorkspaceDir, ".paperclip-runtime", "test-adapter", "home"); + expect(findMapping("workspace-upload.tar")?.writablePath).toBe(remoteWorkspaceDir); + expect(findMapping("git-workspace-upload.tar")?.writablePath).toBe(remoteWorkspaceDir); + expect(findMapping("home-upload.tar")?.writablePath).toBe(remoteAssetDir); + }); + + it("authors the advisory access intent ro on referenced-project inbound mappings", async () => { + const flagKey = "PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC"; + const priorFlag = process.env[flagKey]; + process.env[flagKey] = "1"; + try { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-access-ro-")); + cleanupDirs.push(rootDir); + const localWorkspaceDir = path.join(rootDir, "local-workspace"); + const remoteWorkspaceDir = path.join(rootDir, "remote-workspace"); + const referencedDir = path.join(rootDir, "referenced-project"); + await mkdir(localWorkspaceDir, { recursive: true }); + await mkdir(referencedDir, { recursive: true }); + await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor\n", "utf8"); + await writeFile(path.join(referencedDir, "notes.md"), "reference\n", "utf8"); + + const client: SandboxManagedRuntimeClient = { + makeDir: async (remotePath) => { await mkdir(remotePath, { recursive: true }); }, + writeFile: async (remotePath, bytes) => { + await mkdir(path.dirname(remotePath), { recursive: true }); + await writeFile(remotePath, Buffer.from(bytes)); + }, + readFile: async (remotePath) => await readFile(remotePath), + listFiles: async () => [], + remove: async (remotePath) => { await rm(remotePath, { recursive: true, force: true }); }, + run: async (command) => { await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 }); }, + }; + const captured: SandboxSyncOperation[] = []; + attachCapturingSyncIn(client, captured); + + await prepareSandboxManagedRuntime({ + spec: { + transport: "sandbox", + provider: "test", + sandboxId: "sandbox-1", + remoteCwd: remoteWorkspaceDir, + timeoutMs: 30_000, + apiKey: null, + }, + adapterKey: "test-adapter", + client, + workspaceLocalDir: localWorkspaceDir, + additionalSources: [{ localPath: referencedDir, projectId: "proj-first" }], + }); + + const referencedMapping = captured + .flatMap((op) => op.files) + .find((mapping) => path.posix.basename(mapping.targetPath) === "project-proj-first"); + + // A referenced project is a read-only tree, so the author marks it `access: "ro"`. + expect(referencedMapping).toBeDefined(); + expect(referencedMapping?.kind).toBe("directory"); + expect(referencedMapping?.access).toBe("ro"); + } finally { + if (priorFlag === undefined) delete process.env[flagKey]; + else process.env[flagKey] = priorFlag; + } + }); + it("keeps the sandbox runtime core free of Codex-specific string literals", async () => { const coreSource = await readFile(new URL("./sandbox-managed-runtime.ts", import.meta.url), "utf8"); // The seam must be generic: no adapter (Codex) knowledge may live in the core. @@ -1490,4 +1828,97 @@ describe("sandbox managed runtime", () => { expect(coreSource).not.toMatch(/auth\.json/i); expect(coreSource).not.toMatch(/merge-extract|merge-decision/i); }); + + // End-to-end: drive the WHOLE prepare path — `prepareCommandManagedRuntime` + // building the client, syncing the anchor workspace, then staging the + // referenced projects — through a runner that runs real shell commands on the + // host filesystem (host FS stands in for the sandbox FS). The runner exposes no + // native syncIn, so staging rides the base64/tar fallback, the same transport a + // provider without native sync uses. In production the kill-switch + // `PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC` gates whether run prep resolves any + // referenced projects (OFF ⇒ none reach this layer). Enable it in-test only to + // model the ON scenario, and prove multi-project isolation plus one-failure + // isolation end-to-end. + it("stages multiple referenced projects into isolated sandbox dirs end-to-end, skipping a failing source", async () => { + const flagKey = "PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC"; + const priorFlag = process.env[flagKey]; + process.env[flagKey] = "1"; + try { + const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-e2e-additional-")); + cleanupDirs.push(rootDir); + + const localWorkspaceDir = path.join(rootDir, "local-workspace"); + const remoteWorkspaceDir = path.join(rootDir, "remote-workspace"); + await mkdir(localWorkspaceDir, { recursive: true }); + await mkdir(remoteWorkspaceDir, { recursive: true }); + await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor content\n", "utf8"); + + // Two real referenced-project checkouts (one with a nested file) plus a + // deliberately-missing source between them. + const first = path.join(rootDir, "referenced-first"); + const second = path.join(rootDir, "referenced-second"); + await mkdir(path.join(first, "docs"), { recursive: true }); + await mkdir(second, { recursive: true }); + await writeFile(path.join(first, "docs", "guide.md"), "first guide\n", "utf8"); + await writeFile(path.join(second, "notes.md"), "second notes\n", "utf8"); + + const runner: CommandManagedRuntimeRunner = { + execute: (input) => + new Promise((resolve) => { + const startedAt = new Date().toISOString(); + const command = + input.command === "sh" ? "/bin/sh" : input.command === "bash" ? "/bin/bash" : input.command; + const child = spawn(command, input.args ?? [], { cwd: input.cwd, env: { ...process.env, ...input.env } }); + let stdout = ""; + let stderr = ""; + child.stdout.on("data", (chunk) => { stdout += chunk.toString("utf8"); }); + child.stderr.on("data", (chunk) => { stderr += chunk.toString("utf8"); }); + child.on("error", () => resolve({ exitCode: 127, signal: null, timedOut: false, stdout, stderr, pid: null, startedAt })); + child.on("close", (code) => resolve({ exitCode: code ?? 0, signal: null, timedOut: false, stdout, stderr, pid: child.pid ?? null, startedAt })); + if (input.stdin != null) child.stdin.write(input.stdin); + child.stdin.end(); + }), + }; + + const prepared = await prepareCommandManagedRuntime({ + runner, + spec: { remoteCwd: remoteWorkspaceDir, timeoutMs: 30_000 }, + adapterKey: "test-adapter", + workspaceLocalDir: localWorkspaceDir, + additionalSources: [ + { localPath: first, projectId: "proj-first" }, + { localPath: path.join(rootDir, "referenced-missing"), projectId: "proj-missing" }, + { localPath: second, projectId: "proj-second" }, + ], + }); + + const runtimeRootDir = path.posix.join(remoteWorkspaceDir, ".paperclip-runtime", "test-adapter"); + + // The anchor workspace synced normally and stays byte-identical. + await expect(readFile(path.join(remoteWorkspaceDir, "README.md"), "utf8")).resolves.toBe("anchor content\n"); + + // Each healthy referenced project landed in its OWN isolated dir; the + // missing one is skipped, not fatal. + expect(Object.keys(prepared.additionalSourceDirs).sort()).toEqual(["proj-first", "proj-second"]); + expect(prepared.additionalSourceDirs["proj-first"]).toBe(path.posix.join(runtimeRootDir, "project-proj-first")); + expect(prepared.additionalSourceDirs["proj-second"]).toBe(path.posix.join(runtimeRootDir, "project-proj-second")); + expect(prepared.additionalSourceDirs["proj-missing"]).toBeUndefined(); + + await expect(readFile(path.join(prepared.additionalSourceDirs["proj-first"], "docs", "guide.md"), "utf8")) + .resolves.toBe("first guide\n"); + await expect(readFile(path.join(prepared.additionalSourceDirs["proj-second"], "notes.md"), "utf8")) + .resolves.toBe("second notes\n"); + + // Neither project's tree leaked into the anchor workspace or into the other + // project's dir. + await expect(readFile(path.join(remoteWorkspaceDir, "notes.md"), "utf8")).rejects.toMatchObject({ code: "ENOENT" }); + await expect(readFile(path.join(prepared.additionalSourceDirs["proj-first"], "notes.md"), "utf8")).rejects + .toMatchObject({ code: "ENOENT" }); + await expect(readFile(path.join(runtimeRootDir, "project-proj-missing"), "utf8")).rejects + .toMatchObject({ code: "ENOENT" }); + } finally { + if (priorFlag === undefined) delete process.env[flagKey]; + else process.env[flagKey] = priorFlag; + } + }); }); diff --git a/packages/adapter-utils/src/sandbox-managed-runtime.ts b/packages/adapter-utils/src/sandbox-managed-runtime.ts index 1eef71f91c7..0cff256464c 100644 --- a/packages/adapter-utils/src/sandbox-managed-runtime.ts +++ b/packages/adapter-utils/src/sandbox-managed-runtime.ts @@ -121,6 +121,22 @@ export interface SandboxManagedRuntimeAsset { restore?: (ctx: SandboxManagedRuntimeAssetRestoreContext) => Promise; } +/** + * A referenced (additional) project to stage into the run sandbox as a plain, + * read-only tree. `localPath` is the host checkout directory. Upstream code + * already authorized and realized this directory (`project:read`); this layer + * adds no authorization logic. `projectId` names the isolated remote + * subdirectory (`project-` under the runtime root) the tree lands in. + * + * Additional sources are plain trees only. They never carry the anchor + * workspace's git-history, overlay, or `.paperclip-runtime` preservation + * semantics — those stay anchor-only. + */ +export interface SandboxAdditionalSource { + localPath: string; + projectId: string; +} + /** * Per-call byte-level progress hook. `transferredBytes`/`totalBytes` are decoded * file bytes (not the base64 wire size). `totalBytes` is null when the size is @@ -145,6 +161,24 @@ export interface SandboxSyncFileMapping { mode?: number; exclude?: string[]; followSymlinks?: boolean; + /** + * Advisory read-write intent for the sandbox target. `"rw"` marks a target the + * agent may change and keep; `"ro"` marks a read-only tree. An absent value + * defaults to `"ro"` (read-only is the safe default for an advisory signal). + * The field is advisory metadata for an optional sandbox feedback wrapper. It + * does not change the transfer and adds no security. + */ + access?: "rw" | "ro"; + /** + * The sandbox directory that becomes read-write when `access` is `"rw"` and a + * post-upload command extracts `targetPath` into a different directory. A tar + * mapping uploads an archive under the runtime root, so its `targetPath` is the + * staging archive, not the directory the extract command fills. This field + * names that final destination directory. When absent, the read-write + * destination is the parent directory of `targetPath`. Advisory; ignored when + * `access` is not `"rw"`. + */ + writablePath?: string; } /** @@ -207,7 +241,7 @@ export interface SandboxManagedRuntimeClient { ): Promise; listFiles(remotePath: string): Promise; remove(remotePath: string): Promise; - run(command: string, options: { timeoutMs: number; noProfile?: boolean }): Promise; + run(command: string, options: { timeoutMs: number }): Promise; /** * Optional native inbound transfer. Present only when the sandbox provider * advertises both `environmentSyncIn` and `environmentSyncOut`; otherwise the @@ -259,6 +293,13 @@ export interface PreparedSandboxManagedRuntime { workspaceRemoteDir: string; runtimeRootDir: string; assetDirs: Record; + /** + * Remote directory of each additional (referenced) project that staged + * successfully, keyed by `projectId`. A project whose staging failed is + * absent (per-project failure isolation). Empty when no additional sources + * were requested. + */ + additionalSourceDirs: Record; restoreWorkspace(onProgress?: RuntimeProgressSink): Promise; } @@ -637,6 +678,12 @@ export async function prepareSandboxManagedRuntime(input: { workspaceExclude?: string[]; preserveAbsentOnRestore?: string[]; assets?: SandboxManagedRuntimeAsset[]; + /** + * Referenced (additional) projects to stage into the sandbox as plain, + * read-only trees, each in its own isolated `project-` directory. + * Defaults to none, so a legacy/anchor-only call is behavior-identical. + */ + additionalSources?: SandboxAdditionalSource[]; // Upload progress sink. Threaded for the byte-counting transport rewrite; the // child task wires it into writeFile/readFile. onProgress?: RuntimeProgressSink; @@ -686,6 +733,14 @@ export async function prepareSandboxManagedRuntime(input: { // Opaque, ordered, non-sensitive operation tokens — never a caller/asset id. const nextSyncOperationId = () => `sync-op-${++syncOperationSeq}`; + // Remote directory of each additional (referenced) project that stages + // successfully, keyed by projectId. A project that fails to stage is absent. + const additionalSourceDirs: Record = {}; + // Additional projects stage as plain trees. Drop the heavy build/cache dirs a + // reference tree does not need, and `.git` — additional sources never carry + // git-history semantics (anchor-only). + const additionalSourceExclude = mergeExcludes(SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES, [".git"]); + // Every delegated post-upload command (extract/wipe/remove-deleted/asset merge) // must run under the run-specific timeout (`spec.timeoutMs`), not the provider // sync client's default timeout — the two can differ, and before staging was @@ -709,74 +764,93 @@ export async function prepareSandboxManagedRuntime(input: { ...(input.preserveAbsentOnRestore ?? []), ]); - // Build one `SandboxSyncOperation` uploading a host tarball as a single file - // mapping (rides native `uploadFiles`, or the base64-tar fallback) with the - // extract/wipe/merge steps carried as ordered `postUploadCommands`, confine - // it, and delegate to `syncIn`. `finish` emits the terminal progress line. - const stageTarball = async (input2: { - tarPath: string; - remoteTar: string; - postUploadCommands: SandboxPostUploadCommand[]; + // Stage one source directory into an isolated remote subdirectory through the + // unified `syncIn` seam. Both the workspace/asset tar path and the additional + // (referenced) project path use it: build one `SandboxSyncOperation` from the + // caller's `files` mappings (a host tarball as a single `file` mapping, or a + // whole `directory` mapping), carry any extract/wipe/merge steps as ordered + // `postUploadCommands`, confine the operation's source and target to their + // own roots (fail-closed), and delegate to `syncIn` (native transfer, or the + // base64-tar fallback). `finish` emits the terminal progress line. + const stageConfinedSyncIn = async (params: { + files: SandboxSyncFileMapping[]; + postUploadCommands?: SandboxPostUploadCommand[]; + sourceRoots: string[]; + targetRoots: string[]; progressLabel: string; statusPhase: RuntimeStatusPhase; + progressBytes: number; }): Promise => { - const tarSize = (await fs.stat(input2.tarPath)).size; const operations: SandboxSyncOperation[] = [{ operationId: nextSyncOperationId(), - files: [{ sourcePath: input2.tarPath, targetPath: input2.remoteTar, kind: "file" }], - postUploadCommands: withRunTimeout(input2.postUploadCommands), + files: params.files, + ...(params.postUploadCommands + ? { postUploadCommands: withRunTimeout(params.postUploadCommands) } + : {}), }]; assertSyncOperationsConfined(operations, { - sourceRoots: [tempDir], - targetRoots: [runtimeRootDir], + sourceRoots: params.sourceRoots, + targetRoots: params.targetRoots, }); const upload = makeTransferProgress( input.onProgress, "Syncing", "to", - input2.progressLabel, - { sink: input.onRuntimeProgress, phase: input2.statusPhase }, + params.progressLabel, + { sink: input.onRuntimeProgress, phase: params.statusPhase }, ); await syncIn(operations); - await upload.finish(tarSize, tarSize); + await upload.finish(params.progressBytes, params.progressBytes); }; - if (syncWorkspace && gitSnapshot) { - await emitRuntimeStatus(input.onRuntimeProgress, "git_sync", "Syncing git history to sandbox"); - await withShallowGitWorkspaceClone({ - localDir: input.workspaceLocalDir, - snapshot: gitSnapshot, - }, async (cloneDir) => { - // git-workspace preserves `.paperclip-runtime` on the target and the - // workspace overlay merges on top rather than replacing — expressed as - // the operation's ordered post-upload commands, not a plain replace. - const gitTarPath = path.join(tempDir, "git-workspace.tar"); - await createTarballFromDirectory({ - localDir: cloneDir, - archivePath: gitTarPath, - exclude: [".paperclip-runtime"], - }); - const remoteGitTar = path.posix.join(runtimeRootDir, "git-workspace-upload.tar"); - await stageTarball({ - tarPath: gitTarPath, - remoteTar: remoteGitTar, - postUploadCommands: [{ - command: buildWorkspaceTarExtractCommand({ - workspaceRemoteDir, - remoteTar: remoteGitTar, - wipeExceptNames: [".paperclip-runtime"], - }), - }], - progressLabel: "git history", - statusPhase: "git_sync", - }); - }); - } - if (syncWorkspace) { + // A git-backed workspace and a plain workspace both stage through ONE + // confined `syncIn` operation. A git-backed workspace carries TWO host tars — + // the git-history clone and the working-tree overlay — as two `file` mappings + // on the SAME operation, with their extract commands as ordered + // `postUploadCommands`. One operation shares one mkdir, one confine guard, one + // `uploadFiles`, and one rename exec, so the second `syncIn` round trip is + // removed. Build the whole merged file set and command list BEFORE the confine + // guard runs (inside `stageConfinedSyncIn`); never append a mapping after it. + const workspaceFiles: SandboxSyncFileMapping[] = []; + const workspacePostUploadCommands: SandboxPostUploadCommand[] = []; + let workspaceUploadBytes = 0; + + // 1. git-history tar (git-backed workspace only). Both tar targets live under + // `runtimeRootDir` (`.paperclip-runtime/`). The git extract + // wipes the target tree EXCEPT `.paperclip-runtime`, so the overlay tar, + // which sits under `.paperclip-runtime`, survives to run its own extract. + if (gitSnapshot) { + await emitRuntimeStatus(input.onRuntimeProgress, "git_sync", "Syncing git history to sandbox"); + const gitTarPath = path.join(tempDir, "git-workspace.tar"); + const remoteGitTar = path.posix.join(runtimeRootDir, "git-workspace-upload.tar"); + await withShallowGitWorkspaceClone({ + localDir: input.workspaceLocalDir, + snapshot: gitSnapshot, + }, async (cloneDir) => { + await createTarballFromDirectory({ + localDir: cloneDir, + archivePath: gitTarPath, + exclude: [".paperclip-runtime"], + }); + }); + workspaceFiles.push({ sourcePath: gitTarPath, targetPath: remoteGitTar, kind: "file", access: "rw", writablePath: workspaceRemoteDir }); + workspacePostUploadCommands.push({ + command: buildWorkspaceTarExtractCommand({ + workspaceRemoteDir, + remoteTar: remoteGitTar, + wipeExceptNames: [".paperclip-runtime"], + }), + }); + workspaceUploadBytes += (await fs.stat(gitTarPath)).size; + } + + // 2. workspace-overlay tar. A git-backed overlay merges on top of the just + // extracted git tree (no wipe); a plain workspace wipes every child except + // the preserved names first. The extract runs AFTER the git extract. + await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing workspace to sandbox"); const workspaceTarPath = path.join(tempDir, "workspace.tar"); const workspaceArchiveDir = gitSnapshot ? path.join(tempDir, "workspace-overlay") : input.workspaceLocalDir; - await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing workspace to sandbox"); if (gitSnapshot) { await copySelectedWorkspaceEntries({ sourceDir: input.workspaceLocalDir, @@ -791,15 +865,15 @@ export async function prepareSandboxManagedRuntime(input: { exclude: gitSnapshot ? undefined : workspaceArchiveExclude, }); const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-upload.tar"); - // git overlay merges on top of the just-extracted git tree (no wipe); - // non-git workspace wipes every child except the preserved names first. - const workspacePostUploadCommands: SandboxPostUploadCommand[] = [{ + workspaceFiles.push({ sourcePath: workspaceTarPath, targetPath: remoteWorkspaceTar, kind: "file", access: "rw", writablePath: workspaceRemoteDir }); + workspacePostUploadCommands.push({ command: buildWorkspaceTarExtractCommand({ workspaceRemoteDir, remoteTar: remoteWorkspaceTar, wipeExceptNames: gitSnapshot ? null : [...preservedNames], }), - }]; + }); + // 3. Optional remove-deleted-paths command runs LAST, after both extracts. if (gitSnapshot && gitSnapshot.deletedPaths.length > 0) { workspacePostUploadCommands.push({ command: buildRemoveDeletedPathsCommand({ @@ -808,12 +882,19 @@ export async function prepareSandboxManagedRuntime(input: { }), }); } - await stageTarball({ - tarPath: workspaceTarPath, - remoteTar: remoteWorkspaceTar, + workspaceUploadBytes += (await fs.stat(workspaceTarPath)).size; + + // One confined `syncIn` for the whole merged workspace file set. The confine + // guard covers every mapping BEFORE any bytes upload (fail-closed): a source + // or target escape in EITHER tar mapping stops the upload of both. + await stageConfinedSyncIn({ + files: workspaceFiles, postUploadCommands: workspacePostUploadCommands, + sourceRoots: [tempDir], + targetRoots: [runtimeRootDir], progressLabel: "workspace", statusPhase: "config_sync", + progressBytes: workspaceUploadBytes, }); } @@ -835,7 +916,7 @@ export async function prepareSandboxManagedRuntime(input: { exclude: asset.exclude, }); const files: SandboxSyncFileMapping[] = [ - { sourcePath: assetTarPath, targetPath: remoteAssetTar, kind: "file" }, + { sourcePath: assetTarPath, targetPath: remoteAssetTar, kind: "file", access: "rw", writablePath: remoteAssetDir }, ]; // Stage provision helper files (e.g. the merge scripts) into the temp dir // and map them alongside the asset tar so they ride the same native upload. @@ -849,10 +930,14 @@ export async function prepareSandboxManagedRuntime(input: { : stageFile.contents; const stageHostPath = path.join(tempDir, `${asset.key}.stage.${safeName}`); await fs.writeFile(stageHostPath, stageBytes); + // A stage helper file (for example a merge script) is a read-only input + // that the provision command reads; the agent does not change it and does + // not keep it. So it is `access: "ro"` and never joins the writable set. files.push({ sourcePath: stageHostPath, targetPath: path.posix.join(runtimeRootDir, safeName), kind: "file", + access: "ro", }); } const postUploadCommand = asset.provision?.postUploadCommand?.({ @@ -860,25 +945,64 @@ export async function prepareSandboxManagedRuntime(input: { assetDir: remoteAssetDir, runtimeRootDir, }) ?? buildDefaultExtractRuntimeAssetCommand({ remoteAssetDir, remoteAssetTar }); - const operations: SandboxSyncOperation[] = [{ - operationId: nextSyncOperationId(), + const assetTarSize = (await fs.stat(assetTarPath)).size; + await stageConfinedSyncIn({ files, - postUploadCommands: withRunTimeout([{ command: postUploadCommand }]), - }]; - assertSyncOperationsConfined(operations, { + postUploadCommands: [{ command: postUploadCommand }], sourceRoots: [tempDir], targetRoots: [runtimeRootDir], + progressLabel: asset.key, + statusPhase: "config_sync", + progressBytes: assetTarSize, }); - const assetTarSize = (await fs.stat(assetTarPath)).size; - const assetUpload = makeTransferProgress( - input.onProgress, - "Syncing", - "to", - asset.key, - { sink: input.onRuntimeProgress, phase: "config_sync" }, - ); - await syncIn(operations); - await assetUpload.finish(assetTarSize, assetTarSize); + } + + // Stage each referenced (additional) project as a plain, read-only tree in + // its OWN isolated remote directory (`project-`). An additional + // project rides one confined `syncIn` directory mapping — a native directory + // transfer, or the base64-tar fallback — with source and target confined to + // their own roots. No workspace, git-history, or `.paperclip-runtime` + // semantics apply; those stay anchor-only. Per-project failure isolation: one + // project's confinement or sync failure logs a warning and is skipped, and + // the run plus the other projects continue. Only a project that stages + // successfully appears in `additionalSourceDirs`. + for (const source of input.additionalSources ?? []) { + const { localPath, projectId } = source; + const label = `project-${projectId}`; + try { + if (!path.posix.isAbsolute(localPath)) { + throw new Error(`additional source localPath is not an absolute path: ${localPath}`); + } + if ( + projectId.length === 0 || + projectId.includes("/") || + projectId.includes("\\") || + projectId.includes("..") + ) { + throw new Error(`additional source projectId is not a simple path segment: ${projectId}`); + } + const remoteProjectDir = path.posix.join(runtimeRootDir, label); + await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing referenced project to sandbox"); + await stageConfinedSyncIn({ + files: [{ + sourcePath: localPath, + targetPath: remoteProjectDir, + kind: "directory", + exclude: additionalSourceExclude, + access: "ro", + }], + sourceRoots: [localPath], + targetRoots: [remoteProjectDir], + progressLabel: label, + statusPhase: "config_sync", + progressBytes: 0, + }); + additionalSourceDirs[projectId] = remoteProjectDir; + } catch (error) { + console.warn( + `[paperclip] Failed to stage referenced project ${projectId}; skipping it. ${String(error)}`, + ); + } } }); @@ -892,6 +1016,7 @@ export async function prepareSandboxManagedRuntime(input: { workspaceRemoteDir, runtimeRootDir, assetDirs, + additionalSourceDirs, restoreWorkspace: async (onProgress?: RuntimeProgressSink) => { const restoreSink = onProgress ?? input.onProgress; if (!syncWorkspace) { diff --git a/packages/adapter-utils/src/ssh-fixture.test.ts b/packages/adapter-utils/src/ssh-fixture.test.ts index ba2da73222b..0b8a45a0bf4 100644 --- a/packages/adapter-utils/src/ssh-fixture.test.ts +++ b/packages/adapter-utils/src/ssh-fixture.test.ts @@ -170,6 +170,49 @@ describe("ssh env-lab fixture", () => { await stopSshEnvLabFixture(statePath); }, SSH_FIXTURE_TEST_TIMEOUT_MS); + it("builds a remote script that sources login profiles but no nvm", async () => { + const target = await buildSshSpawnTarget({ + spec: { + host: "ssh.example.test", + port: 22, + username: "ssh-user", + remoteCwd: "/srv/paperclip/workspace", + remoteWorkspacePath: "/srv/paperclip/workspace", + privateKey: null, + knownHosts: null, + strictHostKeyChecking: true, + }, + command: "node", + args: ["--version"], + env: { FOO: "bar" }, + }); + + // The remote script rides the last ssh argument. The SSH target is an + // operator-configured host that can expose `node` only through a login + // profile, so the wrapper sources the profiles. It no longer sources + // `nvm.sh`; a profile that adds nvm still runs. + const remoteScript = String(target.args.at(-1) ?? ""); + expect(remoteScript).not.toContain("nvm.sh"); + expect(remoteScript).not.toContain("NVM_DIR"); + // Source /etc/profile so a host that exposes the PATH through + // /etc/profile.d scripts still resolves node and the agent CLI. + expect(remoteScript).toContain("/etc/profile"); + expect(remoteScript).toContain(".profile"); + expect(remoteScript).toContain(".bash_profile"); + expect(remoteScript).toContain(".zprofile"); + // Fall back to .bashrc when no .bash_profile exists, so a host that adds + // nvm in .bashrc still resolves node under a non-login SSH command. + expect(remoteScript).toContain(".bashrc"); + // The last ssh argument wraps the script as `sh -c '...'`, so the inner + // quotes are escaped. Assert the command still runs: cd, env, and the argv. + expect(remoteScript).toContain("cd "); + expect(remoteScript).toContain("/srv/paperclip/workspace"); + expect(remoteScript).toContain("exec env "); + expect(remoteScript).toContain("node"); + expect(remoteScript).toContain("--version"); + await target.cleanup(); + }); + it("rejects invalid environment variable keys when constructing SSH spawn targets", async () => { await expect( buildSshSpawnTarget({ diff --git a/packages/adapter-utils/src/ssh.ts b/packages/adapter-utils/src/ssh.ts index d3efffd7252..b2aa909c9a8 100644 --- a/packages/adapter-utils/src/ssh.ts +++ b/packages/adapter-utils/src/ssh.ts @@ -1166,14 +1166,22 @@ export async function runSshCommand( } } - // Mirror buildSshSpawnTarget: source login profiles first, then run - // `env KEY=VAL cmd` so user-supplied identity overrides win over anything - // a profile re-exports. Without this, a remote profile that resets HOME - // / NVM_DIR / etc. would silently undo the explicit env passed in here. + // Mirror buildSshSpawnTarget: source the login profiles first, then run + // `env KEY=VAL cmd` so user-supplied identity overrides win over anything a + // profile re-exports. The SSH target is an operator-configured host, not a + // Paperclip sandbox image, so it can expose `node` or an agent CLI only + // through a login profile; a non-login SSH command would miss that PATH. + // Source `/etc/profile` first so a host that exposes the PATH through + // `/etc/profile.d` scripts still resolves node and the agent CLI. + // The script no longer sources `nvm.sh`; a profile that adds nvm still runs. + // .bash_profile typically sources .bashrc itself; only source .bashrc + // directly when no .bash_profile exists, so a host that adds nvm in + // .bashrc still resolves node without a double-run of the setup. const envArgs = envEntries.map(([key, value]) => `${key}=${shellQuote(value)}`); const remoteScript = [ + 'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi', 'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi', - 'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; fi', + 'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi', 'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi', envArgs.length > 0 ? `exec env ${envArgs.join(" ")} sh -c ${shellQuote(remoteCommand)}` @@ -1223,12 +1231,22 @@ export async function buildSshSpawnTarget(input: { .filter((entry): entry is [string, string] => typeof entry[1] === "string") .map(([key, value]) => `${key}=${shellQuote(value)}`); const remoteCommandParts = [shellQuote(input.command), ...input.args.map((arg) => shellQuote(arg))].join(" "); + // Source the login profiles first, then run `env KEY=VAL cmd` so + // user-supplied identity overrides win over anything a profile re-exports. + // The SSH target is an operator-configured host, not a Paperclip sandbox + // image, so it can expose `node` or an agent CLI only through a login + // profile; a non-login SSH command would miss that PATH. Source + // `/etc/profile` first so a host that exposes the PATH through + // `/etc/profile.d` scripts still resolves node and the agent CLI. The script + // no longer sources `nvm.sh`; a profile that adds nvm still runs. + // .bash_profile typically sources .bashrc itself; only source .bashrc + // directly when no .bash_profile exists, so a host that adds nvm in + // .bashrc still resolves node without a double-run of the setup. const remoteScript = [ + 'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi', 'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi', - 'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; fi', + 'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi', 'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi', - 'export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"', - '[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" >/dev/null 2>&1 || true', `cd ${shellQuote(input.spec.remoteCwd)}`, envArgs.length > 0 ? `exec env ${envArgs.join(" ")} ${remoteCommandParts}` diff --git a/packages/adapter-utils/src/types.ts b/packages/adapter-utils/src/types.ts index b0b16682e29..988e1318e7e 100644 --- a/packages/adapter-utils/src/types.ts +++ b/packages/adapter-utils/src/types.ts @@ -175,6 +175,14 @@ export interface AdapterExecutionContext { onRuntimeProgress?: RuntimeStatusSink; onSpawn?: (meta: { pid: number; processGroupId: number | null; startedAt: string }) => Promise; authToken?: string; + /** + * The injected OpenTelemetry startup trace context (tracer + root + * parent-context helper). The server passes the real, endpoint-gated + * implementation; when absent, the ACPX engine uses a no-op, so the whole + * span path stays inert. The type is an inline import so this module keeps no + * top-level dependency on the timing helper. + */ + startupTraceContext?: import("./acpx-engine/startup-timing.js").StartupTraceContext; } export interface AdapterModel { diff --git a/packages/db/src/migrations/0196_drop_cloud_upstream_tables.sql b/packages/db/src/migrations/0196_drop_cloud_upstream_tables.sql new file mode 100644 index 00000000000..f90bb718f1e --- /dev/null +++ b/packages/db/src/migrations/0196_drop_cloud_upstream_tables.sql @@ -0,0 +1,9 @@ +-- Drop the cloud sync (cloud upstream) tables. The host-to-host transport +-- has been removed in favor of the Import/Export flow, and nothing reads or +-- writes these tables any more. The sender tables (connections, runs) came +-- from 0089; the receiver-side tables never shipped, so these two are all +-- that exist. The feature was experimental and flag-gated off by default, +-- so its run history is intentionally discarded. +DROP TABLE IF EXISTS "cloud_upstream_runs"; +--> statement-breakpoint +DROP TABLE IF EXISTS "cloud_upstream_connections"; diff --git a/packages/db/src/migrations/0196_issue_watchdog_mode.sql b/packages/db/src/migrations/0197_issue_watchdog_mode.sql similarity index 100% rename from packages/db/src/migrations/0196_issue_watchdog_mode.sql rename to packages/db/src/migrations/0197_issue_watchdog_mode.sql diff --git a/packages/db/src/migrations/meta/_journal.json b/packages/db/src/migrations/meta/_journal.json index 2c394f799c3..e4f1bb2e74d 100644 --- a/packages/db/src/migrations/meta/_journal.json +++ b/packages/db/src/migrations/meta/_journal.json @@ -1363,8 +1363,15 @@ { "idx": 196, "version": "7", - "when": 1785170000002, - "tag": "0196_issue_watchdog_mode", + "when": 1785170001001, + "tag": "0196_drop_cloud_upstream_tables", + "breakpoints": true + }, + { + "idx": 197, + "version": "7", + "when": 1785170001002, + "tag": "0197_issue_watchdog_mode", "breakpoints": true } ] diff --git a/packages/db/src/schema/cloud_upstreams.ts b/packages/db/src/schema/cloud_upstreams.ts deleted file mode 100644 index 93b4341f7e4..00000000000 --- a/packages/db/src/schema/cloud_upstreams.ts +++ /dev/null @@ -1,75 +0,0 @@ -import { boolean, index, integer, jsonb, pgTable, text, timestamp, uuid } from "drizzle-orm/pg-core"; -import { companies } from "./companies.js"; - -export const cloudUpstreamConnections = pgTable( - "cloud_upstream_connections", - { - id: uuid("id").primaryKey().defaultRandom(), - companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }), - remoteUrl: text("remote_url").notNull(), - sourceInstanceId: text("source_instance_id").notNull(), - sourceInstanceFingerprint: text("source_instance_fingerprint").notNull(), - sourcePublicKey: text("source_public_key").notNull(), - // Stored through the Cloud Upstream service as an encrypted credential envelope. - privateKeyPem: text("private_key_pem").notNull(), - tokenStatus: text("token_status").notNull(), - scopes: text("scopes").array().notNull().default([]), - authorizedGlobalUserId: text("authorized_global_user_id"), - // Stored through the Cloud Upstream service as an encrypted credential envelope. - accessToken: text("access_token"), - tokenId: text("token_id"), - tokenExpiresAt: timestamp("token_expires_at", { withTimezone: true }), - - targetStackId: text("target_stack_id").notNull(), - targetStackSlug: text("target_stack_slug"), - targetStackDisplayName: text("target_stack_display_name"), - targetCompanyId: text("target_company_id").notNull(), - targetOrigin: text("target_origin").notNull(), - targetPrimaryHost: text("target_primary_host").notNull(), - targetProduct: text("target_product").notNull(), - targetSchemaMajor: integer("target_schema_major").notNull(), - targetMaxChunkBytes: integer("target_max_chunk_bytes").notNull(), - - pendingState: text("pending_state"), - pendingCodeVerifier: text("pending_code_verifier"), - pendingRedirectUri: text("pending_redirect_uri"), - pendingTokenUrl: text("pending_token_url"), - - lastRunId: uuid("last_run_id"), - createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), - updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), - }, - (table) => [ - index("cloud_upstream_connections_company_idx").on(table.companyId), - ], -); - -export const cloudUpstreamRuns = pgTable( - "cloud_upstream_runs", - { - id: uuid("id").primaryKey().defaultRandom(), - connectionId: uuid("connection_id").notNull().references(() => cloudUpstreamConnections.id, { onDelete: "cascade" }), - companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }), - remoteRunId: text("remote_run_id"), - status: text("status").notNull(), - activeStep: text("active_step").notNull(), - progressPercent: integer("progress_percent").notNull().default(0), - dryRun: boolean("dry_run").notNull().default(false), - retryOfRunId: uuid("retry_of_run_id"), - summary: jsonb("summary").$type().notNull().default([]), - warnings: jsonb("warnings").$type().notNull().default([]), - conflicts: jsonb("conflicts").$type().notNull().default([]), - events: jsonb("events").$type().notNull().default([]), - report: jsonb("report").$type>().notNull().default({}), - idempotencyKey: text("idempotency_key").notNull(), - manifestHash: text("manifest_hash").notNull(), - targetUrl: text("target_url"), - createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(), - updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(), - completedAt: timestamp("completed_at", { withTimezone: true }), - }, - (table) => [ - index("cloud_upstream_runs_company_created_idx").on(table.companyId, table.createdAt), - index("cloud_upstream_runs_connection_idx").on(table.connectionId), - ], -); diff --git a/packages/db/src/schema/index.ts b/packages/db/src/schema/index.ts index 8e558866b7a..3589c16ab04 100644 --- a/packages/db/src/schema/index.ts +++ b/packages/db/src/schema/index.ts @@ -2,7 +2,6 @@ export { companies } from "./companies.js"; export { companyLogos } from "./company_logos.js"; export { authUsers, authSessions, authAccounts, authVerifications } from "./auth.js"; export { instanceSettings } from "./instance_settings.js"; -export { cloudUpstreamConnections, cloudUpstreamRuns } from "./cloud_upstreams.js"; export { instanceUserRoles } from "./instance_user_roles.js"; export { userSidebarPreferences } from "./user_sidebar_preferences.js"; export { agents } from "./agents.js"; diff --git a/packages/db/src/table-size-estimates.ts b/packages/db/src/table-size-estimates.ts index 8a6d680af4d..f701fc5ea2c 100644 --- a/packages/db/src/table-size-estimates.ts +++ b/packages/db/src/table-size-estimates.ts @@ -92,8 +92,6 @@ export const LOCAL_TABLE_ROW_COUNTS = [ { table: "budget_incidents", localRows: 0 }, { table: "budget_policies", localRows: 0 }, { table: "cli_auth_challenges", localRows: 0 }, - { table: "cloud_upstream_connections", localRows: 0 }, - { table: "cloud_upstream_runs", localRows: 0 }, { table: "company_logos", localRows: 0 }, { table: "company_secret_provider_configs", localRows: 0 }, { table: "company_skill_comments", localRows: 0 }, diff --git a/packages/db/src/test-embedded-postgres.test.ts b/packages/db/src/test-embedded-postgres.test.ts new file mode 100644 index 00000000000..1d4f78c797d --- /dev/null +++ b/packages/db/src/test-embedded-postgres.test.ts @@ -0,0 +1,113 @@ +import fs from "node:fs"; +import { afterEach, describe, expect, it } from "vitest"; +import { + __embeddedPostgresStartMaxAttemptsForTests as MAX_ATTEMPTS, + __setEmbeddedPostgresCtorProviderForTests, + __startEmbeddedPostgresWithRetryForTests as startWithRetry, +} from "./test-embedded-postgres.js"; + +// A fake embedded-postgres constructor. It records every constructed instance so +// the test can assert the retry uses a fresh port and a fresh data directory each +// attempt. `start()` emits the same output the real cluster writes for a port +// conflict, then rejects with an empty message (the real rejection shape). The +// option type matches the real constructor so no type cast is needed. +type FakeOptions = { + databaseDir: string; + user: string; + password: string; + port: number; + persistent: boolean; + initdbFlags?: string[]; + onLog?: (message: unknown) => void; + onError?: (message: unknown) => void; +}; + +const BIND_CONFLICT_LOG = 'could not bind IPv4 address "127.0.0.1": Address already in use'; + +function makeFakeCtor(failFirst: number) { + const constructed: FakeOptions[] = []; + let started = 0; + + class FakeEmbeddedPostgres { + private readonly options: FakeOptions; + constructor(options: FakeOptions) { + this.options = options; + constructed.push(options); + } + async initialise(): Promise {} + async start(): Promise { + started += 1; + if (started <= failFirst) { + // Mirror the real failure: Postgres logs the reason, then `start()` + // rejects with an Error whose message is empty. + this.options.onLog?.(BIND_CONFLICT_LOG); + throw new Error(); + } + } + async stop(): Promise {} + } + + return { ctor: FakeEmbeddedPostgres, constructed }; +} + +describe("startEmbeddedPostgresWithRetry", () => { + afterEach(() => { + __setEmbeddedPostgresCtorProviderForTests(null); + }); + + it("recovers from a transient port conflict and returns on a later attempt", async () => { + const { ctor, constructed } = makeFakeCtor(2); + __setEmbeddedPostgresCtorProviderForTests(async () => ctor); + + const started = await startWithRetry("paperclip-retry-recover-"); + + // The first two attempts fail, the third succeeds. + expect(constructed).toHaveLength(3); + + // Each attempt uses a fresh data directory. The two failed directories are + // removed. The returned directory still exists. + const dataDirs = constructed.map((options) => options.databaseDir); + expect(new Set(dataDirs).size).toBe(3); + expect(fs.existsSync(dataDirs[0])).toBe(false); + expect(fs.existsSync(dataDirs[1])).toBe(false); + expect(started.dataDir).toBe(dataDirs[2]); + expect(fs.existsSync(started.dataDir)).toBe(true); + + // Each attempt allocates a port. + for (const options of constructed) { + expect(Number.isInteger(options.port)).toBe(true); + expect(options.port).toBeGreaterThan(0); + } + + // Clean up the returned attempt. + await started.instance.stop(); + fs.rmSync(started.dataDir, { recursive: true, force: true }); + }); + + it("throws with the real Postgres output after the attempt bound", async () => { + const { ctor, constructed } = makeFakeCtor(Number.POSITIVE_INFINITY); + __setEmbeddedPostgresCtorProviderForTests(async () => ctor); + + await expect(startWithRetry("paperclip-retry-fail-")).rejects.toThrow(/after \d+ attempts/); + + // The retry stops at the bound and does not loop forever. + expect(constructed).toHaveLength(MAX_ATTEMPTS); + + // Every failed attempt removes its data directory. + for (const options of constructed) { + expect(fs.existsSync(options.databaseDir)).toBe(false); + } + }); + + it("keeps the real failure reason instead of a generic fallback", async () => { + const { ctor } = makeFakeCtor(Number.POSITIVE_INFINITY); + __setEmbeddedPostgresCtorProviderForTests(async () => ctor); + + const error = await startWithRetry("paperclip-retry-reason-").catch((caught: unknown) => caught); + + expect(error).toBeInstanceOf(Error); + // The thrown message carries the captured Postgres output, not only the + // generic "embedded Postgres startup failed" text. + expect((error as Error).message).toContain("Address already in use"); + }); +}); diff --git a/packages/db/src/test-embedded-postgres.ts b/packages/db/src/test-embedded-postgres.ts index d88ca59c377..e85fd1a7218 100644 --- a/packages/db/src/test-embedded-postgres.ts +++ b/packages/db/src/test-embedded-postgres.ts @@ -3,6 +3,10 @@ import net from "node:net"; import os from "node:os"; import path from "node:path"; import { applyPendingMigrations, ensurePostgresDatabase } from "./client.js"; +import { + createEmbeddedPostgresLogBuffer, + formatEmbeddedPostgresError, +} from "./embedded-postgres-error.js"; import { prepareEmbeddedPostgresNativeRuntime } from "./embedded-postgres-native.js"; type EmbeddedPostgresInstance = { @@ -47,12 +51,29 @@ function getReservedTestPorts(): Set { return new Set(configuredPorts.filter((port) => Number.isInteger(port) && port > 0 && port <= 65535)); } -async function getEmbeddedPostgresCtor(): Promise { +type EmbeddedPostgresCtorProvider = () => Promise; + +async function loadEmbeddedPostgresCtor(): Promise { const mod = await import("embedded-postgres"); await prepareEmbeddedPostgresNativeRuntime(); return mod.default as EmbeddedPostgresCtor; } +let embeddedPostgresCtorProvider: EmbeddedPostgresCtorProvider = loadEmbeddedPostgresCtor; + +// Test seam. Replace the embedded-postgres constructor provider so a test can +// simulate a failed start without the native runtime. Pass `null` to restore +// the default provider. This module is test support only, so the seam is safe. +export function __setEmbeddedPostgresCtorProviderForTests( + provider: EmbeddedPostgresCtorProvider | null, +): void { + embeddedPostgresCtorProvider = provider ?? loadEmbeddedPostgresCtor; +} + +async function getEmbeddedPostgresCtor(): Promise { + return await embeddedPostgresCtorProvider(); +} + async function getAvailablePort(): Promise { const reservedPorts = getReservedTestPorts(); for (let attempt = 0; attempt < 20; attempt += 1) { @@ -88,6 +109,11 @@ async function createEmbeddedPostgresTestInstance(tempDirPrefix: string) { const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), tempDirPrefix)); const port = await getAvailablePort(); const EmbeddedPostgres = await getEmbeddedPostgresCtor(); + // Postgres writes the true reason for a failed start to its output, for + // example `could not bind IPv4 address "127.0.0.1": Address already in use`. + // The `start()` rejection carries an empty message, so we capture the output + // in a bounded buffer and surface it in the thrown error. + const logBuffer = createEmbeddedPostgresLogBuffer(); const instance = new EmbeddedPostgres({ databaseDir: dataDir, user: "paperclip", @@ -95,11 +121,11 @@ async function createEmbeddedPostgresTestInstance(tempDirPrefix: string) { port, persistent: true, initdbFlags: ["--encoding=UTF8", "--locale=C", "--lc-messages=C"], - onLog: () => {}, - onError: () => {}, + onLog: (message) => logBuffer.append(message), + onError: (message) => logBuffer.append(message), }); - return { dataDir, port, instance }; + return { dataDir, port, instance, getRecentLogs: () => logBuffer.getRecentLogs() }; } function cleanupEmbeddedPostgresTestDirs(dataDir: string) { @@ -161,34 +187,72 @@ async function stopEmbeddedPostgresBounded( } } -function formatEmbeddedPostgresError(error: unknown): string { - if (error instanceof Error && error.message.length > 0) return error.message; - if (typeof error === "string" && error.length > 0) return error; - return "embedded Postgres startup failed"; +// Upper bound on start attempts. `getAvailablePort` uses a check-then-use probe: +// it binds port 0, reads the assigned port, closes the probe, then Postgres binds +// that port. Under load another process can take the port in that window, so the +// bind fails with "Address already in use" and `start()` rejects. Each retry uses +// a fresh port and a fresh data directory, so a transient collision clears. +const EMBEDDED_POSTGRES_START_MAX_ATTEMPTS = 5; + +// Start one embedded Postgres cluster with a bounded retry. Each attempt gets a +// fresh port and a fresh data directory. On a failed attempt we stop the cluster +// and remove its data directory before the next attempt. After the last attempt +// we throw with the real Postgres output so the failure is loud and diagnosable. +async function startEmbeddedPostgresWithRetry(tempDirPrefix: string): Promise<{ + port: number; + dataDir: string; + instance: EmbeddedPostgresInstance; +}> { + let lastError = new Error("embedded Postgres startup failed"); + + for (let attempt = 1; attempt <= EMBEDDED_POSTGRES_START_MAX_ATTEMPTS; attempt += 1) { + const created = await createEmbeddedPostgresTestInstance(tempDirPrefix); + try { + await created.instance.initialise(); + await created.instance.start(); + return { port: created.port, dataDir: created.dataDir, instance: created.instance }; + } catch (error) { + lastError = formatEmbeddedPostgresError(error, { + fallbackMessage: "embedded Postgres startup failed", + recentLogs: created.getRecentLogs(), + }); + // Stop the failed cluster and remove its data directory. The next attempt + // allocates a fresh port and a fresh data directory. + await stopEmbeddedPostgresBounded(created.instance, () => + cleanupEmbeddedPostgresTestDirs(created.dataDir), + ); + } + } + + throw new Error( + `Failed to start embedded PostgreSQL test database after ${EMBEDDED_POSTGRES_START_MAX_ATTEMPTS} attempts: ${lastError.message}`, + ); } +// Test-only accessors. Production callers use `startEmbeddedPostgresTestDatabase` +// or `getEmbeddedPostgresTestSupport`. A test drives the bounded retry directly +// so it does not need a real Postgres connection. +export const __startEmbeddedPostgresWithRetryForTests = startEmbeddedPostgresWithRetry; +export const __embeddedPostgresStartMaxAttemptsForTests = EMBEDDED_POSTGRES_START_MAX_ATTEMPTS; + async function probeEmbeddedPostgresSupport(): Promise { - let dataDir: string | null = null; - let instance: EmbeddedPostgresInstance | null = null; + let started: { dataDir: string; instance: EmbeddedPostgresInstance } | null = null; try { - const created = await createEmbeddedPostgresTestInstance( - "paperclip-embedded-postgres-probe-", - ); - dataDir = created.dataDir; - instance = created.instance; - await instance.initialise(); - await instance.start(); + started = await startEmbeddedPostgresWithRetry("paperclip-embedded-postgres-probe-"); return { supported: true }; } catch (error) { return { supported: false, - reason: formatEmbeddedPostgresError(error), + reason: formatEmbeddedPostgresError(error, { + fallbackMessage: "embedded Postgres startup failed", + }).message, }; } finally { - await stopEmbeddedPostgresBounded(instance, () => { - if (dataDir) cleanupEmbeddedPostgresTestDirs(dataDir); - }); + if (started) { + const { dataDir, instance } = started; + await stopEmbeddedPostgresBounded(instance, () => cleanupEmbeddedPostgresTestDirs(dataDir)); + } } } @@ -202,17 +266,11 @@ export async function getEmbeddedPostgresTestSupport(): Promise { - let dataDir: string | null = null; - let instance: EmbeddedPostgresInstance | null = null; + // The bounded retry hardens the cluster start against the port race. It throws + // with the real Postgres output if every attempt fails. + const { port, dataDir, instance } = await startEmbeddedPostgresWithRetry(tempDirPrefix); try { - const created = await createEmbeddedPostgresTestInstance(tempDirPrefix); - dataDir = created.dataDir; - instance = created.instance; - const { port } = created; - await instance.initialise(); - await instance.start(); - const adminConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${port}/postgres`; await ensurePostgresDatabase(adminConnectionString, "paperclip"); const connectionString = `postgres://paperclip:paperclip@127.0.0.1:${port}/paperclip`; @@ -221,17 +279,17 @@ export async function startEmbeddedPostgresTestDatabase( return { connectionString, cleanup: async () => { - await stopEmbeddedPostgresBounded(instance, () => { - if (dataDir) cleanupEmbeddedPostgresTestDirs(dataDir); - }); + await stopEmbeddedPostgresBounded(instance, () => cleanupEmbeddedPostgresTestDirs(dataDir)); }, }; } catch (error) { - await stopEmbeddedPostgresBounded(instance, () => { - if (dataDir) cleanupEmbeddedPostgresTestDirs(dataDir); - }); + await stopEmbeddedPostgresBounded(instance, () => cleanupEmbeddedPostgresTestDirs(dataDir)); throw new Error( - `Failed to start embedded PostgreSQL test database: ${formatEmbeddedPostgresError(error)}`, + `Failed to start embedded PostgreSQL test database: ${ + formatEmbeddedPostgresError(error, { + fallbackMessage: "embedded Postgres startup failed", + }).message + }`, ); } } diff --git a/packages/plugins/sandbox-providers/SANDBOX-REQUIREMENTS.md b/packages/plugins/sandbox-providers/SANDBOX-REQUIREMENTS.md new file mode 100644 index 00000000000..1d762085a3a --- /dev/null +++ b/packages/plugins/sandbox-providers/SANDBOX-REQUIREMENTS.md @@ -0,0 +1,70 @@ +# Sandbox Runtime Requirements + +This document states the sandbox environment as a contract. The sandbox owner +must meet this contract. The Paperclip runtime does not build the environment at +exec time. The environment is a requirement, not a build step. + +This document states requirements. It does not state build steps. + +## Required on PATH + +- `node` must be installed and on the PATH. +- Each agent CLI that the run uses must be installed and on the PATH. The set of + agent CLIs includes `claude`, `codex`, `gemini`, and similar CLIs. +- The owner installs only the CLIs that the run uses. The owner does not need to + install a CLI that no run uses. + +## Runtime dependencies + +The sandbox execution and synchronization paths need more than `node` and the +agent CLIs. The owner must also supply these: + +- A POSIX shell as `sh`, normally `/bin/sh`. The runtime runs each command with + `sh -c