mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 19:35:13 +01:00
Merge origin/master: resolve cloud-sync removal and renumber watchdog-mode migration to 0197
Master removed the Cloud Sync experimental flag and took migration slot 0196 (drop_cloud_upstream_tables), so 0196_issue_watchdog_mode moves to 0197 with a later journal timestamp. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
commit
c6895a6840
205 files changed
+18508
-6716
No files matched your search
@@ -0,0 +1,265 @@
|
||||
---
|
||||
name: create-paperclip-bundled-skill
|
||||
description: >
|
||||
Turn an idea, tweet, or task into a skill in the Paperclip skills catalog
|
||||
(packages/skills-catalog). Use when asked to FIND or MAKE a skill and publish
|
||||
it as a bundled/optional catalog skill: research prior art, reference or
|
||||
author it, add examples, regenerate the manifest, open a PR.
|
||||
---
|
||||
|
||||
# Create a Paperclip Bundled Skill
|
||||
|
||||
Take source material — a tweet, a task description, a blog post, "make a skill
|
||||
that does X" — and land it as a skill in the Paperclip skills catalog
|
||||
(`packages/skills-catalog/`), delivered as a reviewed PR. The catalog is the
|
||||
shelf every Paperclip company browses and installs from, so the bar is: correct
|
||||
metadata, useful instructions, worked examples, and a clean validation run.
|
||||
|
||||
The core rule is **FIND before MAKE**: if a good skill already exists (in the
|
||||
catalog, in this repo, or published on GitHub), reference or adapt it instead
|
||||
of writing a duplicate from scratch.
|
||||
|
||||
## When to use
|
||||
|
||||
- A human sends a tweet/link/idea and asks for it to become a Paperclip skill.
|
||||
- A task asks to bundle an existing repo skill into the catalog.
|
||||
- A task asks to add an external published skill to the catalog.
|
||||
|
||||
## When not to use
|
||||
|
||||
- The skill is company-private (belongs in that company's library via the
|
||||
Skills UI/API, not the shipped catalog).
|
||||
- You only need a repo-internal agent skill for working on Paperclip itself —
|
||||
that goes in `.agents/skills/` or `skills/`, with no catalog machinery.
|
||||
|
||||
## Step 0 — Capture the source material
|
||||
|
||||
Understand exactly what the skill should teach before writing anything.
|
||||
|
||||
**Tweets / X links.** Use the `xc` CLI (X API client). Paperclip engineering
|
||||
agent environments ship it preinstalled and pre-authenticated; it is not a
|
||||
tool you install or mint credentials for yourself. Check availability before
|
||||
relying on it:
|
||||
|
||||
```sh
|
||||
command -v xc && xc whoami # on PATH and authenticated? if not, use the fallback below
|
||||
```
|
||||
|
||||
```sh
|
||||
xc get <post-url-or-id> --json # the post itself (conversation_id, author)
|
||||
xc search 'conversation_id:<id>' --archive --json # rest of the thread (>7 days old needs --archive)
|
||||
xc user <username> # author context
|
||||
xc search '<topic keywords>' -n 30 # related discussion
|
||||
```
|
||||
|
||||
If `xc` is not on PATH, is unauthenticated, or the account lacks read access
|
||||
(the check above fails for any reason), delegate the
|
||||
fetch to a teammate with X/Twitter access (e.g. the Content Strategist agent)
|
||||
via a child issue: give them the URL and ask for full text of the post + thread
|
||||
+ any linked content.
|
||||
|
||||
**Other sources.** Fetch linked articles/READMEs directly. Record the source
|
||||
URL — it goes in the skill body or PR description as attribution.
|
||||
|
||||
Distill: what is the repeatable procedure? What inputs does it take? What does
|
||||
"done" look like? If the source is just an aspiration ("agents should write
|
||||
better commit messages"), you are authoring the procedure yourself — say so in
|
||||
the PR.
|
||||
|
||||
## Step 1 — FIND: search for an existing skill
|
||||
|
||||
Search in this order; stop when you have a clear winner.
|
||||
|
||||
1. **Already in the catalog?** Avoid duplicates (duplicate slugs fail the
|
||||
build):
|
||||
```sh
|
||||
grep -i '<topic>' packages/skills-catalog/generated/catalog.json
|
||||
ls packages/skills-catalog/catalog/{bundled,optional}/*/
|
||||
```
|
||||
2. **Already in this repo?** Check `.agents/skills/`, `skills/`, and issue
|
||||
history (`gh search issues` / Paperclip board) for prior work on the topic.
|
||||
3. **Published on GitHub?** Skills are conventionally a directory with a
|
||||
`SKILL.md`:
|
||||
```sh
|
||||
gh search code --filename SKILL.md "<topic>" --limit 20
|
||||
gh search repos "<topic> skill" --limit 20
|
||||
```
|
||||
Also check known collections (e.g. `anthropics/skills`) and do a web search
|
||||
for `<topic> agent skill SKILL.md`.
|
||||
|
||||
Judge candidates by: does the SKILL.md actually contain the procedure (not a
|
||||
stub)? Is it maintained? What does it bundle (scripts raise the trust level)?
|
||||
Is the license compatible with redistribution? Then pick a path:
|
||||
|
||||
- **Good external skill exists** → add it as an **external reference**
|
||||
(Step 2A). It stays attributed to and pinned at the upstream repo.
|
||||
- **Partial match** → author a local skill (Step 2B) that adapts the idea;
|
||||
credit the source with a link in the SKILL.md body.
|
||||
- **Nothing usable** → author a new local skill (Step 2B).
|
||||
|
||||
## Step 2 — Choose kind, category, and slug
|
||||
|
||||
- **kind**: default to `optional`. Use `bundled` only when the skill should
|
||||
ship to every Paperclip company by default — that needs explicit human/board
|
||||
direction, not your judgment call.
|
||||
- **category**: reuse an existing directory when one fits (`browser`,
|
||||
`content`, `docs`, `finance`, `paperclip-operations`, `product`, `quality`,
|
||||
`research`, `software-development`). New categories are allowed but must be
|
||||
lowercase kebab-case slugs.
|
||||
- **slug**: lowercase kebab-case (`^[a-z0-9]+(-[a-z0-9]+)*$`), unique across
|
||||
the whole catalog (both kinds).
|
||||
|
||||
The skill lives at
|
||||
`packages/skills-catalog/catalog/<kind>/<category>/<slug>/` and its canonical
|
||||
key is `paperclipai/<kind>/<category>/<slug>`.
|
||||
|
||||
## Step 2A — External reference path (`catalog-ref.json`)
|
||||
|
||||
The directory contains **only** `catalog-ref.json` (a directory with both
|
||||
`catalog-ref.json` and `SKILL.md` fails the build). The manifest builder
|
||||
fetches the pinned files from GitHub at build time and inventories them.
|
||||
|
||||
```sh
|
||||
# Pin the exact commit for the chosen ref (tag or branch)
|
||||
gh api repos/<owner>/<repo>/commits/<ref> --jq .sha
|
||||
```
|
||||
|
||||
```json
|
||||
{
|
||||
"source": {
|
||||
"type": "github",
|
||||
"hostname": "github.com",
|
||||
"owner": "<owner>",
|
||||
"repo": "<repo>",
|
||||
"ref": "<tag-or-branch>",
|
||||
"commit": "<40-char sha from above>",
|
||||
"path": "<dir inside the repo containing SKILL.md, or ''>"
|
||||
},
|
||||
"files": ["SKILL.md", "references/**", "scripts/run.py"],
|
||||
"defaultInstall": false,
|
||||
"recommendedForRoles": ["researcher"],
|
||||
"requires": ["python3"],
|
||||
"tags": ["topic", "keywords"]
|
||||
}
|
||||
```
|
||||
|
||||
Rules the builder enforces:
|
||||
|
||||
- `files` entries are exact relative paths or `dir/**` globs; `SKILL.md` must
|
||||
be included and must have frontmatter with `name` and `description`.
|
||||
- If the upstream frontmatter declares `key`/`slug`, they must match the
|
||||
catalog placement — otherwise pick a matching slug or use the local path.
|
||||
- `commit` must be a full 40-hex SHA; every listed file must be ≤ 1 MiB.
|
||||
- `recommendedForRoles`, `requires`, `tags` live in the JSON (there is no
|
||||
local SKILL.md to carry them).
|
||||
|
||||
See `catalog/optional/research/last30days/catalog-ref.json` for the live
|
||||
example, and `examples/external-reference.md` next to this skill.
|
||||
|
||||
## Step 2B — Author a local catalog skill
|
||||
|
||||
Layout:
|
||||
|
||||
```
|
||||
catalog/<kind>/<category>/<slug>/
|
||||
├── SKILL.md # required entrypoint
|
||||
├── examples/ # 1–2 worked examples (Step 3)
|
||||
├── references/ # optional deep-dive docs
|
||||
├── scripts/ # optional — raises trust level, avoid unless needed
|
||||
└── assets/ # optional templates/images
|
||||
```
|
||||
|
||||
`SKILL.md` frontmatter (all validated by the builder):
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: <slug>
|
||||
description: >
|
||||
40–300 chars. Routing logic, not marketing: what it does, when to use it,
|
||||
when not to.
|
||||
key: paperclipai/<kind>/<category>/<slug>
|
||||
recommendedForRoles:
|
||||
- engineer # non-empty; used for staffing suggestions
|
||||
tags:
|
||||
- topic # non-empty; used for browse/search
|
||||
---
|
||||
```
|
||||
|
||||
Optional frontmatter: `defaultInstall: true` (only for skills every new
|
||||
company should get), `requires: [node, python3, ...]` for runtime deps.
|
||||
|
||||
Body: follow `docs/guides/agent-developer/writing-a-skill.md` — "When to use"
|
||||
/ "When not to use" sections, concrete commands over prose, supporting detail
|
||||
in `references/`. If the skill came from a tweet or external source, link it
|
||||
in the body for attribution.
|
||||
|
||||
Trust level is derived from files, not declared: any `scripts/` file makes the
|
||||
skill `scripts_executables` (install becomes audit-gated and you must extend
|
||||
the `scriptBearing` expectation in `src/shipped-catalog.test.ts`); `assets/`
|
||||
or non-markdown files make it `assets`; markdown-only skills stay
|
||||
`markdown_only`. Prefer markdown-only.
|
||||
|
||||
## Step 3 — Write 1–2 worked examples
|
||||
|
||||
Create `examples/` inside the skill directory with one or two markdown files,
|
||||
each a complete input → application → output walkthrough (realistic input, the
|
||||
skill's steps applied, the finished artifact). These ship with the skill so
|
||||
installers can judge it before running it, and they keep the trust level at
|
||||
`markdown_only` because they are `.md` files.
|
||||
|
||||
Name them by scenario, e.g. `examples/rewrite-release-note.md`.
|
||||
|
||||
## Step 4 — Regenerate the manifest and update tests
|
||||
|
||||
Never hand-edit `generated/catalog.json`; it is deterministic build output.
|
||||
|
||||
```sh
|
||||
pnpm --filter @paperclipai/skills-catalog build:manifest # regenerates generated/catalog.json
|
||||
pnpm --filter @paperclipai/skills-catalog validate # must report no errors
|
||||
```
|
||||
|
||||
(External references need network access to GitHub during these steps.)
|
||||
|
||||
Then update `packages/skills-catalog/src/shipped-catalog.test.ts`:
|
||||
|
||||
- add the new key to `EXPECTED_BUNDLED_KEYS` or `EXPECTED_OPTIONAL_KEYS`
|
||||
(alphabetical order);
|
||||
- if the skill bears scripts, add it to the `scriptBearing` expectation.
|
||||
|
||||
```sh
|
||||
pnpm --filter @paperclipai/skills-catalog test
|
||||
```
|
||||
|
||||
The test suite also enforces the ≤300-char frontmatter description budget
|
||||
across the repo and the ≥40-char description / non-empty roles+tags rules for
|
||||
every catalog skill.
|
||||
|
||||
## Step 5 — Open the PR
|
||||
|
||||
Follow the `prepare-paperclip-pr` skill (`.agents/skills/prepare-paperclip-pr/`)
|
||||
against `paperclipai/paperclip` master. The diff should contain exactly:
|
||||
|
||||
1. the new skill directory (SKILL.md + examples/ + supporting files, **or**
|
||||
catalog-ref.json),
|
||||
2. the regenerated `generated/catalog.json`,
|
||||
3. the `shipped-catalog.test.ts` expectation update.
|
||||
|
||||
In the PR body: link the source material (tweet URL, upstream repo), state
|
||||
whether this is a new skill / adaptation / external reference, and note the
|
||||
trust level. Reference PR #10410 (simplified-english) as the shape of a
|
||||
minimal optional-skill PR.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- `generated/catalog.json` staleness is a validation error — always rerun
|
||||
`build:manifest` after any file change inside the skill directory (the
|
||||
inventory carries per-file sha256 hashes).
|
||||
- Duplicate `slug` across bundled *and* optional fails the build, not just
|
||||
duplicate keys.
|
||||
- Symlinks inside a skill directory must resolve within it; directory
|
||||
symlinks are rejected — copy files in.
|
||||
- The `bundled` kind and `defaultInstall` are independent axes; don't set
|
||||
`defaultInstall: true` casually on optional skills.
|
||||
- For external references the builder fetches from GitHub on every manifest
|
||||
build; a moved/deleted upstream breaks the build, which is why `commit` is
|
||||
pinned — prefer upstream tags for `ref`.
|
||||
@@ -0,0 +1,84 @@
|
||||
# Example — FIND path: tweet → existing skill → external reference
|
||||
|
||||
Real artifact: `packages/skills-catalog/catalog/optional/research/last30days/`.
|
||||
|
||||
## Input
|
||||
|
||||
Dotta sends a tweet praising a "last 30 days" research workflow that sweeps
|
||||
Reddit/X/YouTube for what changed recently on a topic.
|
||||
|
||||
## Step 0 — Capture
|
||||
|
||||
```sh
|
||||
xc get https://x.com/<author>/status/<id> --json # post text + conversation_id
|
||||
xc search 'conversation_id:<id>' --archive --json # the rest of the thread
|
||||
```
|
||||
|
||||
The thread links a GitHub repo: `mvanhorn/last30days-skill`, which already
|
||||
contains a proper skill (`skills/last30days/SKILL.md` plus scripts and
|
||||
references).
|
||||
|
||||
## Step 1 — FIND
|
||||
|
||||
- Not in the catalog, not in this repo.
|
||||
- The upstream repo IS the skill — maintained, tagged releases, real SKILL.md.
|
||||
- Verdict: **FIND** — add it as an external reference, keep attribution and
|
||||
updates upstream.
|
||||
|
||||
## Step 2A — catalog-ref.json
|
||||
|
||||
Placement: `optional` / `research` / `last30days`. Pin the release tag to an
|
||||
exact commit:
|
||||
|
||||
```sh
|
||||
gh api repos/mvanhorn/last30days-skill/commits/v3.3.0 --jq .sha
|
||||
# → daca71f89eb71d0d56d01a43ed7627aa919dba4f
|
||||
```
|
||||
|
||||
`catalog/optional/research/last30days/catalog-ref.json` (the only file in the
|
||||
directory):
|
||||
|
||||
```json
|
||||
{
|
||||
"source": {
|
||||
"type": "github",
|
||||
"hostname": "github.com",
|
||||
"owner": "mvanhorn",
|
||||
"repo": "last30days-skill",
|
||||
"ref": "v3.3.0",
|
||||
"commit": "daca71f89eb71d0d56d01a43ed7627aa919dba4f",
|
||||
"path": "skills/last30days"
|
||||
},
|
||||
"files": [
|
||||
"SKILL.md",
|
||||
"agents/openai.yaml",
|
||||
"references/**",
|
||||
"scripts/briefing.py",
|
||||
"scripts/compare.sh",
|
||||
"scripts/last30days.py",
|
||||
"scripts/lib/**",
|
||||
"scripts/setup-keychain.sh",
|
||||
"scripts/store.py",
|
||||
"scripts/watchlist.py"
|
||||
],
|
||||
"defaultInstall": false,
|
||||
"recommendedForRoles": ["researcher", "marketer", "product-manager", "analyst"],
|
||||
"requires": ["node", "python3"],
|
||||
"tags": ["research", "last-30-days", "social-media", "trends", "citations", "reddit", "x", "youtube"]
|
||||
}
|
||||
```
|
||||
|
||||
Metadata (`recommendedForRoles`, `requires`, `tags`) lives in the JSON because
|
||||
there is no local SKILL.md to carry it.
|
||||
|
||||
## Steps 4–5 — Manifest, tests, PR
|
||||
|
||||
- `pnpm --filter @paperclipai/skills-catalog build:manifest` fetches the
|
||||
pinned files from GitHub and inventories them (network required).
|
||||
- The skill bundles `scripts/`, so trust level derives to
|
||||
`scripts_executables` → it must also be added to the `scriptBearing`
|
||||
expectation in `src/shipped-catalog.test.ts`, alongside
|
||||
`EXPECTED_OPTIONAL_KEYS`.
|
||||
- PR diff: `catalog-ref.json`, regenerated `generated/catalog.json`, test
|
||||
expectations. PR body links both the tweet and the upstream repo, and calls
|
||||
out the elevated trust level so review is deliberate.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Example — MAKE path: idea → new optional catalog skill
|
||||
|
||||
Real run: PAP-15684 → PR #10410 (`feat(skills-catalog): add optional
|
||||
/simplified-english skill`).
|
||||
|
||||
## Input
|
||||
|
||||
Task from Dotta: "make a skill that has agents write user-facing text in
|
||||
Simplified English." No tweet this time — the source is a known public
|
||||
specification (ASD-STE100 Simplified Technical English).
|
||||
|
||||
## Step 1 — FIND
|
||||
|
||||
- `grep -i 'simplified\|plain.english' packages/skills-catalog/generated/catalog.json`
|
||||
→ no hits; nothing in `.agents/skills/` or `skills/` either.
|
||||
- `gh search code --filename SKILL.md "simplified technical english"` → no
|
||||
usable published skill (only STE checker tools, no SKILL.md procedure).
|
||||
- Verdict: **MAKE** a new local skill.
|
||||
|
||||
## Step 2 — Placement
|
||||
|
||||
- kind: `optional` (useful, but not something every company must ship with).
|
||||
- category: `content` (existing category, fits writing/communication).
|
||||
- slug: `simplified-english`.
|
||||
- Path: `packages/skills-catalog/catalog/optional/content/simplified-english/`.
|
||||
|
||||
## Step 2B — Authoring
|
||||
|
||||
One markdown-only `SKILL.md` (trust level stays `markdown_only`):
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: simplified-english
|
||||
description: Write user-facing comments, plans, and documents in ASD-STE100 Simplified Technical English — short, unambiguous sentences with approved words and one meaning each — so readers understand them the first time.
|
||||
key: paperclipai/optional/content/simplified-english
|
||||
recommendedForRoles:
|
||||
- engineer
|
||||
- product
|
||||
- writer
|
||||
- devrel
|
||||
tags:
|
||||
- writing
|
||||
- communication
|
||||
- clarity
|
||||
- style
|
||||
---
|
||||
|
||||
# Simplified English
|
||||
|
||||
For user-facing comments, plans, and documents, write using only ASD-STE100
|
||||
Simplified Technical English.
|
||||
|
||||
## Core rules
|
||||
|
||||
- Use short sentences (procedures ≤ 20 words, descriptions ≤ 25 words).
|
||||
- Give one instruction per sentence.
|
||||
- Use approved words with one meaning each; avoid synonyms and jargon.
|
||||
...
|
||||
```
|
||||
|
||||
Note the frontmatter hits every builder rule: description is 40–300 chars,
|
||||
`key` matches the placement, roles and tags are non-empty.
|
||||
|
||||
## Steps 3–4 — Examples, manifest, tests
|
||||
|
||||
- Add `examples/rewrite-status-comment.md` showing a jargon-heavy status
|
||||
comment rewritten under the rules (before/after).
|
||||
- `pnpm --filter @paperclipai/skills-catalog build:manifest` → regenerates
|
||||
`generated/catalog.json` with the new entry.
|
||||
- Add `"paperclipai/optional/content/simplified-english"` to
|
||||
`EXPECTED_OPTIONAL_KEYS` in `src/shipped-catalog.test.ts` (alphabetical).
|
||||
- `pnpm --filter @paperclipai/skills-catalog test` → green.
|
||||
|
||||
## Step 5 — PR
|
||||
|
||||
Four-part diff:
|
||||
|
||||
```
|
||||
packages/skills-catalog/catalog/optional/content/simplified-english/SKILL.md
|
||||
packages/skills-catalog/catalog/optional/content/simplified-english/examples/rewrite-status-comment.md
|
||||
packages/skills-catalog/generated/catalog.json
|
||||
packages/skills-catalog/src/shipped-catalog.test.ts
|
||||
```
|
||||
|
||||
(The historical PR #10410 predates this skill and shipped as a three-part
|
||||
diff without the `examples/` file; a run that follows this skill includes the
|
||||
worked example from Step 3 in the same PR.)
|
||||
|
||||
PR body links the ASD-STE100 spec as the source and states trust level
|
||||
`markdown_only`.
|
||||
@@ -0,0 +1,98 @@
|
||||
---
|
||||
name: garden-inbox
|
||||
description: Scan a Paperclip user's Mine inbox, classify reversible archive candidates, request checkbox confirmation, and archive only accepted selections. Use when asked to garden, clean up, prune, or tidy a Paperclip inbox without changing issues, branches, or workspaces.
|
||||
---
|
||||
|
||||
# Garden Inbox
|
||||
|
||||
Use the bundled script for every stage. Keep the workflow strictly ordered: `scan` → `confirm` → `apply`.
|
||||
|
||||
## Safety contract
|
||||
|
||||
- Treat `scan` as read-only. It may read inbox, workspace, close-readiness, and local Git metadata only.
|
||||
- Treat `confirm` as a confirmation-card write only. It must not archive inbox entries or mutate issue fields, branches, or workspaces.
|
||||
- Run `apply` only after a resolved `request_checkbox_confirmation` interaction. It archives only accepted option IDs that also occur in the originating `candidates.json`.
|
||||
- Remember that inbox archive state is per-user presentation state. It is reversible and does not change the underlying issue.
|
||||
- Never substitute issue status changes, branch deletion, workspace cleanup, or issue deletion for inbox archiving.
|
||||
|
||||
## Inputs
|
||||
|
||||
Require `PAPERCLIP_API_URL` and `PAPERCLIP_API_KEY`. The script strips a trailing `/api` from the URL. It resolves the target user from the run JWT payload's `responsible_user_id`; use `--user-id <uuid>` only for an explicit target override.
|
||||
|
||||
Use a run-owned output directory when available:
|
||||
|
||||
```bash
|
||||
RUN_DIR="${PAPERCLIP_RUN_SCRATCH_DIR:-${PAPERCLIP_TASK_SCRATCH_DIR:-.}}/garden-inbox"
|
||||
mkdir -p "$RUN_DIR"
|
||||
```
|
||||
|
||||
## 1. Scan
|
||||
|
||||
```bash
|
||||
node .agents/skills/garden-inbox/scripts/garden-inbox.mjs scan \
|
||||
--output-dir "$RUN_DIR" \
|
||||
--stale-days 60
|
||||
```
|
||||
|
||||
Inspect `garden-inbox-report.md` and `candidates.json`. The report groups every inbox row into exactly one bucket:
|
||||
|
||||
The scan reads Mine separately for each included issue status so the endpoint's global 500-row cap does not silently omit older rows. If any single-status query reaches that cap, both outputs mark coverage as possibly truncated; do not treat that scan as complete.
|
||||
|
||||
- A: merged/archived workspace and all linked work terminal; selected by default.
|
||||
- B: terminal or workspace-gone work idle beyond the threshold; selected by default.
|
||||
- C: stale work with commits ahead of base; never selected by default.
|
||||
- D: keep; never offered for archiving.
|
||||
|
||||
Do not manually promote bucket D entries into the candidate file.
|
||||
|
||||
## 2. Confirm
|
||||
|
||||
Post checkbox interactions on the driving issue:
|
||||
|
||||
```bash
|
||||
node .agents/skills/garden-inbox/scripts/garden-inbox.mjs confirm \
|
||||
--issue-id "$PAPERCLIP_TASK_ID" \
|
||||
--candidates "$RUN_DIR/candidates.json"
|
||||
```
|
||||
|
||||
The script posts sequential cards when a scan has more than 200 candidates. Re-running `confirm` with the same scan file is idempotent. Leave the driving issue in the waiting posture required by the surrounding Paperclip heartbeat workflow.
|
||||
|
||||
For development or payload review, suppress the POST:
|
||||
|
||||
```bash
|
||||
node .agents/skills/garden-inbox/scripts/garden-inbox.mjs confirm \
|
||||
--issue-id "$PAPERCLIP_TASK_ID" \
|
||||
--candidates "$RUN_DIR/candidates.json" \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
## 3. Apply accepted selections
|
||||
|
||||
After an interaction-resolution wake, take the resolved interaction ID from the wake payload and run:
|
||||
|
||||
```bash
|
||||
node .agents/skills/garden-inbox/scripts/garden-inbox.mjs apply \
|
||||
--issue-id "$PAPERCLIP_TASK_ID" \
|
||||
--interaction-id "$INTERACTION_ID" \
|
||||
--candidates "$RUN_DIR/candidates.json"
|
||||
```
|
||||
|
||||
On rejection, expiry, or an accepted empty selection, the script archives nothing. Apply preserves the scan file's target user, including an explicit `--user-id` override. Its summary includes the API undo path and target-user body for every archived row.
|
||||
|
||||
Test `apply` without API writes by supplying a saved interaction response:
|
||||
|
||||
```bash
|
||||
node .agents/skills/garden-inbox/scripts/garden-inbox.mjs apply \
|
||||
--issue-id "$PAPERCLIP_TASK_ID" \
|
||||
--interaction-file resolved-interaction.json \
|
||||
--candidates "$RUN_DIR/candidates.json" \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
## Verify the bundled logic
|
||||
|
||||
Run the zero-dependency Node tests after changing classification or selection safety:
|
||||
|
||||
```bash
|
||||
node --test .agents/skills/garden-inbox/scripts/garden-inbox.test.mjs
|
||||
```
|
||||
+699
@@ -0,0 +1,699 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
|
||||
const TERMINAL_STATUSES = new Set(["done", "cancelled"]);
|
||||
const OFFERED_BUCKETS = new Set(["A", "B", "C"]);
|
||||
const DEFAULT_STALE_DAYS = 60;
|
||||
const INTERACTION_LIMIT = 200;
|
||||
const INBOX_MINE_STATUSES = ["backlog", "todo", "in_progress", "in_review", "blocked", "done"];
|
||||
const INBOX_QUERY_CAP = 500;
|
||||
|
||||
class ApiError extends Error {
|
||||
constructor(status, message, body = null) {
|
||||
super(`${status}: ${message}`);
|
||||
this.name = "ApiError";
|
||||
this.status = status;
|
||||
this.body = body;
|
||||
}
|
||||
}
|
||||
|
||||
function usage() {
|
||||
return `Usage:
|
||||
garden-inbox.mjs [scan] [--user-id UUID] [--stale-days 60] [--output-dir DIR]
|
||||
garden-inbox.mjs confirm [--issue-id ID] [--candidates FILE] [--dry-run]
|
||||
garden-inbox.mjs apply [--issue-id ID] (--interaction-id ID | --interaction-file FILE) [--candidates FILE] [--dry-run]
|
||||
|
||||
Common environment:
|
||||
PAPERCLIP_API_URL, PAPERCLIP_API_KEY, PAPERCLIP_RUN_ID, PAPERCLIP_TASK_ID`;
|
||||
}
|
||||
|
||||
function parseArgs(argv) {
|
||||
const args = [...argv];
|
||||
let command = "scan";
|
||||
if (args[0] && !args[0].startsWith("-")) command = args.shift();
|
||||
if (!["scan", "confirm", "apply"].includes(command)) throw new Error(`Unknown command: ${command}`);
|
||||
|
||||
const options = {};
|
||||
while (args.length > 0) {
|
||||
const token = args.shift();
|
||||
if (token === "--help" || token === "-h") {
|
||||
options.help = true;
|
||||
continue;
|
||||
}
|
||||
if (!token.startsWith("--")) throw new Error(`Unexpected argument: ${token}`);
|
||||
const key = token.slice(2).replaceAll("-", "_");
|
||||
if (key === "dry_run") {
|
||||
options[key] = true;
|
||||
continue;
|
||||
}
|
||||
const value = args.shift();
|
||||
if (value === undefined || value.startsWith("--")) throw new Error(`${token} requires a value`);
|
||||
if (options[key] === undefined) options[key] = value;
|
||||
else options[key] = Array.isArray(options[key]) ? [...options[key], value] : [options[key], value];
|
||||
}
|
||||
return { command, options };
|
||||
}
|
||||
|
||||
function required(value, name) {
|
||||
if (typeof value !== "string" || value.trim() === "") throw new Error(`${name} is required`);
|
||||
return value.trim();
|
||||
}
|
||||
|
||||
function normalizeApiBase(value) {
|
||||
let base = required(value, "PAPERCLIP_API_URL").replace(/\/+$/, "");
|
||||
if (base.endsWith("/api")) base = base.slice(0, -4);
|
||||
return base;
|
||||
}
|
||||
|
||||
function runtime(options, { requireApi = true } = {}) {
|
||||
const apiUrl = options.api_url ?? process.env.PAPERCLIP_API_URL;
|
||||
const apiKey = options.api_key ?? process.env.PAPERCLIP_API_KEY;
|
||||
return {
|
||||
apiBase: requireApi ? normalizeApiBase(apiUrl) : (apiUrl ? normalizeApiBase(apiUrl) : null),
|
||||
apiKey: requireApi ? required(apiKey, "PAPERCLIP_API_KEY") : apiKey ?? null,
|
||||
runId: options.run_id ?? process.env.PAPERCLIP_RUN_ID ?? null,
|
||||
};
|
||||
}
|
||||
|
||||
async function apiRequest(context, path, { method = "GET", body } = {}) {
|
||||
const headers = { Accept: "application/json", Authorization: `Bearer ${context.apiKey}` };
|
||||
if (body !== undefined) headers["Content-Type"] = "application/json";
|
||||
if (method !== "GET" && context.runId) headers["X-Paperclip-Run-Id"] = context.runId;
|
||||
const response = await fetch(`${context.apiBase}/api${path}`, {
|
||||
method,
|
||||
headers,
|
||||
body: body === undefined ? undefined : JSON.stringify(body),
|
||||
});
|
||||
const text = await response.text();
|
||||
let parsed = null;
|
||||
if (text) {
|
||||
try { parsed = JSON.parse(text); } catch { parsed = text; }
|
||||
}
|
||||
if (!response.ok) {
|
||||
const message = parsed && typeof parsed === "object"
|
||||
? parsed.error ?? parsed.message ?? response.statusText
|
||||
: parsed ?? response.statusText;
|
||||
throw new ApiError(response.status, String(message), parsed);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function decodeJwtPayload(token) {
|
||||
const parts = required(token, "PAPERCLIP_API_KEY").split(".");
|
||||
if (parts.length < 2) throw new Error("PAPERCLIP_API_KEY is not a JWT; pass --user-id explicitly");
|
||||
try {
|
||||
return JSON.parse(Buffer.from(parts[1], "base64url").toString("utf8"));
|
||||
} catch {
|
||||
throw new Error("Could not decode the PAPERCLIP_API_KEY JWT payload; pass --user-id explicitly");
|
||||
}
|
||||
}
|
||||
|
||||
function resolveUserId(options, apiKey) {
|
||||
if (options.user_id) return required(options.user_id, "--user-id");
|
||||
const userId = decodeJwtPayload(apiKey).responsible_user_id;
|
||||
if (typeof userId !== "string" || userId.trim() === "") {
|
||||
throw new Error("Run JWT has no responsible_user_id; pass --user-id explicitly");
|
||||
}
|
||||
return userId.trim();
|
||||
}
|
||||
|
||||
function parsePositiveInteger(value, fallback, name) {
|
||||
if (value === undefined) return fallback;
|
||||
const parsed = Number(value);
|
||||
if (!Number.isInteger(parsed) || parsed < 1 || parsed > 3650) {
|
||||
throw new Error(`${name} must be an integer from 1 to 3650`);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function validDate(value) {
|
||||
if (!value) return null;
|
||||
const date = new Date(value);
|
||||
return Number.isNaN(date.getTime()) ? null : date;
|
||||
}
|
||||
|
||||
function latestDate(...values) {
|
||||
return values.flat().map(validDate).filter(Boolean).sort((left, right) => right.getTime() - left.getTime())[0] ?? null;
|
||||
}
|
||||
|
||||
function runGit(args, cwd) {
|
||||
try {
|
||||
return execFileSync("git", ["-C", cwd, ...args], {
|
||||
encoding: "utf8",
|
||||
stdio: ["ignore", "pipe", "ignore"],
|
||||
timeout: 5_000,
|
||||
}).trim();
|
||||
} catch {
|
||||
return "";
|
||||
}
|
||||
}
|
||||
|
||||
function gitLastCommitAt(branchName, roots) {
|
||||
if (!branchName) return null;
|
||||
for (const root of [...new Set(roots.filter(Boolean))]) {
|
||||
const parsed = validDate(runGit(["log", "-1", "--format=%cI", branchName], root));
|
||||
if (parsed) return parsed;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function findLocalBranch(identifier, roots) {
|
||||
if (!identifier) return null;
|
||||
const needle = identifier.toLowerCase();
|
||||
for (const root of [...new Set(roots.filter(Boolean))]) {
|
||||
const branches = runGit(["for-each-ref", "--format=%(refname:short)", "refs/heads"], root)
|
||||
.split("\n").map((branch) => branch.trim()).filter(Boolean)
|
||||
.filter((branch) => branch.toLowerCase().includes(needle));
|
||||
if (branches.length === 1) return branches[0];
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
async function mapLimit(values, limit, mapper) {
|
||||
const results = new Array(values.length);
|
||||
let cursor = 0;
|
||||
async function worker() {
|
||||
while (cursor < values.length) {
|
||||
const index = cursor++;
|
||||
results[index] = await mapper(values[index], index);
|
||||
}
|
||||
}
|
||||
await Promise.all(Array.from({ length: Math.min(limit, values.length) }, worker));
|
||||
return results;
|
||||
}
|
||||
|
||||
async function fetchMineInboxRows(context, userId) {
|
||||
const statusResults = await Promise.all(INBOX_MINE_STATUSES.map(async (status) => {
|
||||
const query = new URLSearchParams({ userId, status });
|
||||
const rows = await apiRequest(context, `/agents/me/inbox/mine?${query}`);
|
||||
if (!Array.isArray(rows)) throw new Error(`Inbox API returned a non-array response for status ${status}`);
|
||||
return { status, rows };
|
||||
}));
|
||||
const rowsById = new Map();
|
||||
const statusCounts = {};
|
||||
const cappedStatuses = [];
|
||||
let duplicateCount = 0;
|
||||
for (const { status, rows } of statusResults) {
|
||||
statusCounts[status] = rows.length;
|
||||
if (rows.length >= INBOX_QUERY_CAP) cappedStatuses.push(status);
|
||||
for (const issue of rows) {
|
||||
if (rowsById.has(issue.id)) duplicateCount += 1;
|
||||
rowsById.set(issue.id, issue);
|
||||
}
|
||||
}
|
||||
return {
|
||||
rows: [...rowsById.values()],
|
||||
coverage: {
|
||||
strategy: "per_status",
|
||||
queryCap: INBOX_QUERY_CAP,
|
||||
statusCounts,
|
||||
cappedStatuses,
|
||||
duplicateCount,
|
||||
complete: cappedStatuses.length === 0,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function inspectWorkspace(context, workspaceId, issueIdentifiers, gitRoot) {
|
||||
let workspace = null;
|
||||
let readiness = null;
|
||||
let gone = false;
|
||||
let error = null;
|
||||
try {
|
||||
workspace = await apiRequest(context, `/execution-workspaces/${encodeURIComponent(workspaceId)}`);
|
||||
} catch (caught) {
|
||||
if (caught instanceof ApiError && caught.status === 404) gone = true;
|
||||
else error = caught.message;
|
||||
}
|
||||
if (workspace) {
|
||||
try {
|
||||
readiness = await apiRequest(context, `/execution-workspaces/${encodeURIComponent(workspaceId)}/close-readiness`);
|
||||
} catch (caught) {
|
||||
error = error ?? caught.message;
|
||||
}
|
||||
}
|
||||
|
||||
const roots = [workspace?.cwd, workspace?.providerRef, readiness?.git?.repoRoot, readiness?.git?.workspacePath, gitRoot];
|
||||
let branchName = workspace?.branchName ?? readiness?.git?.branchName ?? null;
|
||||
if (!branchName && gone) {
|
||||
const matches = issueIdentifiers.map((identifier) => findLocalBranch(identifier, roots)).filter(Boolean);
|
||||
if (new Set(matches).size === 1) branchName = matches[0];
|
||||
}
|
||||
const branchCommitAt = gitLastCommitAt(branchName, roots);
|
||||
return {
|
||||
id: workspaceId,
|
||||
gone,
|
||||
error,
|
||||
status: workspace?.status ?? null,
|
||||
branchName,
|
||||
lastUsedAt: workspace?.lastUsedAt ?? null,
|
||||
branchCommitAt: branchCommitAt?.toISOString() ?? null,
|
||||
readiness,
|
||||
};
|
||||
}
|
||||
|
||||
function reason(code, message, facts = {}) {
|
||||
return { code, message, facts };
|
||||
}
|
||||
|
||||
function classify(issue, workspaceInfo, staleDays, now) {
|
||||
const terminal = TERMINAL_STATUSES.has(issue.status);
|
||||
const workspaceArchived = workspaceInfo?.status === "archived";
|
||||
const workspaceGone = Boolean(workspaceInfo?.gone);
|
||||
const readiness = workspaceInfo?.readiness ?? null;
|
||||
const linkedIssues = Array.isArray(readiness?.linkedIssues) ? readiness.linkedIssues : [];
|
||||
const allLinkedIssuesTerminal = Boolean(readiness) && linkedIssues.every((linked) => linked.isTerminal === true);
|
||||
const mergedIntoBase = readiness?.git?.isMergedIntoBase === true;
|
||||
const aheadCount = Number.isInteger(readiness?.git?.aheadCount) ? readiness.git.aheadCount : null;
|
||||
const blockerAttention = issue.blockerAttention?.state;
|
||||
const hasOpenBlockers = issue.status === "blocked"
|
||||
|| (Array.isArray(issue.blockedBy) && issue.blockedBy.some((blocker) => !TERMINAL_STATUSES.has(blocker.status)))
|
||||
|| (typeof blockerAttention === "string" && blockerAttention !== "none");
|
||||
const awaitingUser = issue.blockedInboxAttention?.state === "awaiting_decision"
|
||||
|| (issue.status === "in_review" && Boolean(issue.assigneeUserId));
|
||||
const issueActivity = latestDate(issue.lastActivityAt, issue.updatedAt, issue.createdAt);
|
||||
const branchActivity = latestDate(workspaceInfo?.branchCommitAt, workspaceInfo?.lastUsedAt);
|
||||
const lastActivity = latestDate(issueActivity, branchActivity) ?? new Date(0);
|
||||
const cutoff = new Date(now.getTime() - staleDays * 86_400_000);
|
||||
const stale = lastActivity.getTime() <= cutoff.getTime();
|
||||
const facts = {
|
||||
issueStatus: issue.status,
|
||||
workspaceStatus: workspaceInfo?.status ?? null,
|
||||
workspaceGone,
|
||||
workspaceInspectionError: workspaceInfo?.error ?? null,
|
||||
mergedIntoBase,
|
||||
allLinkedIssuesTerminal,
|
||||
aheadCount,
|
||||
staleDays,
|
||||
};
|
||||
|
||||
if (workspaceInfo?.error) {
|
||||
return {
|
||||
bucket: "D",
|
||||
reason: reason("workspace_inspection_failed", "Keep: workspace archive safety could not be verified.", facts),
|
||||
lastActivity,
|
||||
};
|
||||
}
|
||||
if (hasOpenBlockers) return { bucket: "D", reason: reason("open_blockers", "Keep: the issue has unresolved blocker or liveness attention.", facts), lastActivity };
|
||||
if (awaitingUser) return { bucket: "D", reason: reason("pending_user_action", "Keep: the issue is awaiting a user decision or review.", facts), lastActivity };
|
||||
if (!terminal) {
|
||||
const code = issue.status === "in_review" ? "pending_interaction_or_review" : "non_terminal_status";
|
||||
return { bucket: "D", reason: reason(code, `Keep: issue status ${issue.status} is not terminal.`, facts), lastActivity };
|
||||
}
|
||||
if ((mergedIntoBase || workspaceArchived) && allLinkedIssuesTerminal) {
|
||||
const code = workspaceArchived ? "workspace_archived_finished" : "merged_finished";
|
||||
return { bucket: "A", reason: reason(code, "Archive candidate: work is merged or archived and every linked issue is terminal.", facts), lastActivity };
|
||||
}
|
||||
if (!stale) return { bucket: "D", reason: reason("recent_activity", `Keep: activity is newer than ${staleDays} days.`, facts), lastActivity };
|
||||
if (aheadCount !== null && aheadCount > 0) {
|
||||
return { bucket: "C", reason: reason("stale_unmerged_commits", `Review manually: stale branch is ${aheadCount} commit(s) ahead of base.`, facts), lastActivity };
|
||||
}
|
||||
if (terminal || workspaceGone) {
|
||||
return { bucket: "B", reason: reason("stale_terminal_or_workspace_gone", `Archive candidate: no issue or branch activity for at least ${staleDays} days and the issue is terminal or workspace is gone.`, facts), lastActivity };
|
||||
}
|
||||
return { bucket: "D", reason: reason("insufficient_archive_evidence", "Keep: archive safety conditions were not met.", facts), lastActivity };
|
||||
}
|
||||
|
||||
function candidateFrom(issue, workspace, classification) {
|
||||
return {
|
||||
issueId: issue.id,
|
||||
identifier: issue.identifier ?? null,
|
||||
title: issue.title,
|
||||
bucket: classification.bucket,
|
||||
reason: classification.reason,
|
||||
executionWorkspaceId: issue.executionWorkspaceId ?? null,
|
||||
workspaceStatus: workspace?.status ?? null,
|
||||
workspaceGone: Boolean(workspace?.gone),
|
||||
branchName: workspace?.branchName ?? null,
|
||||
lastActivityAt: classification.lastActivity.toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
function markdownEscape(value) {
|
||||
return String(value ?? "").replaceAll("|", "\\|").replaceAll("\n", " ");
|
||||
}
|
||||
|
||||
function renderReport(scanResult) {
|
||||
const labels = {
|
||||
A: "A — merged and finished (default checked)",
|
||||
B: "B — stale (default checked)",
|
||||
C: "C — stale but unmerged (default unchecked)",
|
||||
D: "D — keep (not offered)",
|
||||
};
|
||||
const lines = [
|
||||
"# Garden inbox scan", "",
|
||||
`- Generated: ${scanResult.generatedAt}`,
|
||||
`- Target user: \`${scanResult.userId}\``,
|
||||
`- Stale threshold: ${scanResult.staleDays} days`,
|
||||
`- Inbox rows classified: ${scanResult.items.length}`,
|
||||
`- Archive candidates: ${scanResult.candidates.length}`,
|
||||
`- Scan coverage: ${scanResult.coverage.complete ? "complete across per-status queries" : "possibly truncated"}`,
|
||||
];
|
||||
if (!scanResult.coverage.complete) {
|
||||
lines.push(
|
||||
"",
|
||||
"> [!WARNING]",
|
||||
`> Coverage may be incomplete: ${scanResult.coverage.cappedStatuses.map((status) => `\`${status}\``).join(", ")} returned at least ${scanResult.coverage.queryCap} rows, the Mine endpoint cap.`,
|
||||
);
|
||||
}
|
||||
for (const bucket of ["A", "B", "C", "D"]) {
|
||||
const items = scanResult.items.filter((item) => item.bucket === bucket);
|
||||
lines.push("", `## ${labels[bucket]} (${items.length})`, "");
|
||||
if (items.length === 0) {
|
||||
lines.push("_None._");
|
||||
continue;
|
||||
}
|
||||
lines.push("| Issue | Title | Reason | Workspace / branch | Last activity |", "|---|---|---|---|---|");
|
||||
for (const item of items) {
|
||||
const issue = item.identifier ? `\`${markdownEscape(item.identifier)}\`` : `\`${item.issueId}\``;
|
||||
const workspace = [item.workspaceStatus, item.branchName].filter(Boolean).join(" / ") || (item.workspaceGone ? "gone" : "none");
|
||||
lines.push(`| ${issue} | ${markdownEscape(item.title)} | \`${item.reason.code}\` — ${markdownEscape(item.reason.message)} | ${markdownEscape(workspace)} | ${item.lastActivityAt} |`);
|
||||
}
|
||||
}
|
||||
lines.push("");
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
function writeText(path, contents) {
|
||||
const target = resolve(path);
|
||||
mkdirSync(dirname(target), { recursive: true });
|
||||
writeFileSync(target, contents);
|
||||
return target;
|
||||
}
|
||||
|
||||
function readJson(path) {
|
||||
return JSON.parse(readFileSync(resolve(path), "utf8"));
|
||||
}
|
||||
|
||||
function stableScanId(scanResult) {
|
||||
const fingerprint = {
|
||||
userId: scanResult.userId,
|
||||
staleDays: scanResult.staleDays,
|
||||
generatedAt: scanResult.generatedAt,
|
||||
candidates: scanResult.candidates.map((item) => ({
|
||||
issueId: item.issueId,
|
||||
bucket: item.bucket,
|
||||
reason: item.reason.code,
|
||||
lastActivityAt: item.lastActivityAt,
|
||||
})),
|
||||
};
|
||||
return createHash("sha256").update(JSON.stringify(fingerprint)).digest("hex").slice(0, 24);
|
||||
}
|
||||
|
||||
async function scan(options) {
|
||||
const context = runtime(options);
|
||||
const userId = resolveUserId(options, context.apiKey);
|
||||
const staleDays = parsePositiveInteger(options.stale_days, DEFAULT_STALE_DAYS, "--stale-days");
|
||||
const now = options.now ? validDate(options.now) : new Date();
|
||||
if (!now) throw new Error("--now must be an ISO date");
|
||||
const outputDir = options.output_dir ?? ".";
|
||||
const candidatesPath = options.candidates ?? resolve(outputDir, "candidates.json");
|
||||
const reportPath = options.report ?? resolve(outputDir, "garden-inbox-report.md");
|
||||
|
||||
const { rows, coverage } = await fetchMineInboxRows(context, userId);
|
||||
const issuesByWorkspace = new Map();
|
||||
for (const issue of rows) {
|
||||
if (!issue.executionWorkspaceId) continue;
|
||||
const identifiers = issuesByWorkspace.get(issue.executionWorkspaceId) ?? [];
|
||||
if (issue.identifier) identifiers.push(issue.identifier);
|
||||
issuesByWorkspace.set(issue.executionWorkspaceId, identifiers);
|
||||
}
|
||||
const workspaceIds = [...issuesByWorkspace.keys()];
|
||||
const inspected = await mapLimit(workspaceIds, 8, (workspaceId) => inspectWorkspace(
|
||||
context, workspaceId, issuesByWorkspace.get(workspaceId), options.git_root ?? process.cwd(),
|
||||
));
|
||||
const workspaceById = new Map(inspected.map((entry) => [entry.id, entry]));
|
||||
const items = rows.map((issue) => {
|
||||
const workspace = issue.executionWorkspaceId ? workspaceById.get(issue.executionWorkspaceId) ?? null : null;
|
||||
return candidateFrom(issue, workspace, classify(issue, workspace, staleDays, now));
|
||||
});
|
||||
const scanResult = {
|
||||
schemaVersion: 1,
|
||||
generatedAt: now.toISOString(),
|
||||
userId,
|
||||
staleDays,
|
||||
sourceCount: rows.length,
|
||||
coverage,
|
||||
items,
|
||||
candidates: items.filter((item) => OFFERED_BUCKETS.has(item.bucket)),
|
||||
};
|
||||
scanResult.scanId = stableScanId(scanResult);
|
||||
const output = {
|
||||
schemaVersion: 1,
|
||||
scanId: scanResult.scanId,
|
||||
generatedAt: scanResult.generatedAt,
|
||||
userId,
|
||||
staleDays,
|
||||
sourceCount: rows.length,
|
||||
coverage,
|
||||
candidates: scanResult.candidates,
|
||||
kept: items.filter((item) => item.bucket === "D"),
|
||||
};
|
||||
const candidatesFile = writeText(candidatesPath, `${JSON.stringify(output, null, 2)}\n`);
|
||||
const report = renderReport(scanResult);
|
||||
const reportFile = writeText(reportPath, report);
|
||||
process.stdout.write(`${report}\nCandidates JSON: ${candidatesFile}\nReport: ${reportFile}\n`);
|
||||
return output;
|
||||
}
|
||||
|
||||
function truncate(value, limit) {
|
||||
const text = String(value ?? "").trim();
|
||||
if (text.length <= limit) return text;
|
||||
return `${text.slice(0, Math.max(0, limit - 1)).trimEnd()}…`;
|
||||
}
|
||||
|
||||
function candidateFile(options) {
|
||||
const data = readJson(options.candidates ?? "candidates.json");
|
||||
if (data?.schemaVersion !== 1
|
||||
|| typeof data.scanId !== "string"
|
||||
|| typeof data.userId !== "string"
|
||||
|| data.userId.trim() === ""
|
||||
|| !Array.isArray(data.candidates)) {
|
||||
throw new Error("Candidates file is not a garden-inbox schemaVersion 1 scan");
|
||||
}
|
||||
for (const item of data.candidates) {
|
||||
if (!item?.issueId || !OFFERED_BUCKETS.has(item.bucket)) {
|
||||
throw new Error("Candidates file contains an invalid or non-offered item");
|
||||
}
|
||||
}
|
||||
return data;
|
||||
}
|
||||
|
||||
function archiveTargetBody(scanData) {
|
||||
return { userId: scanData.userId };
|
||||
}
|
||||
|
||||
function chunk(values, size) {
|
||||
const chunks = [];
|
||||
for (let index = 0; index < values.length; index += size) chunks.push(values.slice(index, index + size));
|
||||
return chunks;
|
||||
}
|
||||
|
||||
function confirmationBody(scanData, candidates, index, count) {
|
||||
const options = candidates.map((candidate) => ({
|
||||
id: candidate.issueId,
|
||||
label: truncate(`${candidate.identifier ?? candidate.issueId} — ${candidate.title}`, 120),
|
||||
description: truncate(`${candidate.reason.message} Last activity: ${candidate.lastActivityAt}.`, 500),
|
||||
}));
|
||||
const part = count > 1 ? ` (${index + 1}/${count})` : "";
|
||||
return {
|
||||
kind: "request_checkbox_confirmation",
|
||||
idempotencyKey: `garden-inbox:${scanData.scanId}:${index + 1}:${count}`,
|
||||
title: `Confirm inbox archive candidates${part}`,
|
||||
summary: `Choose which reversible inbox entries to archive${part}.`,
|
||||
continuationPolicy: "wake_assignee",
|
||||
payload: {
|
||||
version: 1,
|
||||
prompt: `Select the inbox entries to archive${part}. Unchecked entries will remain visible.`,
|
||||
options,
|
||||
defaultSelectedOptionIds: candidates
|
||||
.filter((candidate) => candidate.bucket === "A" || candidate.bucket === "B")
|
||||
.map((candidate) => candidate.issueId),
|
||||
minSelected: 0,
|
||||
acceptLabel: "Archive selected",
|
||||
rejectLabel: "Keep everything",
|
||||
detailsMarkdown: `Scan \`${scanData.scanId}\` used a ${scanData.staleDays}-day stale threshold. Buckets A and B start checked; stale branches with unmerged commits (C) start unchecked. Archiving changes only this user's inbox visibility and is reversible.`,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function confirm(options) {
|
||||
const data = candidateFile(options);
|
||||
const drivingIssueId = options.issue_id ?? process.env.PAPERCLIP_TASK_ID;
|
||||
if (!options.dry_run) required(drivingIssueId, "--issue-id or PAPERCLIP_TASK_ID");
|
||||
const groups = chunk(data.candidates, INTERACTION_LIMIT);
|
||||
if (groups.length === 0) {
|
||||
process.stdout.write("No archive candidates; no confirmation interaction created.\n");
|
||||
return [];
|
||||
}
|
||||
const bodies = groups.map((candidates, index) => confirmationBody(data, candidates, index, groups.length));
|
||||
if (options.dry_run) {
|
||||
process.stdout.write(`${JSON.stringify({ dryRun: true, issueId: drivingIssueId ?? null, interactions: bodies }, null, 2)}\n`);
|
||||
return bodies;
|
||||
}
|
||||
|
||||
const context = runtime(options);
|
||||
const created = [];
|
||||
for (const body of bodies) {
|
||||
created.push(await apiRequest(context, `/issues/${encodeURIComponent(drivingIssueId)}/interactions`, {
|
||||
method: "POST",
|
||||
body,
|
||||
}));
|
||||
}
|
||||
process.stdout.write(`${JSON.stringify({
|
||||
scanId: data.scanId,
|
||||
createdInteractions: created.map((item) => ({
|
||||
id: item.id,
|
||||
status: item.status,
|
||||
idempotencyKey: item.idempotencyKey,
|
||||
})),
|
||||
}, null, 2)}\n`);
|
||||
return created;
|
||||
}
|
||||
|
||||
function flattenInteractionFile(value) {
|
||||
if (Array.isArray(value)) return value;
|
||||
if (Array.isArray(value?.interactions)) return value.interactions;
|
||||
if (value?.interaction && typeof value.interaction === "object") return [value.interaction];
|
||||
return value && typeof value === "object" ? [value] : [];
|
||||
}
|
||||
|
||||
function optionValues(value) {
|
||||
if (value === undefined) return [];
|
||||
return Array.isArray(value) ? value : [value];
|
||||
}
|
||||
|
||||
async function resolveInteractions(options, context, drivingIssueId) {
|
||||
const files = optionValues(options.interaction_file);
|
||||
const ids = optionValues(options.interaction_id);
|
||||
const fromFiles = files.flatMap((file) => flattenInteractionFile(readJson(file)));
|
||||
if (ids.length === 0) return fromFiles;
|
||||
required(drivingIssueId, "--issue-id or PAPERCLIP_TASK_ID");
|
||||
const live = await apiRequest(context, `/issues/${encodeURIComponent(drivingIssueId)}/interactions`);
|
||||
if (!Array.isArray(live)) throw new Error("Interactions API returned a non-array response");
|
||||
const byId = new Map(live.map((interaction) => [interaction.id, interaction]));
|
||||
for (const id of ids) {
|
||||
if (!byId.has(id)) throw new Error(`Interaction ${id} was not found on the driving issue`);
|
||||
fromFiles.push(byId.get(id));
|
||||
}
|
||||
return fromFiles;
|
||||
}
|
||||
|
||||
function acceptedCandidates(interaction, scanData, candidateById) {
|
||||
if (interaction?.kind !== "request_checkbox_confirmation") {
|
||||
throw new Error(`Interaction ${interaction?.id ?? "(unknown)"} is not request_checkbox_confirmation`);
|
||||
}
|
||||
if (typeof interaction.idempotencyKey !== "string"
|
||||
|| !interaction.idempotencyKey.startsWith(`garden-inbox:${scanData.scanId}:`)) {
|
||||
throw new Error(`Interaction ${interaction.id ?? "(unknown)"} does not belong to scan ${scanData.scanId}`);
|
||||
}
|
||||
if (interaction.status !== "accepted" || interaction.result?.outcome !== "accepted") return [];
|
||||
const selected = interaction.result?.selectedOptionIds ?? [];
|
||||
if (!Array.isArray(selected)) throw new Error("Accepted interaction has invalid selectedOptionIds");
|
||||
const optionIds = new Set((interaction.payload?.options ?? []).map((option) => option.id));
|
||||
const seen = new Set();
|
||||
return selected.map((issueId) => {
|
||||
if (seen.has(issueId)) throw new Error(`Interaction selected duplicate option id ${issueId}`);
|
||||
seen.add(issueId);
|
||||
if (!optionIds.has(issueId)) throw new Error(`Selected id ${issueId} was not an option in the interaction`);
|
||||
const candidate = candidateById.get(issueId);
|
||||
if (!candidate || !OFFERED_BUCKETS.has(candidate.bucket)) {
|
||||
throw new Error(`Selected id ${issueId} is not an offered candidate in the originating scan`);
|
||||
}
|
||||
return candidate;
|
||||
});
|
||||
}
|
||||
|
||||
function renderApplySummary(results, interactions, dryRun, userId) {
|
||||
const lines = [
|
||||
`# Garden inbox ${dryRun ? "dry-run " : ""}apply summary`, "",
|
||||
`- Interactions inspected: ${interactions.length}`,
|
||||
`- Entries ${dryRun ? "that would be archived" : "archived"}: ${results.filter((result) => result.ok).length}`,
|
||||
`- Failures: ${results.filter((result) => !result.ok).length}`,
|
||||
];
|
||||
if (results.length === 0) {
|
||||
lines.push("", "Nothing was archived: the interaction was rejected, unresolved, expired, or accepted with no selections.");
|
||||
}
|
||||
for (const result of results) {
|
||||
lines.push("", `- ${result.ok ? "OK" : "FAILED"}: ${result.identifier ?? result.issueId} — ${result.message}`);
|
||||
if (result.ok) {
|
||||
lines.push(` Undo: \`DELETE /api/issues/${result.issueId}/inbox-archive\` with body \`${JSON.stringify({ userId })}\`, or use **Unarchive** in the issue properties pane.`);
|
||||
}
|
||||
}
|
||||
return `${lines.join("\n")}\n`;
|
||||
}
|
||||
|
||||
async function applyAccepted(options) {
|
||||
const data = candidateFile(options);
|
||||
const drivingIssueId = options.issue_id ?? process.env.PAPERCLIP_TASK_ID ?? null;
|
||||
const needsLiveApi = optionValues(options.interaction_id).length > 0 || !options.dry_run;
|
||||
const context = runtime(options, { requireApi: needsLiveApi });
|
||||
const interactions = await resolveInteractions(options, context, drivingIssueId);
|
||||
if (interactions.length === 0) throw new Error("Provide --interaction-id or --interaction-file");
|
||||
const candidateById = new Map(data.candidates.map((candidate) => [candidate.issueId, candidate]));
|
||||
const accepted = [];
|
||||
const seen = new Set();
|
||||
for (const interaction of interactions) {
|
||||
for (const candidate of acceptedCandidates(interaction, data, candidateById)) {
|
||||
if (!seen.has(candidate.issueId)) accepted.push(candidate);
|
||||
seen.add(candidate.issueId);
|
||||
}
|
||||
}
|
||||
|
||||
const results = [];
|
||||
for (const candidate of accepted) {
|
||||
if (options.dry_run) {
|
||||
results.push({ ...candidate, ok: true, message: "would archive this accepted inbox entry" });
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
await apiRequest(context, `/issues/${encodeURIComponent(candidate.issueId)}/inbox-archive`, {
|
||||
method: "POST",
|
||||
body: archiveTargetBody(data),
|
||||
});
|
||||
results.push({ ...candidate, ok: true, message: "archived accepted inbox entry" });
|
||||
} catch (error) {
|
||||
results.push({ ...candidate, ok: false, message: error.message });
|
||||
}
|
||||
}
|
||||
|
||||
const summary = renderApplySummary(results, interactions, Boolean(options.dry_run), data.userId);
|
||||
process.stdout.write(summary);
|
||||
if (options.summary) writeText(options.summary, summary);
|
||||
if (results.some((result) => !result.ok)) process.exitCode = 1;
|
||||
return results;
|
||||
}
|
||||
|
||||
export {
|
||||
acceptedCandidates,
|
||||
archiveTargetBody,
|
||||
classify,
|
||||
confirm,
|
||||
decodeJwtPayload,
|
||||
fetchMineInboxRows,
|
||||
normalizeApiBase,
|
||||
scan,
|
||||
};
|
||||
|
||||
async function main() {
|
||||
const { command, options } = parseArgs(process.argv.slice(2));
|
||||
if (options.help) {
|
||||
process.stdout.write(`${usage()}\n`);
|
||||
return;
|
||||
}
|
||||
if (command === "scan") await scan(options);
|
||||
else if (command === "confirm") await confirm(options);
|
||||
else await applyAccepted(options);
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`garden-inbox: ${error.message}\n`);
|
||||
process.exitCode = 1;
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
|
||||
import {
|
||||
acceptedCandidates,
|
||||
archiveTargetBody,
|
||||
classify,
|
||||
decodeJwtPayload,
|
||||
fetchMineInboxRows,
|
||||
normalizeApiBase,
|
||||
} from "./garden-inbox.mjs";
|
||||
|
||||
const now = new Date("2026-07-29T00:00:00.000Z");
|
||||
|
||||
function issue(overrides = {}) {
|
||||
return {
|
||||
id: "issue-1",
|
||||
identifier: "PAP-1",
|
||||
title: "Example",
|
||||
status: "done",
|
||||
updatedAt: "2025-01-01T00:00:00.000Z",
|
||||
blockedBy: [],
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function workspace(overrides = {}) {
|
||||
return {
|
||||
gone: false,
|
||||
error: null,
|
||||
status: "active",
|
||||
branchCommitAt: "2025-01-01T00:00:00.000Z",
|
||||
readiness: {
|
||||
git: { isMergedIntoBase: false, aheadCount: 0 },
|
||||
linkedIssues: [{ isTerminal: true }],
|
||||
},
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
test("classifies each archive and keep condition into one bucket", () => {
|
||||
const merged = workspace({
|
||||
readiness: {
|
||||
git: { isMergedIntoBase: true, aheadCount: 0 },
|
||||
linkedIssues: [{ isTerminal: true }],
|
||||
},
|
||||
});
|
||||
assert.equal(classify(issue(), merged, 60, now).bucket, "A");
|
||||
assert.equal(classify(issue(), workspace(), 60, now).bucket, "B");
|
||||
assert.equal(classify(issue(), workspace({
|
||||
readiness: {
|
||||
git: { isMergedIntoBase: false, aheadCount: 2 },
|
||||
linkedIssues: [{ isTerminal: true }],
|
||||
},
|
||||
}), 60, now).bucket, "C");
|
||||
assert.equal(classify(issue({ status: "in_progress" }), workspace(), 60, now).bucket, "D");
|
||||
});
|
||||
|
||||
test("keeps candidates when workspace safety inspection fails", () => {
|
||||
const result = classify(issue(), workspace({ error: "503 Service Unavailable", readiness: null }), 60, now);
|
||||
assert.equal(result.bucket, "D");
|
||||
assert.equal(result.reason.code, "workspace_inspection_failed");
|
||||
});
|
||||
|
||||
test("returns only accepted options from the originating scan", () => {
|
||||
const candidate = { issueId: "issue-1", bucket: "A" };
|
||||
const scan = { scanId: "scan-1" };
|
||||
const interaction = {
|
||||
id: "interaction-1",
|
||||
kind: "request_checkbox_confirmation",
|
||||
idempotencyKey: "garden-inbox:scan-1:1:1",
|
||||
status: "accepted",
|
||||
payload: { options: [{ id: "issue-1" }] },
|
||||
result: { outcome: "accepted", selectedOptionIds: ["issue-1"] },
|
||||
};
|
||||
assert.deepEqual(acceptedCandidates(interaction, scan, new Map([[candidate.issueId, candidate]])), [candidate]);
|
||||
});
|
||||
|
||||
test("preserves an overridden scan user for archive and undo requests", () => {
|
||||
assert.deepEqual(archiveTargetBody({ userId: "target-user" }), { userId: "target-user" });
|
||||
});
|
||||
|
||||
test("rejects selected ids that were not offered", () => {
|
||||
const scan = { scanId: "scan-1" };
|
||||
const interaction = {
|
||||
id: "interaction-1",
|
||||
kind: "request_checkbox_confirmation",
|
||||
idempotencyKey: "garden-inbox:scan-1:1:1",
|
||||
status: "accepted",
|
||||
payload: { options: [{ id: "issue-1" }] },
|
||||
result: { outcome: "accepted", selectedOptionIds: ["issue-2"] },
|
||||
};
|
||||
assert.throws(
|
||||
() => acceptedCandidates(interaction, scan, new Map()),
|
||||
/was not an option in the interaction/,
|
||||
);
|
||||
});
|
||||
|
||||
test("decodes the responsible user and normalizes API URLs locally", () => {
|
||||
const payload = Buffer.from(JSON.stringify({ responsible_user_id: "user-1" })).toString("base64url");
|
||||
assert.equal(decodeJwtPayload(`header.${payload}.signature`).responsible_user_id, "user-1");
|
||||
assert.equal(normalizeApiBase("https://paperclip.example/api/"), "https://paperclip.example");
|
||||
});
|
||||
|
||||
test("scans Mine once per status and merges rows by issue id", async (t) => {
|
||||
const requestedStatuses = [];
|
||||
t.mock.method(globalThis, "fetch", async (url) => {
|
||||
const parsed = new URL(url);
|
||||
const status = parsed.searchParams.get("status");
|
||||
requestedStatuses.push(status);
|
||||
const rows = status === "todo"
|
||||
? [issue({ id: "shared", status: "todo" })]
|
||||
: status === "done"
|
||||
? [issue({ id: "shared", status: "done" }), issue({ id: "done-only" })]
|
||||
: [];
|
||||
return new Response(JSON.stringify(rows), { status: 200 });
|
||||
});
|
||||
|
||||
const result = await fetchMineInboxRows({
|
||||
apiBase: "https://paperclip.example",
|
||||
apiKey: "secret",
|
||||
}, "user-1");
|
||||
|
||||
assert.deepEqual(requestedStatuses, ["backlog", "todo", "in_progress", "in_review", "blocked", "done"]);
|
||||
assert.equal(result.rows.length, 2);
|
||||
assert.equal(result.rows.find((row) => row.id === "shared").status, "done");
|
||||
assert.equal(result.coverage.duplicateCount, 1);
|
||||
assert.equal(result.coverage.complete, true);
|
||||
assert.deepEqual(result.coverage.statusCounts, {
|
||||
backlog: 0,
|
||||
todo: 1,
|
||||
in_progress: 0,
|
||||
in_review: 0,
|
||||
blocked: 0,
|
||||
done: 2,
|
||||
});
|
||||
});
|
||||
|
||||
test("warns when an individual status query reaches the Mine endpoint cap", async (t) => {
|
||||
t.mock.method(globalThis, "fetch", async (url) => {
|
||||
const status = new URL(url).searchParams.get("status");
|
||||
const rows = status === "done"
|
||||
? Array.from({ length: 500 }, (_, index) => issue({ id: `done-${index}` }))
|
||||
: [];
|
||||
return new Response(JSON.stringify(rows), { status: 200 });
|
||||
});
|
||||
|
||||
const result = await fetchMineInboxRows({
|
||||
apiBase: "https://paperclip.example",
|
||||
apiKey: "secret",
|
||||
}, "user-1");
|
||||
|
||||
assert.equal(result.rows.length, 500);
|
||||
assert.equal(result.coverage.complete, false);
|
||||
assert.deepEqual(result.coverage.cappedStatuses, ["done"]);
|
||||
assert.equal(result.coverage.queryCap, 500);
|
||||
});
|
||||
@@ -415,7 +415,7 @@ See [doc/DEVELOPING.md](doc/DEVELOPING.md) for the full development guide.
|
||||
- ⚪ Self-Organization
|
||||
- ⚪ Automatic Organizational Learning
|
||||
- ⚪ CEO Chat
|
||||
- 🟡 Cloud deployments (multi-tenant isolation & local→cloud sync shipped)
|
||||
- 🟡 Cloud deployments (multi-tenant isolation & company Import/Export shipped)
|
||||
- ⚪ Desktop App
|
||||
- ⚪ Bring-your-own-ticket-system (Asana / Linear / Jira as on-ramps)
|
||||
- ⚪ Connected Apps (one-click integrations, e.g. Vercel)
|
||||
|
||||
+2
-2
@@ -108,11 +108,11 @@ Paperclip should get better at turning completed work into reusable organization
|
||||
|
||||
We want a lighter-weight way to talk to leadership agents, but those conversations should still resolve to real work objects like plans, issues, approvals, or decisions. This should improve interaction without changing the core task-and-comments model.
|
||||
|
||||
### 🟡 Cloud deployments (multi-tenant isolation & local→cloud sync shipped)
|
||||
### 🟡 Cloud deployments (multi-tenant isolation & company Import/Export shipped)
|
||||
|
||||
Local-first remains important, but Paperclip also needs a cleaner shared deployment story. Teams should be able to run the same product in hosted or semi-hosted environments without changing the mental model.
|
||||
|
||||
Shipped so far: multi-tenant isolation with per-company JWT keys and company-scoped cloud tenants, local→cloud upstream sync, and cloud-managed instance bootstrap.
|
||||
Shipped so far: multi-tenant isolation with per-company JWT keys and company-scoped cloud tenants, portable company Import/Export (zip bundles that move a company between instances, local or cloud), and cloud-managed instance bootstrap. Next: a blob-store relay so large instances can move without a hand-carried bundle.
|
||||
|
||||
### ⚪ Desktop App
|
||||
|
||||
|
||||
+6
-9
@@ -332,20 +332,17 @@ By default, agents run on scheduled heartbeats and event-based triggers (task as
|
||||
|
||||
<br/>
|
||||
|
||||
## Paperclip Cloud Sync
|
||||
## Importing & Exporting Companies
|
||||
|
||||
Cloud upstream sync is behind the `Cloud Sync` experimental setting. Enable it in Instance Settings before pushing.
|
||||
Export a company to a portable package and import it into any other instance — local or cloud — from a local path or GitHub:
|
||||
|
||||
```bash
|
||||
paperclipai cloud connect https://your-stack.paperclip.app
|
||||
paperclipai cloud connect https://your-stack.paperclip.app --no-browser
|
||||
paperclipai cloud push --company <local-company-id> --dry-run
|
||||
paperclipai cloud push --company <local-company-id>
|
||||
paperclipai company export <company-id> --out ./my-export
|
||||
paperclipai company import ./my-export --dry-run
|
||||
paperclipai company import org/repo --target new
|
||||
```
|
||||
|
||||
`cloud connect` authorizes the local instance against the target stack and stores the upstream token in the local instance secret store. The default path opens a browser for consent; `--no-browser` uses the device-code flow and prints the verification URL and user code.
|
||||
|
||||
`cloud push --dry-run` exports the selected local company, sends a preview bundle to the connected Cloud stack, and exits with code `2` when conflicts need user resolution. A schema mismatch exits with code `3`. Running without `--dry-run` stages chunks idempotently, applies the run, and prints the final summary and recent progress events.
|
||||
The board UI has matching Export and Import pages in company settings: the Export page shows a fidelity panel listing what the bundle will not carry, and the Import page starts imported agents and routines paused by default, with a post-import activation step. See the [Importing & Exporting guide](https://github.com/paperclipai/paperclip/blob/master/docs/guides/board-operator/importing-and-exporting.md) for details.
|
||||
|
||||
## Development
|
||||
|
||||
|
||||
@@ -1,243 +0,0 @@
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { CompanyPortabilityExportResult } from "@paperclipai/shared";
|
||||
import {
|
||||
assertDiscoveryCompatible,
|
||||
buildBundleFromLocalCompany,
|
||||
cloudCommandExitCodes,
|
||||
connectCloud,
|
||||
resolveDeviceCodeExpiresAt,
|
||||
} from "../commands/client/cloud.js";
|
||||
import {
|
||||
LocalUpstreamPushCoordinator,
|
||||
normalizedContentHash,
|
||||
type LocalUpstreamExportBundle,
|
||||
} from "../commands/client/cloud-transfer.js";
|
||||
import { getCloudConnection } from "../commands/client/cloud-store.js";
|
||||
|
||||
const originalEnv = { ...process.env };
|
||||
const originalFetch = globalThis.fetch;
|
||||
|
||||
describe("cloud CLI helpers", () => {
|
||||
let tempHome: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tempHome = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-cloud-cli-"));
|
||||
process.env = { ...originalEnv, PAPERCLIP_HOME: tempHome };
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
process.env = { ...originalEnv };
|
||||
globalThis.fetch = originalFetch;
|
||||
vi.restoreAllMocks();
|
||||
fs.rmSync(tempHome, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("connects with the device-code flow and stores the resulting cloud connection", async () => {
|
||||
globalThis.fetch = vi.fn(async (url, init) => {
|
||||
const requestUrl = String(url);
|
||||
if (requestUrl.endsWith("/.well-known/paperclip-upstream")) {
|
||||
return jsonResponse(discovery());
|
||||
}
|
||||
if (requestUrl.endsWith("/api/upstream-sync/device-code")) {
|
||||
expect(JSON.parse(String(init?.body))).toMatchObject({
|
||||
stackId: "stack-1",
|
||||
scopes: ["upstream_import:preview", "upstream_import:write", "upstream_import:read"],
|
||||
});
|
||||
return jsonResponse({
|
||||
deviceCode: "device-1",
|
||||
userCode: "ABCD-EFGH",
|
||||
verificationUri: "https://cloud.example.test/api/upstream-sync/device-code/approve",
|
||||
expiresAt: new Date(Date.now() + 60_000).toISOString(),
|
||||
intervalSeconds: 0,
|
||||
});
|
||||
}
|
||||
if (requestUrl.endsWith("/api/upstream-sync/token")) {
|
||||
return jsonResponse({
|
||||
accessToken: "upt_test",
|
||||
scopes: ["upstream_import:preview"],
|
||||
token: {
|
||||
id: "token-1",
|
||||
companyStackId: "stack-1",
|
||||
targetOrigin: "https://cloud.example.test",
|
||||
sourceInstanceId: "paperclip-local-default",
|
||||
sourceInstanceFingerprint: "sha256:test",
|
||||
scopes: ["upstream_import:preview"],
|
||||
expiresAt: new Date(Date.now() + 60_000).toISOString(),
|
||||
},
|
||||
});
|
||||
}
|
||||
return jsonResponse({ error: "not_found" }, 404);
|
||||
}) as typeof fetch;
|
||||
|
||||
const connection = await connectCloud("https://cloud.example.test", { noBrowser: true, json: true });
|
||||
|
||||
expect(connection.accessToken).toBe("upt_test");
|
||||
expect(getCloudConnection("https://cloud.example.test")?.token.id).toBe("token-1");
|
||||
});
|
||||
|
||||
it("hard-blocks incompatible transfer schema versions with the stable schema exit code", () => {
|
||||
expect(() => assertDiscoveryCompatible(discovery({ supportedSchemaMajor: 99 }))).toThrow(/schema mismatch/i);
|
||||
expect(cloudCommandExitCodes.schemaMismatch).toBe(3);
|
||||
});
|
||||
|
||||
it("falls back to a bounded device-code expiry when the cloud omits or malforms expiresAt", () => {
|
||||
const now = Date.UTC(2026, 4, 22, 13, 0, 0);
|
||||
const validExpiry = "2026-05-22T13:05:00.000Z";
|
||||
|
||||
expect(resolveDeviceCodeExpiresAt(validExpiry, now)).toBe(Date.parse(validExpiry));
|
||||
expect(resolveDeviceCodeExpiresAt(undefined, now)).toBe(now + 15 * 60_000);
|
||||
expect(resolveDeviceCodeExpiresAt("not-a-date", now)).toBe(now + 15 * 60_000);
|
||||
});
|
||||
|
||||
it("builds deterministic chunks with validated payload hashes", async () => {
|
||||
const bundle = await buildTestBundle();
|
||||
|
||||
expect(bundle.chunks).toHaveLength(2);
|
||||
expect(bundle.chunks[0]?.sha256).toBe(normalizedContentHash(bundle.chunks[0]?.payload));
|
||||
expect(bundle.manifest.chunks[0]?.manifestHash).toBe(bundle.manifest.manifestHash);
|
||||
expect(bundle.manifest.idempotencyKey).toBe((await buildTestBundle()).manifest.idempotencyKey);
|
||||
});
|
||||
|
||||
it("reuses the same manifest and chunk identity when an interrupted apply is retried", async () => {
|
||||
const bundle = await buildTestBundle();
|
||||
const calls: Array<{ path: string; body: unknown }> = [];
|
||||
const coordinator = new LocalUpstreamPushCoordinator({
|
||||
targetOrigin: "https://cloud.example.test",
|
||||
paperclipCompanyId: "target-company-1",
|
||||
fetch: async (url, init) => {
|
||||
const parsed = new URL(String(url));
|
||||
const body = init?.body ? JSON.parse(String(init.body)) as unknown : {};
|
||||
calls.push({ path: parsed.pathname, body });
|
||||
if (parsed.pathname.endsWith("/runs")) return jsonResponse({ run: { id: "run-1" } });
|
||||
return jsonResponse({ run: { id: "run-1" }, summary: { create: 0, update: 0, adopt: 0, skip: 2, conflict: 0, staleMapping: 0 } });
|
||||
},
|
||||
});
|
||||
|
||||
await coordinator.apply(bundle);
|
||||
await coordinator.apply(bundle);
|
||||
|
||||
const runBodies = calls.filter((call) => call.path.endsWith("/runs")).map((call) => call.body as { manifest: { idempotencyKey: string } });
|
||||
const chunkBodies = calls.filter((call) => call.path.endsWith("/chunks")).map((call) => call.body as { chunkIndex: number; sha256: string });
|
||||
expect(runBodies).toHaveLength(2);
|
||||
expect(runBodies[0]?.manifest.idempotencyKey).toBe(runBodies[1]?.manifest.idempotencyKey);
|
||||
expect(chunkBodies[0]).toEqual(chunkBodies[2]);
|
||||
expect(chunkBodies[1]).toEqual(chunkBodies[3]);
|
||||
});
|
||||
});
|
||||
|
||||
async function buildTestBundle(): Promise<LocalUpstreamExportBundle> {
|
||||
return buildBundleFromLocalCompany({
|
||||
localCompanyId: "local-company-1",
|
||||
connection: {
|
||||
id: "conn-1",
|
||||
remoteUrl: "https://cloud.example.test",
|
||||
targetOrigin: "https://cloud.example.test",
|
||||
targetHost: "cloud.example.test",
|
||||
stackId: "stack-1",
|
||||
targetCompanyId: "target-company-1",
|
||||
accessToken: "upt_test",
|
||||
token: {
|
||||
id: "token-1",
|
||||
companyStackId: "stack-1",
|
||||
targetOrigin: "https://cloud.example.test",
|
||||
sourceInstanceId: "paperclip-local-default",
|
||||
sourceInstanceFingerprint: "sha256:test",
|
||||
scopes: ["upstream_import:preview"],
|
||||
expiresAt: new Date(Date.now() + 60_000).toISOString(),
|
||||
},
|
||||
privateKeyPem: "unused",
|
||||
sourcePublicKey: "unused",
|
||||
sourceInstanceId: "paperclip-local-default",
|
||||
sourceInstanceFingerprint: "sha256:test",
|
||||
scopes: ["upstream_import:preview"],
|
||||
createdAt: "2026-05-18T00:00:00.000Z",
|
||||
updatedAt: "2026-05-18T00:00:00.000Z",
|
||||
},
|
||||
discovery: discovery(),
|
||||
localApi: {
|
||||
post: async <T>() => portabilityExport() as T,
|
||||
},
|
||||
maxEntitiesPerChunk: 1,
|
||||
mode: "apply",
|
||||
});
|
||||
}
|
||||
|
||||
function discovery(overrides: Partial<{ supportedSchemaMajor: number }> = {}) {
|
||||
return {
|
||||
schema: "paperclip-upstream-discovery-v1",
|
||||
stack: {
|
||||
id: "stack-1",
|
||||
slug: "cloud-test",
|
||||
displayName: "Cloud Test",
|
||||
companyId: "target-company-1",
|
||||
origin: "https://cloud.example.test",
|
||||
},
|
||||
auth: {
|
||||
deviceCode: {
|
||||
deviceCodeUrl: "https://cloud.example.test/api/upstream-sync/device-code",
|
||||
verificationUrl: "https://cloud.example.test/api/upstream-sync/device-code/approve",
|
||||
tokenUrl: "https://cloud.example.test/api/upstream-sync/token",
|
||||
},
|
||||
scopes: ["upstream_import:preview", "upstream_import:write", "upstream_import:read"],
|
||||
},
|
||||
transfer: {
|
||||
supportedSchemaMajor: overrides.supportedSchemaMajor ?? 1,
|
||||
featureFlags: ["cloud_sync"],
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function portabilityExport(): CompanyPortabilityExportResult {
|
||||
return {
|
||||
rootPath: ".",
|
||||
paperclipExtensionPath: ".paperclip.yaml",
|
||||
manifest: {
|
||||
schemaVersion: 1,
|
||||
generatedAt: "2026-05-18T00:00:00.000Z",
|
||||
source: {
|
||||
companyId: "local-company-1",
|
||||
companyName: "Local Company",
|
||||
},
|
||||
includes: {
|
||||
company: true,
|
||||
agents: true,
|
||||
projects: true,
|
||||
issues: true,
|
||||
skills: true,
|
||||
},
|
||||
company: {
|
||||
path: "company.json",
|
||||
name: "Local Company",
|
||||
description: null,
|
||||
brandColor: null,
|
||||
logoPath: null,
|
||||
attachmentMaxBytes: null,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
feedbackDataSharingEnabled: false,
|
||||
feedbackDataSharingConsentAt: null,
|
||||
feedbackDataSharingConsentByUserId: null,
|
||||
feedbackDataSharingTermsVersion: null,
|
||||
},
|
||||
sidebar: null,
|
||||
agents: [],
|
||||
skills: [],
|
||||
projects: [],
|
||||
issues: [],
|
||||
envInputs: [],
|
||||
},
|
||||
files: {
|
||||
"README.md": "Local Company",
|
||||
},
|
||||
warnings: [],
|
||||
};
|
||||
}
|
||||
|
||||
function jsonResponse(body: unknown, status = 200): Response {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
}
|
||||
@@ -18,12 +18,14 @@ describe("resolveInlineSourceFromPath", () => {
|
||||
const tempDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-company-import-zip-"));
|
||||
tempDirs.push(tempDir);
|
||||
|
||||
const blobBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff]);
|
||||
const archivePath = path.join(tempDir, "paperclip-demo.zip");
|
||||
const archive = createStoredZipArchive(
|
||||
{
|
||||
"COMPANY.md": "# Company\n",
|
||||
".paperclip.yaml": "schema: paperclip/v1\n",
|
||||
"agents/ceo/AGENT.md": "# CEO\n",
|
||||
"blobs/4f2d1c9a": blobBytes,
|
||||
"notes/todo.txt": "ignore me\n",
|
||||
},
|
||||
"paperclip-demo",
|
||||
@@ -38,6 +40,11 @@ describe("resolveInlineSourceFromPath", () => {
|
||||
"COMPANY.md": "# Company\n",
|
||||
".paperclip.yaml": "schema: paperclip/v1\n",
|
||||
"agents/ceo/AGENT.md": "# CEO\n",
|
||||
"blobs/4f2d1c9a": {
|
||||
encoding: "base64",
|
||||
data: Buffer.from(blobBytes).toString("base64"),
|
||||
contentType: "application/octet-stream",
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
@@ -509,6 +509,9 @@ describe("renderCompanyImportResult", () => {
|
||||
{ slug: "ops", id: "project-2", action: "updated", name: "Operations", reason: "replace strategy" },
|
||||
{ slug: "archive", id: null, action: "skipped", name: "Archive", reason: "skip strategy" },
|
||||
],
|
||||
routines: [
|
||||
{ slug: "weekly-report", id: "routine-1", action: "created", title: "Weekly report", status: "paused" },
|
||||
],
|
||||
envInputs: [],
|
||||
warnings: ["Review API keys"],
|
||||
},
|
||||
|
||||
@@ -21,7 +21,7 @@ function crc32(bytes: Uint8Array) {
|
||||
return (crc ^ 0xffffffff) >>> 0;
|
||||
}
|
||||
|
||||
export function createStoredZipArchive(files: Record<string, string>, rootPath: string) {
|
||||
export function createStoredZipArchive(files: Record<string, string | Uint8Array>, rootPath: string) {
|
||||
const encoder = new TextEncoder();
|
||||
const localChunks: Uint8Array[] = [];
|
||||
const centralChunks: Uint8Array[] = [];
|
||||
@@ -30,7 +30,7 @@ export function createStoredZipArchive(files: Record<string, string>, rootPath:
|
||||
|
||||
for (const [relativePath, content] of Object.entries(files).sort(([left], [right]) => left.localeCompare(right))) {
|
||||
const fileName = encoder.encode(`${rootPath}/${relativePath}`);
|
||||
const body = encoder.encode(content);
|
||||
const body = typeof content === "string" ? encoder.encode(content) : content;
|
||||
const checksum = crc32(body);
|
||||
|
||||
const localHeader = new Uint8Array(30 + fileName.length);
|
||||
|
||||
@@ -276,6 +276,7 @@ describe("worktree helpers", () => {
|
||||
path.resolve("/tmp/paperclip-worktrees", "instances", "feature-worktree-support", "db"),
|
||||
);
|
||||
expect(config.database.embeddedPostgresPort).toBe(54339);
|
||||
expect(config.database.backup.enabled).toBe(false);
|
||||
expect(config.server.port).toBe(3110);
|
||||
expect(config.auth.publicBaseUrl).toBe("http://127.0.0.1:3110/");
|
||||
expect(config.storage.localDisk.baseDir).toBe(
|
||||
@@ -289,6 +290,7 @@ describe("worktree helpers", () => {
|
||||
expect(env.PAPERCLIP_HOME).toBe(path.resolve("/tmp/paperclip-worktrees"));
|
||||
expect(env.PAPERCLIP_INSTANCE_ID).toBe("feature-worktree-support");
|
||||
expect(env.PAPERCLIP_IN_WORKTREE).toBe("true");
|
||||
expect(env.PAPERCLIP_DB_BACKUP_ENABLED).toBe("false");
|
||||
expect(env.PAPERCLIP_WORKTREE_NAME).toBe("feature-worktree-support");
|
||||
expect(env.PAPERCLIP_WORKTREE_COLOR).toBe("#3abf7a");
|
||||
expect(formatShellExports(env)).toContain("export PAPERCLIP_INSTANCE_ID='feature-worktree-support'");
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { bytesToPortableFileEntry, isBlobStorePath, readZipArchive } from "../commands/client/zip.js";
|
||||
import { createStoredZipArchive } from "./helpers/zip.js";
|
||||
|
||||
describe("isBlobStorePath", () => {
|
||||
it("matches blobs/ entries at the archive root and under a package root", () => {
|
||||
expect(isBlobStorePath("blobs/4f2d1c9a")).toBe(true);
|
||||
expect(isBlobStorePath("paperclip-demo/blobs/4f2d1c9a")).toBe(true);
|
||||
expect(isBlobStorePath("tasks/pap-1/TASK.md")).toBe(false);
|
||||
expect(isBlobStorePath("blobs/nested/file")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("bytesToPortableFileEntry", () => {
|
||||
it("keeps blobs/ entries as base64 octet streams regardless of extension", () => {
|
||||
const bytes = new Uint8Array([0x00, 0x01, 0x80, 0xfe, 0xff]);
|
||||
expect(bytesToPortableFileEntry("blobs/4f2d1c9a", bytes)).toEqual({
|
||||
encoding: "base64",
|
||||
data: Buffer.from(bytes).toString("base64"),
|
||||
contentType: "application/octet-stream",
|
||||
});
|
||||
});
|
||||
|
||||
it("falls back to base64 when bytes are not valid UTF-8", () => {
|
||||
const invalidUtf8 = new Uint8Array([0x68, 0x69, 0xff, 0xfe, 0xc0]);
|
||||
expect(bytesToPortableFileEntry("tasks/pap-1/raw-notes", invalidUtf8)).toEqual({
|
||||
encoding: "base64",
|
||||
data: Buffer.from(invalidUtf8).toString("base64"),
|
||||
contentType: "application/octet-stream",
|
||||
});
|
||||
});
|
||||
|
||||
it("decodes valid UTF-8 entries to text", () => {
|
||||
const bytes = new TextEncoder().encode("# Notes\n\ncafé ✅\n");
|
||||
expect(bytesToPortableFileEntry("tasks/pap-1/TASK.md", bytes)).toBe("# Notes\n\ncafé ✅\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("readZipArchive", () => {
|
||||
it("round-trips blob and invalid UTF-8 entries byte-exactly", async () => {
|
||||
const blobBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff]);
|
||||
const invalidUtf8 = new Uint8Array([0x68, 0x69, 0xff, 0xfe, 0xc0]);
|
||||
const archive = createStoredZipArchive(
|
||||
{
|
||||
"COMPANY.md": "# Company\n",
|
||||
"blobs/4f2d1c9a": blobBytes,
|
||||
"notes/raw": invalidUtf8,
|
||||
},
|
||||
"paperclip-demo",
|
||||
);
|
||||
|
||||
await expect(readZipArchive(archive)).resolves.toEqual({
|
||||
rootPath: "paperclip-demo",
|
||||
files: {
|
||||
"COMPANY.md": "# Company\n",
|
||||
"blobs/4f2d1c9a": {
|
||||
encoding: "base64",
|
||||
data: Buffer.from(blobBytes).toString("base64"),
|
||||
contentType: "application/octet-stream",
|
||||
},
|
||||
"notes/raw": {
|
||||
encoding: "base64",
|
||||
data: Buffer.from(invalidUtf8).toString("base64"),
|
||||
contentType: "application/octet-stream",
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -1,177 +0,0 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { resolvePaperclipInstanceRoot } from "../../config/home.js";
|
||||
|
||||
export interface CloudConnectionTokenRecord {
|
||||
id: string;
|
||||
companyStackId: string;
|
||||
targetOrigin: string;
|
||||
sourceInstanceId: string;
|
||||
sourceInstanceFingerprint: string;
|
||||
scopes: string[];
|
||||
expiresAt: string;
|
||||
[key: string]: unknown;
|
||||
}
|
||||
|
||||
export interface CloudConnection {
|
||||
id: string;
|
||||
remoteUrl: string;
|
||||
targetOrigin: string;
|
||||
targetHost: string;
|
||||
stackId: string;
|
||||
stackSlug?: string | null;
|
||||
stackDisplayName?: string | null;
|
||||
targetCompanyId: string;
|
||||
accessToken: string;
|
||||
token: CloudConnectionTokenRecord;
|
||||
privateKeyPem: string;
|
||||
sourcePublicKey: string;
|
||||
sourceInstanceId: string;
|
||||
sourceInstanceFingerprint: string;
|
||||
scopes: string[];
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
}
|
||||
|
||||
interface CloudConnectionStore {
|
||||
version: 1;
|
||||
connections: Record<string, CloudConnection>;
|
||||
currentConnectionId?: string;
|
||||
}
|
||||
|
||||
function defaultStore(): CloudConnectionStore {
|
||||
return {
|
||||
version: 1,
|
||||
connections: {},
|
||||
};
|
||||
}
|
||||
|
||||
export function resolveCloudConnectionStorePath(): string {
|
||||
return path.resolve(resolvePaperclipInstanceRoot(), "secrets", "cloud-upstream-connections.json");
|
||||
}
|
||||
|
||||
export function readCloudConnectionStore(storePath = resolveCloudConnectionStorePath()): CloudConnectionStore {
|
||||
if (!fs.existsSync(storePath)) return defaultStore();
|
||||
const raw = JSON.parse(fs.readFileSync(storePath, "utf8")) as Partial<CloudConnectionStore> | null;
|
||||
const connections: Record<string, CloudConnection> = {};
|
||||
if (raw?.connections && typeof raw.connections === "object") {
|
||||
for (const [id, value] of Object.entries(raw.connections)) {
|
||||
const normalized = normalizeConnection(value);
|
||||
if (normalized) connections[id] = normalized;
|
||||
}
|
||||
}
|
||||
const currentConnectionId =
|
||||
typeof raw?.currentConnectionId === "string" && connections[raw.currentConnectionId]
|
||||
? raw.currentConnectionId
|
||||
: Object.values(connections).sort((left, right) => right.updatedAt.localeCompare(left.updatedAt))[0]?.id;
|
||||
return {
|
||||
version: 1,
|
||||
connections,
|
||||
currentConnectionId,
|
||||
};
|
||||
}
|
||||
|
||||
export function writeCloudConnectionStore(
|
||||
store: CloudConnectionStore,
|
||||
storePath = resolveCloudConnectionStorePath(),
|
||||
): void {
|
||||
fs.mkdirSync(path.dirname(storePath), { recursive: true });
|
||||
fs.writeFileSync(storePath, `${JSON.stringify(store, null, 2)}\n`, { mode: 0o600 });
|
||||
}
|
||||
|
||||
export function upsertCloudConnection(
|
||||
connection: CloudConnection,
|
||||
storePath = resolveCloudConnectionStorePath(),
|
||||
): CloudConnection {
|
||||
const store = readCloudConnectionStore(storePath);
|
||||
const existing = store.connections[connection.id];
|
||||
const now = new Date().toISOString();
|
||||
const next = {
|
||||
...connection,
|
||||
createdAt: existing?.createdAt ?? connection.createdAt ?? now,
|
||||
updatedAt: now,
|
||||
};
|
||||
store.connections[next.id] = next;
|
||||
store.currentConnectionId = next.id;
|
||||
writeCloudConnectionStore(store, storePath);
|
||||
return next;
|
||||
}
|
||||
|
||||
export function getCloudConnection(
|
||||
remoteUrlOrOrigin?: string,
|
||||
storePath = resolveCloudConnectionStorePath(),
|
||||
): CloudConnection | null {
|
||||
const store = readCloudConnectionStore(storePath);
|
||||
if (remoteUrlOrOrigin?.trim()) {
|
||||
const needle = normalizeRemoteLookup(remoteUrlOrOrigin);
|
||||
return Object.values(store.connections).find((connection) =>
|
||||
normalizeRemoteLookup(connection.remoteUrl) === needle ||
|
||||
normalizeRemoteLookup(connection.targetOrigin) === needle
|
||||
) ?? null;
|
||||
}
|
||||
return store.currentConnectionId ? store.connections[store.currentConnectionId] ?? null : null;
|
||||
}
|
||||
|
||||
function normalizeRemoteLookup(value: string): string {
|
||||
try {
|
||||
const url = new URL(value);
|
||||
return url.origin.replace(/\/+$/u, "");
|
||||
} catch {
|
||||
return value.trim().replace(/\/+$/u, "");
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeConnection(value: unknown): CloudConnection | null {
|
||||
if (typeof value !== "object" || value === null || Array.isArray(value)) return null;
|
||||
const record = value as Record<string, unknown>;
|
||||
const id = stringValue(record.id);
|
||||
const remoteUrl = stringValue(record.remoteUrl);
|
||||
const targetOrigin = stringValue(record.targetOrigin);
|
||||
const targetHost = stringValue(record.targetHost);
|
||||
const stackId = stringValue(record.stackId);
|
||||
const targetCompanyId = stringValue(record.targetCompanyId);
|
||||
const accessToken = stringValue(record.accessToken);
|
||||
const token = typeof record.token === "object" && record.token !== null && !Array.isArray(record.token)
|
||||
? record.token as CloudConnectionTokenRecord
|
||||
: null;
|
||||
const privateKeyPem = stringValue(record.privateKeyPem);
|
||||
const sourcePublicKey = stringValue(record.sourcePublicKey);
|
||||
const sourceInstanceId = stringValue(record.sourceInstanceId);
|
||||
const sourceInstanceFingerprint = stringValue(record.sourceInstanceFingerprint);
|
||||
const createdAt = stringValue(record.createdAt);
|
||||
const updatedAt = stringValue(record.updatedAt);
|
||||
if (
|
||||
!id || !remoteUrl || !targetOrigin || !targetHost || !stackId || !targetCompanyId ||
|
||||
!accessToken || !token || !privateKeyPem || !sourcePublicKey || !sourceInstanceId ||
|
||||
!sourceInstanceFingerprint || !createdAt || !updatedAt
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
return {
|
||||
id,
|
||||
remoteUrl,
|
||||
targetOrigin,
|
||||
targetHost,
|
||||
stackId,
|
||||
stackSlug: stringValue(record.stackSlug),
|
||||
stackDisplayName: stringValue(record.stackDisplayName),
|
||||
targetCompanyId,
|
||||
accessToken,
|
||||
token,
|
||||
privateKeyPem,
|
||||
sourcePublicKey,
|
||||
sourceInstanceId,
|
||||
sourceInstanceFingerprint,
|
||||
scopes: stringArray(record.scopes),
|
||||
createdAt,
|
||||
updatedAt,
|
||||
};
|
||||
}
|
||||
|
||||
function stringValue(value: unknown): string | null {
|
||||
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
|
||||
}
|
||||
|
||||
function stringArray(value: unknown): string[] {
|
||||
return Array.isArray(value) ? value.filter((entry): entry is string => typeof entry === "string") : [];
|
||||
}
|
||||
@@ -1,297 +0,0 @@
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
export const upstreamTransferSchema = {
|
||||
family: "paperclip-upstream-transfer",
|
||||
version: "1.0.0",
|
||||
major: 1,
|
||||
minor: 0,
|
||||
} as const;
|
||||
|
||||
export type NormalizedSha256 = `sha256:${string}`;
|
||||
|
||||
export interface SourceEntityKey {
|
||||
sourceInstanceId: string;
|
||||
sourceCompanyId: string;
|
||||
sourceEntityType: string;
|
||||
sourceEntityId: string;
|
||||
sourceNaturalKey?: string;
|
||||
}
|
||||
|
||||
export interface UpstreamTransferWarning {
|
||||
code: string;
|
||||
severity: "info" | "warning" | "blocker";
|
||||
message: string;
|
||||
entity?: SourceEntityKey;
|
||||
}
|
||||
|
||||
export interface UpstreamTransferEntityRecord {
|
||||
key: SourceEntityKey;
|
||||
contentHash: NormalizedSha256;
|
||||
dependencies: SourceEntityKey[];
|
||||
warnings: UpstreamTransferWarning[];
|
||||
}
|
||||
|
||||
export interface UpstreamTransferManifestSource {
|
||||
sourceInstanceId: string;
|
||||
sourceCompanyId: string;
|
||||
sourceInstanceKeyFingerprint: string;
|
||||
exporterVersion: string;
|
||||
sourceSchemaVersion: string;
|
||||
}
|
||||
|
||||
export interface UpstreamTransferManifestTarget {
|
||||
targetStackId: string;
|
||||
targetCompanyId: string;
|
||||
targetOrigin: string;
|
||||
supportedSchemaMajor: number;
|
||||
}
|
||||
|
||||
export interface UpstreamTransferChunk {
|
||||
chunkIndex: number;
|
||||
totalChunks: number;
|
||||
byteLength: number;
|
||||
sha256: NormalizedSha256;
|
||||
manifestHash: NormalizedSha256;
|
||||
}
|
||||
|
||||
export interface UpstreamTransferManifest {
|
||||
schema: typeof upstreamTransferSchema;
|
||||
source: UpstreamTransferManifestSource;
|
||||
target: UpstreamTransferManifestTarget;
|
||||
runId: string;
|
||||
idempotencyKey: string;
|
||||
generatedAt: string;
|
||||
entityCount: number;
|
||||
entities: UpstreamTransferEntityRecord[];
|
||||
chunks: UpstreamTransferChunk[];
|
||||
warnings: UpstreamTransferWarning[];
|
||||
featureFlags: string[];
|
||||
manifestHash: NormalizedSha256;
|
||||
}
|
||||
|
||||
export interface LocalUpstreamExportEntityInput {
|
||||
key: SourceEntityKey;
|
||||
body: Record<string, unknown>;
|
||||
dependencies?: SourceEntityKey[];
|
||||
warnings?: UpstreamTransferWarning[];
|
||||
conflictKeys?: string[];
|
||||
}
|
||||
|
||||
export interface LocalUpstreamExportEntity {
|
||||
record: UpstreamTransferEntityRecord;
|
||||
body: Record<string, unknown>;
|
||||
conflictKeys?: string[];
|
||||
}
|
||||
|
||||
export interface LocalUpstreamExportChunk {
|
||||
chunkIndex: number;
|
||||
totalChunks: number;
|
||||
byteLength: number;
|
||||
sha256: NormalizedSha256;
|
||||
payload: {
|
||||
entityKeys: SourceEntityKey[];
|
||||
};
|
||||
}
|
||||
|
||||
export interface LocalUpstreamExportBundle {
|
||||
manifest: UpstreamTransferManifest;
|
||||
entities: LocalUpstreamExportEntity[];
|
||||
chunks: LocalUpstreamExportChunk[];
|
||||
}
|
||||
|
||||
export interface BuildLocalUpstreamExportBundleInput {
|
||||
source: UpstreamTransferManifestSource;
|
||||
target: UpstreamTransferManifestTarget;
|
||||
runId: string;
|
||||
idempotencyKey: string;
|
||||
entities: LocalUpstreamExportEntityInput[];
|
||||
warnings?: UpstreamTransferWarning[];
|
||||
featureFlags?: string[];
|
||||
maxEntitiesPerChunk?: number;
|
||||
}
|
||||
|
||||
export interface LocalUpstreamPushCoordinatorOptions {
|
||||
targetOrigin: string;
|
||||
paperclipCompanyId: string;
|
||||
fetch?: typeof fetch;
|
||||
headers?: (input: { method: string; path: string }) => HeadersInit | Promise<HeadersInit>;
|
||||
}
|
||||
|
||||
export class UpstreamImportRequestError extends Error {
|
||||
readonly status: number;
|
||||
readonly body: unknown;
|
||||
|
||||
constructor(status: number, message: string, body: unknown) {
|
||||
super(message);
|
||||
this.status = status;
|
||||
this.body = body;
|
||||
}
|
||||
}
|
||||
|
||||
export class LocalUpstreamPushCoordinator {
|
||||
readonly #targetOrigin: string;
|
||||
readonly #paperclipCompanyId: string;
|
||||
readonly #fetch: typeof fetch;
|
||||
readonly #headers: NonNullable<LocalUpstreamPushCoordinatorOptions["headers"]>;
|
||||
|
||||
constructor(options: LocalUpstreamPushCoordinatorOptions) {
|
||||
this.#targetOrigin = options.targetOrigin.replace(/\/+$/u, "");
|
||||
this.#paperclipCompanyId = options.paperclipCompanyId;
|
||||
this.#fetch = options.fetch ?? fetch;
|
||||
this.#headers = options.headers ?? (() => ({}));
|
||||
}
|
||||
|
||||
async preview(bundle: LocalUpstreamExportBundle): Promise<unknown> {
|
||||
return this.post(`/api/companies/${encodeURIComponent(this.#paperclipCompanyId)}/upstream-imports/preview`, {
|
||||
manifest: bundle.manifest,
|
||||
entities: bundle.entities,
|
||||
});
|
||||
}
|
||||
|
||||
async apply(bundle: LocalUpstreamExportBundle): Promise<unknown> {
|
||||
const run = await this.post(`/api/companies/${encodeURIComponent(this.#paperclipCompanyId)}/upstream-imports/runs`, {
|
||||
mode: "apply",
|
||||
manifest: bundle.manifest,
|
||||
entities: bundle.entities,
|
||||
}) as { run?: { id?: unknown } };
|
||||
const runId = typeof run.run?.id === "string" ? run.run.id : undefined;
|
||||
if (!runId) {
|
||||
throw new Error("Remote upstream importer did not return a run id");
|
||||
}
|
||||
|
||||
for (const chunk of bundle.chunks) {
|
||||
await this.post(`/api/upstream-import-runs/${encodeURIComponent(runId)}/chunks`, chunk);
|
||||
}
|
||||
|
||||
return this.post(`/api/upstream-import-runs/${encodeURIComponent(runId)}/apply`, {});
|
||||
}
|
||||
|
||||
async events(runId: string): Promise<unknown> {
|
||||
return this.get(`/api/upstream-import-runs/${encodeURIComponent(runId)}/events`);
|
||||
}
|
||||
|
||||
private async get(path: string): Promise<unknown> {
|
||||
const response = await this.#fetch(`${this.#targetOrigin}${path}`, {
|
||||
method: "GET",
|
||||
headers: await this.#headers({ method: "GET", path }),
|
||||
});
|
||||
return parseCoordinatorResponse(response);
|
||||
}
|
||||
|
||||
private async post(path: string, body: unknown): Promise<unknown> {
|
||||
const response = await this.#fetch(`${this.#targetOrigin}${path}`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
...(await this.#headers({ method: "POST", path })),
|
||||
},
|
||||
body: JSON.stringify(body),
|
||||
});
|
||||
return parseCoordinatorResponse(response);
|
||||
}
|
||||
}
|
||||
|
||||
export function buildLocalUpstreamExportBundle(
|
||||
input: BuildLocalUpstreamExportBundleInput,
|
||||
): LocalUpstreamExportBundle {
|
||||
const entities = input.entities.map<LocalUpstreamExportEntity>((entity) => ({
|
||||
record: {
|
||||
key: entity.key,
|
||||
contentHash: normalizedContentHash(entity.body),
|
||||
dependencies: entity.dependencies ?? [],
|
||||
warnings: entity.warnings ?? [],
|
||||
},
|
||||
body: entity.body,
|
||||
conflictKeys: entity.conflictKeys,
|
||||
}));
|
||||
const chunks = buildLocalChunks(entities, input.maxEntitiesPerChunk ?? 100);
|
||||
const manifestWithoutHash = {
|
||||
schema: upstreamTransferSchema,
|
||||
source: input.source,
|
||||
target: input.target,
|
||||
runId: input.runId,
|
||||
idempotencyKey: input.idempotencyKey,
|
||||
generatedAt: new Date(0).toISOString(),
|
||||
entityCount: entities.length,
|
||||
entities: entities.map((entity) => entity.record),
|
||||
chunks: chunks.map(({ payload: _payload, ...chunk }) => chunk),
|
||||
warnings: input.warnings ?? [],
|
||||
featureFlags: (input.featureFlags ?? ["cloud_sync"]).slice().sort(),
|
||||
};
|
||||
const manifestHash = normalizedContentHash(manifestWithoutHash);
|
||||
return {
|
||||
manifest: {
|
||||
...manifestWithoutHash,
|
||||
chunks: manifestWithoutHash.chunks.map((chunk) => ({ ...chunk, manifestHash })),
|
||||
manifestHash,
|
||||
},
|
||||
entities,
|
||||
chunks,
|
||||
};
|
||||
}
|
||||
|
||||
export function normalizedContentHash(value: unknown): NormalizedSha256 {
|
||||
return `sha256:${createHash("sha256").update(canonicalJson(value)).digest("hex")}`;
|
||||
}
|
||||
|
||||
export function canonicalJson(value: unknown): string {
|
||||
return JSON.stringify(sortJson(value));
|
||||
}
|
||||
|
||||
function buildLocalChunks(
|
||||
entities: LocalUpstreamExportEntity[],
|
||||
maxEntitiesPerChunk: number,
|
||||
): LocalUpstreamExportChunk[] {
|
||||
if (!Number.isInteger(maxEntitiesPerChunk) || maxEntitiesPerChunk < 1) {
|
||||
throw new Error("maxEntitiesPerChunk must be a positive integer");
|
||||
}
|
||||
if (entities.length === 0) return [];
|
||||
|
||||
const groups: LocalUpstreamExportEntity[][] = [];
|
||||
for (let index = 0; index < entities.length; index += maxEntitiesPerChunk) {
|
||||
groups.push(entities.slice(index, index + maxEntitiesPerChunk));
|
||||
}
|
||||
|
||||
return groups.map((group, index) => {
|
||||
const payload = {
|
||||
entityKeys: group.map((entity) => entity.record.key),
|
||||
};
|
||||
return {
|
||||
chunkIndex: index,
|
||||
totalChunks: groups.length,
|
||||
byteLength: Buffer.byteLength(canonicalJson(payload)),
|
||||
sha256: normalizedContentHash(payload),
|
||||
payload,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
function sortJson(value: unknown): unknown {
|
||||
if (Array.isArray(value)) return value.map(sortJson);
|
||||
if (typeof value !== "object" || value === null) return value;
|
||||
return Object.fromEntries(
|
||||
Object.entries(value as Record<string, unknown>)
|
||||
.sort(([left], [right]) => left.localeCompare(right))
|
||||
.map(([key, entry]) => [key, sortJson(entry)]),
|
||||
);
|
||||
}
|
||||
|
||||
async function parseCoordinatorResponse(response: Response): Promise<unknown> {
|
||||
const text = await response.text();
|
||||
const parsed = text.trim() ? safeParseJson(text) : {};
|
||||
if (!response.ok) {
|
||||
const message = typeof parsed === "object" && parsed !== null && "error" in parsed
|
||||
? String((parsed as { error: unknown }).error)
|
||||
: `Upstream importer request failed with ${response.status}`;
|
||||
throw new UpstreamImportRequestError(response.status, message, parsed);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
function safeParseJson(text: string): unknown {
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch {
|
||||
return text;
|
||||
}
|
||||
}
|
||||
@@ -1,722 +0,0 @@
|
||||
import { createHash, generateKeyPairSync, randomBytes, randomUUID, sign } from "node:crypto";
|
||||
import { createServer, type Server } from "node:http";
|
||||
import { URL } from "node:url";
|
||||
import { Command } from "commander";
|
||||
import pc from "picocolors";
|
||||
import type {
|
||||
CompanyPortabilityExportResult,
|
||||
CompanyPortabilityFileEntry,
|
||||
InstanceExperimentalSettings,
|
||||
} from "@paperclipai/shared";
|
||||
import { openUrl } from "../../client/board-auth.js";
|
||||
import { resolvePaperclipInstanceId } from "../../config/home.js";
|
||||
import {
|
||||
addCommonClientOptions,
|
||||
apiPath,
|
||||
handleCommandError,
|
||||
printOutput,
|
||||
resolveCommandContext,
|
||||
type BaseClientOptions,
|
||||
} from "./common.js";
|
||||
import {
|
||||
buildLocalUpstreamExportBundle,
|
||||
LocalUpstreamPushCoordinator,
|
||||
normalizedContentHash,
|
||||
upstreamTransferSchema,
|
||||
UpstreamImportRequestError,
|
||||
type LocalUpstreamExportBundle,
|
||||
type LocalUpstreamExportEntityInput,
|
||||
type SourceEntityKey,
|
||||
type UpstreamTransferManifestSource,
|
||||
type UpstreamTransferManifestTarget,
|
||||
type UpstreamTransferWarning,
|
||||
} from "./cloud-transfer.js";
|
||||
import {
|
||||
getCloudConnection,
|
||||
upsertCloudConnection,
|
||||
type CloudConnection,
|
||||
type CloudConnectionTokenRecord,
|
||||
} from "./cloud-store.js";
|
||||
|
||||
const CLOUD_SYNC_CONFLICT_EXIT_CODE = 2;
|
||||
const CLOUD_SYNC_SCHEMA_MISMATCH_EXIT_CODE = 3;
|
||||
const CLOUD_SYNC_SCOPES = ["upstream_import:preview", "upstream_import:write", "upstream_import:read"];
|
||||
const DEVICE_CODE_FALLBACK_EXPIRES_MS = 15 * 60_000;
|
||||
|
||||
interface CloudConnectOptions extends BaseClientOptions {
|
||||
noBrowser?: boolean;
|
||||
}
|
||||
|
||||
interface CloudPushOptions extends BaseClientOptions {
|
||||
company?: string;
|
||||
remoteUrl?: string;
|
||||
dryRun?: boolean;
|
||||
maxEntitiesPerChunk?: number;
|
||||
}
|
||||
|
||||
interface UpstreamDiscovery {
|
||||
schema: string;
|
||||
stack: {
|
||||
id: string;
|
||||
slug?: string;
|
||||
displayName?: string;
|
||||
companyId: string;
|
||||
origin: string;
|
||||
};
|
||||
auth: {
|
||||
pkce?: {
|
||||
authorizeUrl: string;
|
||||
tokenUrl: string;
|
||||
codeChallengeMethod: string;
|
||||
};
|
||||
deviceCode?: {
|
||||
deviceCodeUrl: string;
|
||||
verificationUrl: string;
|
||||
tokenUrl: string;
|
||||
};
|
||||
scopes?: string[];
|
||||
};
|
||||
transfer: {
|
||||
supportedSchemaMajor: number;
|
||||
featureFlags?: string[];
|
||||
};
|
||||
}
|
||||
|
||||
interface TokenResponse {
|
||||
accessToken: string;
|
||||
token: CloudConnectionTokenRecord;
|
||||
scopes?: string[];
|
||||
expiresAt?: string;
|
||||
}
|
||||
|
||||
class CloudAuthRequestError extends Error {
|
||||
readonly status: number;
|
||||
readonly body: unknown;
|
||||
|
||||
constructor(status: number, message: string, body: unknown) {
|
||||
super(message);
|
||||
this.status = status;
|
||||
this.body = body;
|
||||
}
|
||||
}
|
||||
|
||||
export function registerCloudCommands(program: Command): void {
|
||||
const cloud = program.command("cloud").description("Paperclip Cloud upstream sync commands");
|
||||
|
||||
addCommonClientOptions(
|
||||
cloud
|
||||
.command("connect")
|
||||
.description("Authorize this local instance to push into a Paperclip Cloud stack")
|
||||
.argument("<remote-url>", "Paperclip Cloud stack URL")
|
||||
.option("--no-browser", "Use the device-code flow instead of opening a browser", false)
|
||||
.action(async (remoteUrl: string, opts: CloudConnectOptions) => {
|
||||
try {
|
||||
await connectCloud(remoteUrl, opts);
|
||||
} catch (err) {
|
||||
handleCommandError(err);
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
addCommonClientOptions(
|
||||
cloud
|
||||
.command("push")
|
||||
.description("Preview or apply a local company push into the connected Paperclip Cloud stack")
|
||||
.requiredOption("--company <local-company-id>", "Local company ID to export")
|
||||
.option("--remote-url <remote-url>", "Use a specific stored cloud connection")
|
||||
.option("--dry-run", "Preview without applying", false)
|
||||
.option("--max-entities-per-chunk <count>", "Chunk size for upstream uploads", (value) => Number(value), 100)
|
||||
.action(async (opts: CloudPushOptions) => {
|
||||
try {
|
||||
await pushCloud(opts);
|
||||
} catch (err) {
|
||||
if (isSchemaMismatchError(err)) {
|
||||
console.error(pc.red(err instanceof Error ? err.message : String(err)));
|
||||
process.exitCode = CLOUD_SYNC_SCHEMA_MISMATCH_EXIT_CODE;
|
||||
return;
|
||||
}
|
||||
handleCommandError(err);
|
||||
}
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
export async function connectCloud(remoteUrl: string, opts: CloudConnectOptions = {}): Promise<CloudConnection> {
|
||||
const ctx = resolveCommandContext(opts);
|
||||
const discovery = await discoverUpstream(remoteUrl);
|
||||
assertDiscoveryCompatible(discovery);
|
||||
const source = createSourceIdentity();
|
||||
const token = await authorizeConnection(discovery, source, {
|
||||
noBrowser: Boolean(opts.noBrowser),
|
||||
});
|
||||
const targetOrigin = discovery.stack.origin.replace(/\/+$/u, "");
|
||||
const targetHost = new URL(targetOrigin).host;
|
||||
const now = new Date().toISOString();
|
||||
const connection = upsertCloudConnection({
|
||||
id: connectionId(targetOrigin),
|
||||
remoteUrl,
|
||||
targetOrigin,
|
||||
targetHost,
|
||||
stackId: discovery.stack.id,
|
||||
stackSlug: discovery.stack.slug ?? null,
|
||||
stackDisplayName: discovery.stack.displayName ?? null,
|
||||
targetCompanyId: discovery.stack.companyId,
|
||||
accessToken: token.accessToken,
|
||||
token: token.token,
|
||||
privateKeyPem: source.privateKeyPem,
|
||||
sourcePublicKey: source.sourcePublicKey,
|
||||
sourceInstanceId: source.sourceInstanceId,
|
||||
sourceInstanceFingerprint: source.sourceInstanceFingerprint,
|
||||
scopes: token.scopes ?? token.token.scopes ?? CLOUD_SYNC_SCOPES,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
});
|
||||
|
||||
if (ctx.json) {
|
||||
printOutput(redactConnection(connection), { json: true });
|
||||
} else {
|
||||
console.log(pc.bold("Connected to Paperclip Cloud"));
|
||||
console.log(`stack=${connection.stackDisplayName ?? connection.stackSlug ?? connection.stackId}`);
|
||||
console.log(`origin=${connection.targetOrigin}`);
|
||||
console.log(`company=${connection.targetCompanyId}`);
|
||||
}
|
||||
return connection;
|
||||
}
|
||||
|
||||
export async function pushCloud(opts: CloudPushOptions): Promise<unknown> {
|
||||
const ctx = resolveCommandContext(opts, { requireCompany: false });
|
||||
const localCompanyId = requiredString(opts.company, "--company");
|
||||
await assertCloudSyncEnabled(ctx.api.get<InstanceExperimentalSettings>("/api/instance/settings/experimental"));
|
||||
const connection = getCloudConnection(opts.remoteUrl);
|
||||
if (!connection) {
|
||||
throw new Error("No cloud connection found. Run `paperclipai cloud connect <remote-url>` first.");
|
||||
}
|
||||
|
||||
const discovery = await discoverUpstream(connection.targetOrigin);
|
||||
assertDiscoveryCompatible(discovery);
|
||||
const bundle = await buildBundleFromLocalCompany({
|
||||
localCompanyId,
|
||||
connection,
|
||||
discovery,
|
||||
localApi: ctx.api,
|
||||
maxEntitiesPerChunk: opts.maxEntitiesPerChunk,
|
||||
mode: opts.dryRun ? "preview" : "apply",
|
||||
});
|
||||
const coordinator = new LocalUpstreamPushCoordinator({
|
||||
targetOrigin: connection.targetOrigin,
|
||||
paperclipCompanyId: connection.targetCompanyId,
|
||||
headers: ({ method, path }) => cloudProofHeaders(connection, method, path),
|
||||
});
|
||||
|
||||
const result = opts.dryRun ? await coordinator.preview(bundle) : await coordinator.apply(bundle);
|
||||
const runId = getRunId(result);
|
||||
const events = !opts.dryRun && runId ? await coordinator.events(runId).catch(() => null) : null;
|
||||
const summary = summarizeResult(result);
|
||||
const conflictCount = summary.conflict + summary.staleMapping;
|
||||
|
||||
if (ctx.json) {
|
||||
printOutput({ result, events }, { json: true });
|
||||
} else {
|
||||
console.log(pc.bold(opts.dryRun ? "Cloud Push Preview" : "Cloud Push Applied"));
|
||||
console.log(`run=${runId ?? "-"}`);
|
||||
console.log(`manifest=${bundle.manifest.manifestHash}`);
|
||||
console.log(
|
||||
`create=${summary.create} update=${summary.update} adopt=${summary.adopt} ` +
|
||||
`skip=${summary.skip} conflict=${summary.conflict} staleMapping=${summary.staleMapping}`,
|
||||
);
|
||||
printWarnings(result);
|
||||
printConflicts(result);
|
||||
printEvents(events);
|
||||
}
|
||||
|
||||
if (conflictCount > 0) {
|
||||
process.exitCode = CLOUD_SYNC_CONFLICT_EXIT_CODE;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
export async function discoverUpstream(remoteUrl: string): Promise<UpstreamDiscovery> {
|
||||
const base = new URL(remoteUrl);
|
||||
const discoveryUrl = new URL("/.well-known/paperclip-upstream", base);
|
||||
return requestCloudJson<UpstreamDiscovery>(discoveryUrl.toString(), { method: "GET" });
|
||||
}
|
||||
|
||||
export function assertDiscoveryCompatible(discovery: UpstreamDiscovery): void {
|
||||
if (discovery.schema !== "paperclip-upstream-discovery-v1") {
|
||||
throw new Error("Remote URL is not a Paperclip Cloud upstream target.");
|
||||
}
|
||||
if (discovery.transfer.supportedSchemaMajor !== upstreamTransferSchema.major) {
|
||||
throw new Error(
|
||||
`Cloud upstream schema mismatch: local major ${upstreamTransferSchema.major}, remote supports ${discovery.transfer.supportedSchemaMajor}.`,
|
||||
);
|
||||
}
|
||||
if (!discovery.transfer.featureFlags?.includes("cloud_sync")) {
|
||||
throw new Error("Remote Paperclip Cloud stack does not advertise the cloud_sync transfer flag.");
|
||||
}
|
||||
}
|
||||
|
||||
export function resolveDeviceCodeExpiresAt(expiresAt: string | undefined, nowMs = Date.now()): number {
|
||||
const parsed = typeof expiresAt === "string" ? Date.parse(expiresAt) : NaN;
|
||||
return Number.isFinite(parsed) ? parsed : nowMs + DEVICE_CODE_FALLBACK_EXPIRES_MS;
|
||||
}
|
||||
|
||||
export async function buildBundleFromLocalCompany(input: {
|
||||
localCompanyId: string;
|
||||
connection: CloudConnection;
|
||||
discovery: UpstreamDiscovery;
|
||||
localApi: {
|
||||
post<T>(path: string, body?: unknown): Promise<T | null>;
|
||||
};
|
||||
maxEntitiesPerChunk?: number;
|
||||
mode: "preview" | "apply";
|
||||
}): Promise<LocalUpstreamExportBundle> {
|
||||
const exported = await input.localApi.post<CompanyPortabilityExportResult>(
|
||||
apiPath`/api/companies/${input.localCompanyId}/export`,
|
||||
{
|
||||
include: {
|
||||
company: true,
|
||||
agents: true,
|
||||
projects: true,
|
||||
issues: true,
|
||||
skills: true,
|
||||
},
|
||||
expandReferencedSkills: true,
|
||||
},
|
||||
);
|
||||
if (!exported) throw new Error("Local company export returned no data.");
|
||||
|
||||
const sourceHash = normalizedContentHash({
|
||||
manifest: exported.manifest,
|
||||
files: exported.files,
|
||||
});
|
||||
const source: UpstreamTransferManifestSource = {
|
||||
sourceInstanceId: input.connection.sourceInstanceId,
|
||||
sourceCompanyId: input.localCompanyId,
|
||||
sourceInstanceKeyFingerprint: input.connection.sourceInstanceFingerprint,
|
||||
exporterVersion: "paperclipai-cli-cloud-v1",
|
||||
sourceSchemaVersion: "paperclip-local-portability-v1",
|
||||
};
|
||||
const target: UpstreamTransferManifestTarget = {
|
||||
targetStackId: input.discovery.stack.id,
|
||||
targetCompanyId: input.discovery.stack.companyId,
|
||||
targetOrigin: input.discovery.stack.origin,
|
||||
supportedSchemaMajor: input.discovery.transfer.supportedSchemaMajor,
|
||||
};
|
||||
const entities = buildEntitiesFromPortableExport(input.localCompanyId, input.connection.sourceInstanceId, exported);
|
||||
const idempotencyKey = [
|
||||
input.mode,
|
||||
input.connection.sourceInstanceId,
|
||||
input.localCompanyId,
|
||||
input.discovery.stack.id,
|
||||
sourceHash,
|
||||
].join(":");
|
||||
return buildLocalUpstreamExportBundle({
|
||||
source,
|
||||
target,
|
||||
runId: `local-${input.mode}-${shortHash(idempotencyKey)}`,
|
||||
idempotencyKey,
|
||||
entities,
|
||||
warnings: exported.warnings.map((message): UpstreamTransferWarning => ({
|
||||
code: "local_company_export_warning",
|
||||
severity: "warning",
|
||||
message,
|
||||
})),
|
||||
featureFlags: ["cloud_sync"],
|
||||
maxEntitiesPerChunk: input.maxEntitiesPerChunk,
|
||||
});
|
||||
}
|
||||
|
||||
async function authorizeConnection(
|
||||
discovery: UpstreamDiscovery,
|
||||
source: ReturnType<typeof createSourceIdentity>,
|
||||
opts: { noBrowser: boolean },
|
||||
): Promise<TokenResponse> {
|
||||
if (!opts.noBrowser && canOpenBrowser() && discovery.auth.pkce) {
|
||||
try {
|
||||
return await authorizeWithBrowser(discovery, source);
|
||||
} catch (error) {
|
||||
console.error(pc.yellow(`Browser authorization failed; falling back to device-code flow. ${errorMessage(error)}`));
|
||||
}
|
||||
}
|
||||
if (!discovery.auth.deviceCode) {
|
||||
throw new Error("Remote Paperclip Cloud stack does not support device-code authorization.");
|
||||
}
|
||||
return authorizeWithDeviceCode(discovery, source, { openBrowser: !opts.noBrowser && canOpenBrowser() });
|
||||
}
|
||||
|
||||
async function authorizeWithBrowser(
|
||||
discovery: UpstreamDiscovery,
|
||||
source: ReturnType<typeof createSourceIdentity>,
|
||||
): Promise<TokenResponse> {
|
||||
const pkce = discovery.auth.pkce;
|
||||
if (!pkce) throw new Error("Remote did not advertise PKCE authorization.");
|
||||
const callback = await startPkceCallbackServer();
|
||||
const verifier = randomBytes(32).toString("base64url");
|
||||
const challenge = createHash("sha256").update(verifier).digest("base64url");
|
||||
const state = randomUUID();
|
||||
const authorizeUrl = new URL(pkce.authorizeUrl);
|
||||
authorizeUrl.searchParams.set("redirectUri", callback.redirectUri);
|
||||
authorizeUrl.searchParams.set("state", state);
|
||||
authorizeUrl.searchParams.set("codeChallenge", challenge);
|
||||
authorizeUrl.searchParams.set("codeChallengeMethod", "S256");
|
||||
authorizeUrl.searchParams.set("sourceInstanceId", source.sourceInstanceId);
|
||||
authorizeUrl.searchParams.set("sourceInstanceFingerprint", source.sourceInstanceFingerprint);
|
||||
authorizeUrl.searchParams.set("sourcePublicKey", source.sourcePublicKey);
|
||||
authorizeUrl.searchParams.set("scopes", CLOUD_SYNC_SCOPES.join(" "));
|
||||
|
||||
try {
|
||||
console.error(`Open this URL to approve cloud sync:\n${authorizeUrl.toString()}`);
|
||||
if (!(await openUrl(authorizeUrl.toString()))) {
|
||||
throw new Error("Could not open a browser.");
|
||||
}
|
||||
const code = await callback.waitForCode(state);
|
||||
return requestCloudJson<TokenResponse>(pkce.tokenUrl, {
|
||||
method: "POST",
|
||||
body: JSON.stringify({
|
||||
grantType: "authorization_code",
|
||||
code,
|
||||
redirectUri: callback.redirectUri,
|
||||
codeVerifier: verifier,
|
||||
}),
|
||||
});
|
||||
} finally {
|
||||
await callback.close();
|
||||
}
|
||||
}
|
||||
|
||||
async function authorizeWithDeviceCode(
|
||||
discovery: UpstreamDiscovery,
|
||||
source: ReturnType<typeof createSourceIdentity>,
|
||||
opts: { openBrowser: boolean },
|
||||
): Promise<TokenResponse> {
|
||||
const device = discovery.auth.deviceCode;
|
||||
if (!device) throw new Error("Remote did not advertise device-code authorization.");
|
||||
const response = await requestCloudJson<{
|
||||
deviceCode: string;
|
||||
userCode: string;
|
||||
verificationUri: string;
|
||||
expiresAt?: string;
|
||||
intervalSeconds?: number;
|
||||
}>(device.deviceCodeUrl, {
|
||||
method: "POST",
|
||||
body: JSON.stringify({
|
||||
stackId: discovery.stack.id,
|
||||
sourceInstanceId: source.sourceInstanceId,
|
||||
sourceInstanceFingerprint: source.sourceInstanceFingerprint,
|
||||
sourcePublicKey: source.sourcePublicKey,
|
||||
scopes: CLOUD_SYNC_SCOPES,
|
||||
}),
|
||||
});
|
||||
console.error(pc.bold("Cloud device authorization required"));
|
||||
console.error(`Open: ${response.verificationUri}`);
|
||||
console.error(`Code: ${response.userCode}`);
|
||||
if (opts.openBrowser) await openUrl(response.verificationUri);
|
||||
|
||||
const expiresAt = resolveDeviceCodeExpiresAt(response.expiresAt);
|
||||
const intervalMs = Math.max(500, (response.intervalSeconds ?? 5) * 1000);
|
||||
while (Date.now() < expiresAt) {
|
||||
await sleep(intervalMs);
|
||||
try {
|
||||
return await requestCloudJson<TokenResponse>(device.tokenUrl, {
|
||||
method: "POST",
|
||||
body: JSON.stringify({
|
||||
grantType: "device_code",
|
||||
deviceCode: response.deviceCode,
|
||||
}),
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof CloudAuthRequestError && error.body && typeof error.body === "object") {
|
||||
const code = (error.body as { error?: unknown }).error;
|
||||
if (code === "authorization_pending") continue;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
throw new Error("Device-code authorization expired before it was approved.");
|
||||
}
|
||||
|
||||
function buildEntitiesFromPortableExport(
|
||||
localCompanyId: string,
|
||||
sourceInstanceId: string,
|
||||
exported: CompanyPortabilityExportResult,
|
||||
): LocalUpstreamExportEntityInput[] {
|
||||
const companyKey: SourceEntityKey = {
|
||||
sourceInstanceId,
|
||||
sourceCompanyId: localCompanyId,
|
||||
sourceEntityType: "company",
|
||||
sourceEntityId: localCompanyId,
|
||||
sourceNaturalKey: exported.manifest.company?.name ?? localCompanyId,
|
||||
};
|
||||
const entities: LocalUpstreamExportEntityInput[] = [
|
||||
{
|
||||
key: companyKey,
|
||||
body: {
|
||||
kind: "paperclip_company_portability_manifest",
|
||||
manifest: exported.manifest,
|
||||
rootPath: exported.rootPath,
|
||||
paperclipExtensionPath: exported.paperclipExtensionPath,
|
||||
fileCount: Object.keys(exported.files).length,
|
||||
},
|
||||
conflictKeys: [`company:${companyKey.sourceNaturalKey ?? localCompanyId}`],
|
||||
},
|
||||
];
|
||||
|
||||
for (const [filePath, entry] of Object.entries(exported.files).sort(([left], [right]) => left.localeCompare(right))) {
|
||||
entities.push({
|
||||
key: {
|
||||
sourceInstanceId,
|
||||
sourceCompanyId: localCompanyId,
|
||||
sourceEntityType: "company_setting",
|
||||
sourceEntityId: shortHash(filePath),
|
||||
sourceNaturalKey: filePath,
|
||||
},
|
||||
body: {
|
||||
kind: "paperclip_portable_file",
|
||||
path: filePath,
|
||||
entry: normalizePortableFileEntry(entry),
|
||||
},
|
||||
dependencies: [companyKey],
|
||||
conflictKeys: [`portable_file:${filePath}`],
|
||||
});
|
||||
}
|
||||
return entities;
|
||||
}
|
||||
|
||||
function normalizePortableFileEntry(entry: CompanyPortabilityFileEntry): Record<string, unknown> {
|
||||
if (typeof entry === "string") {
|
||||
return { encoding: "utf8", data: entry };
|
||||
}
|
||||
return { ...entry };
|
||||
}
|
||||
|
||||
async function assertCloudSyncEnabled(settingsPromise: Promise<InstanceExperimentalSettings | null>): Promise<void> {
|
||||
const settings = await settingsPromise;
|
||||
if (settings?.enableCloudSync !== true) {
|
||||
throw new Error(
|
||||
"Cloud sync is disabled. Enable the cloud sync experimental setting before running `paperclipai cloud push`.",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function cloudProofHeaders(connection: CloudConnection, method: string, pathAndSearch: string): Record<string, string> {
|
||||
const timestamp = new Date().toISOString();
|
||||
const nonce = randomUUID();
|
||||
const payload = [
|
||||
method,
|
||||
connection.targetHost.toLowerCase(),
|
||||
pathAndSearch,
|
||||
connection.token.id,
|
||||
connection.sourceInstanceId,
|
||||
timestamp,
|
||||
nonce,
|
||||
].join("\n");
|
||||
return {
|
||||
Authorization: `Bearer ${connection.accessToken}`,
|
||||
"X-Paperclip-Upstream-Source-Instance-Id": connection.sourceInstanceId,
|
||||
"X-Paperclip-Upstream-Proof-Timestamp": timestamp,
|
||||
"X-Paperclip-Upstream-Proof-Nonce": nonce,
|
||||
"X-Paperclip-Upstream-Proof-Signature": sign(
|
||||
null,
|
||||
Buffer.from(payload, "utf8"),
|
||||
connection.privateKeyPem,
|
||||
).toString("base64url"),
|
||||
};
|
||||
}
|
||||
|
||||
async function requestCloudJson<T>(url: string, init: RequestInit): Promise<T> {
|
||||
const headers = new Headers(init.headers);
|
||||
headers.set("accept", "application/json");
|
||||
if (init.body !== undefined && !headers.has("content-type")) {
|
||||
headers.set("content-type", "application/json");
|
||||
}
|
||||
const response = await fetch(url, { ...init, headers });
|
||||
const text = await response.text();
|
||||
const parsed = text.trim() ? JSON.parse(text) as unknown : {};
|
||||
if (!response.ok) {
|
||||
const message = typeof parsed === "object" && parsed !== null && "error" in parsed
|
||||
? String((parsed as { error: unknown }).error)
|
||||
: `Cloud request failed with ${response.status}`;
|
||||
throw new CloudAuthRequestError(response.status, message, parsed);
|
||||
}
|
||||
return parsed as T;
|
||||
}
|
||||
|
||||
function createSourceIdentity() {
|
||||
const { publicKey, privateKey } = generateKeyPairSync("ed25519");
|
||||
const sourcePublicKey = publicKey.export({ type: "spki", format: "pem" }).toString();
|
||||
const sourceInstanceFingerprint = `sha256:${createHash("sha256")
|
||||
.update(publicKey.export({ type: "spki", format: "der" }))
|
||||
.digest("hex")}`;
|
||||
return {
|
||||
sourceInstanceId: `paperclip-local-${resolvePaperclipInstanceId()}`,
|
||||
sourceInstanceFingerprint,
|
||||
sourcePublicKey,
|
||||
privateKeyPem: privateKey.export({ type: "pkcs8", format: "pem" }).toString(),
|
||||
};
|
||||
}
|
||||
|
||||
async function startPkceCallbackServer(): Promise<{
|
||||
redirectUri: string;
|
||||
waitForCode: (state: string) => Promise<string>;
|
||||
close: () => Promise<void>;
|
||||
}> {
|
||||
let resolveCode: ((code: string) => void) | null = null;
|
||||
let rejectCode: ((error: Error) => void) | null = null;
|
||||
let expectedState = "";
|
||||
const codePromise = new Promise<string>((resolve, reject) => {
|
||||
resolveCode = resolve;
|
||||
rejectCode = reject;
|
||||
});
|
||||
const server = createServer((req, res) => {
|
||||
const url = new URL(req.url ?? "/", "http://127.0.0.1");
|
||||
const code = url.searchParams.get("code");
|
||||
const state = url.searchParams.get("state");
|
||||
if (!code || state !== expectedState) {
|
||||
res.writeHead(400, { "Content-Type": "text/plain" });
|
||||
res.end("Paperclip Cloud authorization failed. You can close this tab.");
|
||||
rejectCode?.(new Error("Authorization callback was missing a valid code or state."));
|
||||
return;
|
||||
}
|
||||
res.writeHead(200, { "Content-Type": "text/plain" });
|
||||
res.end("Paperclip Cloud authorization complete. You can close this tab.");
|
||||
resolveCode?.(code);
|
||||
});
|
||||
await listenOnLoopback(server);
|
||||
const address = server.address();
|
||||
if (typeof address !== "object" || !address?.port) {
|
||||
throw new Error("Failed to start local authorization callback server.");
|
||||
}
|
||||
return {
|
||||
redirectUri: `http://127.0.0.1:${address.port}/cloud/callback`,
|
||||
waitForCode: (state: string) => {
|
||||
expectedState = state;
|
||||
return codePromise;
|
||||
},
|
||||
close: () => closeServer(server),
|
||||
};
|
||||
}
|
||||
|
||||
function listenOnLoopback(server: Server): Promise<void> {
|
||||
return new Promise((resolve, reject) => {
|
||||
server.once("error", reject);
|
||||
server.listen(0, "127.0.0.1", () => {
|
||||
server.off("error", reject);
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function closeServer(server: Server): Promise<void> {
|
||||
return new Promise((resolve, reject) => {
|
||||
server.close((error) => error ? reject(error) : resolve());
|
||||
});
|
||||
}
|
||||
|
||||
function canOpenBrowser(): boolean {
|
||||
if (process.platform === "darwin" || process.platform === "win32") return true;
|
||||
return Boolean(process.env.DISPLAY || process.env.WAYLAND_DISPLAY);
|
||||
}
|
||||
|
||||
function summarizeResult(result: unknown): {
|
||||
create: number;
|
||||
update: number;
|
||||
adopt: number;
|
||||
skip: number;
|
||||
conflict: number;
|
||||
staleMapping: number;
|
||||
} {
|
||||
const summary = asRecord(asRecord(result)?.summary);
|
||||
return {
|
||||
create: numberValue(summary?.create),
|
||||
update: numberValue(summary?.update),
|
||||
adopt: numberValue(summary?.adopt),
|
||||
skip: numberValue(summary?.skip),
|
||||
conflict: numberValue(summary?.conflict),
|
||||
staleMapping: numberValue(summary?.staleMapping),
|
||||
};
|
||||
}
|
||||
|
||||
function printWarnings(result: unknown): void {
|
||||
const warnings = Array.isArray(asRecord(result)?.warnings) ? asRecord(result)?.warnings as unknown[] : [];
|
||||
for (const warning of warnings) {
|
||||
const record = asRecord(warning);
|
||||
console.log(pc.yellow(`warning=${record?.code ?? "warning"} ${record?.message ?? ""}`.trim()));
|
||||
}
|
||||
}
|
||||
|
||||
function printConflicts(result: unknown): void {
|
||||
const conflicts = Array.isArray(asRecord(result)?.conflicts) ? asRecord(result)?.conflicts as unknown[] : [];
|
||||
for (const conflict of conflicts.slice(0, 10)) {
|
||||
const record = asRecord(conflict);
|
||||
console.log(pc.red(`conflict=${record?.conflictKind ?? "target_conflict"} target=${record?.targetEntityId ?? "-"}`));
|
||||
}
|
||||
if (conflicts.length > 10) console.log(pc.red(`conflicts_truncated=${conflicts.length - 10}`));
|
||||
}
|
||||
|
||||
function printEvents(events: unknown): void {
|
||||
const rows = Array.isArray(asRecord(events)?.events) ? asRecord(events)?.events as unknown[] : [];
|
||||
for (const row of rows.slice(-10)) {
|
||||
const event = asRecord(row);
|
||||
console.log(pc.dim(`event=${event?.action ?? "-"} target=${event?.targetEntityId ?? "-"}`));
|
||||
}
|
||||
}
|
||||
|
||||
function getRunId(result: unknown): string | null {
|
||||
const run = asRecord(asRecord(result)?.run);
|
||||
return typeof run?.id === "string" ? run.id : null;
|
||||
}
|
||||
|
||||
function redactConnection(connection: CloudConnection): Record<string, unknown> {
|
||||
return {
|
||||
id: connection.id,
|
||||
remoteUrl: connection.remoteUrl,
|
||||
targetOrigin: connection.targetOrigin,
|
||||
stackId: connection.stackId,
|
||||
targetCompanyId: connection.targetCompanyId,
|
||||
scopes: connection.scopes,
|
||||
expiresAt: connection.token.expiresAt,
|
||||
};
|
||||
}
|
||||
|
||||
function connectionId(targetOrigin: string): string {
|
||||
return `cloud-${shortHash(targetOrigin)}`;
|
||||
}
|
||||
|
||||
function shortHash(value: string): string {
|
||||
return createHash("sha256").update(value).digest("hex").slice(0, 16);
|
||||
}
|
||||
|
||||
function requiredString(value: unknown, label: string): string {
|
||||
if (typeof value === "string" && value.trim()) return value.trim();
|
||||
throw new Error(`${label} is required.`);
|
||||
}
|
||||
|
||||
function numberValue(value: unknown): number {
|
||||
return typeof value === "number" && Number.isFinite(value) ? value : 0;
|
||||
}
|
||||
|
||||
function asRecord(value: unknown): Record<string, unknown> | null {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value)
|
||||
? value as Record<string, unknown>
|
||||
: null;
|
||||
}
|
||||
|
||||
function isSchemaMismatchError(error: unknown): boolean {
|
||||
if (error instanceof UpstreamImportRequestError) {
|
||||
return JSON.stringify(error.body).toLowerCase().includes("schema");
|
||||
}
|
||||
return error instanceof Error && error.message.toLowerCase().includes("schema mismatch");
|
||||
}
|
||||
|
||||
function errorMessage(error: unknown): string {
|
||||
return error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
|
||||
function sleep(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
|
||||
export const cloudCommandExitCodes = {
|
||||
conflict: CLOUD_SYNC_CONFLICT_EXIT_CODE,
|
||||
schemaMismatch: CLOUD_SYNC_SCHEMA_MISMATCH_EXIT_CODE,
|
||||
} as const;
|
||||
@@ -15,7 +15,7 @@ import type {
|
||||
import { getTelemetryClient, trackCompanyImported } from "../../telemetry.js";
|
||||
import { ApiRequestError } from "../../client/http.js";
|
||||
import { openUrl } from "../../client/board-auth.js";
|
||||
import { binaryContentTypeByExtension, readZipArchive } from "./zip.js";
|
||||
import { binaryContentTypeByExtension, bytesToPortableFileEntry, isBlobStorePath, readZipArchive } from "./zip.js";
|
||||
import {
|
||||
addCommonClientOptions,
|
||||
apiPath,
|
||||
@@ -140,16 +140,6 @@ type ImportSelectionState = {
|
||||
skills: Set<string>;
|
||||
};
|
||||
|
||||
function readPortableFileEntry(filePath: string, contents: Buffer): CompanyPortabilityFileEntry {
|
||||
const contentType = binaryContentTypeByExtension[path.extname(filePath).toLowerCase()];
|
||||
if (!contentType) return contents.toString("utf8");
|
||||
return {
|
||||
encoding: "base64",
|
||||
data: contents.toString("base64"),
|
||||
contentType,
|
||||
};
|
||||
}
|
||||
|
||||
function portableFileEntryToWriteValue(entry: CompanyPortabilityFileEntry): string | Uint8Array {
|
||||
if (typeof entry === "string") return entry;
|
||||
return Buffer.from(entry.data, "base64");
|
||||
@@ -213,7 +203,7 @@ function shouldIncludePortableFile(filePath: string): boolean {
|
||||
const isMarkdown = baseName.endsWith(".md");
|
||||
const isPaperclipYaml = baseName === ".paperclip.yaml" || baseName === ".paperclip.yml";
|
||||
const contentType = binaryContentTypeByExtension[path.extname(baseName).toLowerCase()];
|
||||
return isMarkdown || isPaperclipYaml || Boolean(contentType);
|
||||
return isMarkdown || isPaperclipYaml || Boolean(contentType) || isBlobStorePath(filePath);
|
||||
}
|
||||
|
||||
function findPortableExtensionPath(files: Record<string, CompanyPortabilityFileEntry>): string | null {
|
||||
@@ -932,7 +922,7 @@ async function collectPackageFiles(
|
||||
if (!entry.isFile()) continue;
|
||||
const relativePath = path.relative(root, absolutePath).replace(/\\/g, "/");
|
||||
if (!shouldIncludePortableFile(relativePath)) continue;
|
||||
files[relativePath] = readPortableFileEntry(relativePath, await readFile(absolutePath));
|
||||
files[relativePath] = bytesToPortableFileEntry(relativePath, await readFile(absolutePath));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+10
-129
@@ -1,129 +1,10 @@
|
||||
import { inflateRawSync } from "node:zlib";
|
||||
import path from "node:path";
|
||||
import type { CompanyPortabilityFileEntry } from "@paperclipai/shared";
|
||||
|
||||
const textDecoder = new TextDecoder();
|
||||
|
||||
export const binaryContentTypeByExtension: Record<string, string> = {
|
||||
".gif": "image/gif",
|
||||
".jpeg": "image/jpeg",
|
||||
".jpg": "image/jpeg",
|
||||
".png": "image/png",
|
||||
".svg": "image/svg+xml",
|
||||
".webp": "image/webp",
|
||||
};
|
||||
|
||||
function normalizeArchivePath(pathValue: string) {
|
||||
return pathValue
|
||||
.replace(/\\/g, "/")
|
||||
.split("/")
|
||||
.filter(Boolean)
|
||||
.join("/");
|
||||
}
|
||||
|
||||
function readUint16(source: Uint8Array, offset: number) {
|
||||
return source[offset]! | (source[offset + 1]! << 8);
|
||||
}
|
||||
|
||||
function readUint32(source: Uint8Array, offset: number) {
|
||||
return (
|
||||
source[offset]! |
|
||||
(source[offset + 1]! << 8) |
|
||||
(source[offset + 2]! << 16) |
|
||||
(source[offset + 3]! << 24)
|
||||
) >>> 0;
|
||||
}
|
||||
|
||||
function sharedArchiveRoot(paths: string[]) {
|
||||
if (paths.length === 0) return null;
|
||||
const firstSegments = paths
|
||||
.map((entry) => normalizeArchivePath(entry).split("/").filter(Boolean))
|
||||
.filter((parts) => parts.length > 0);
|
||||
if (firstSegments.length === 0) return null;
|
||||
const candidate = firstSegments[0]![0]!;
|
||||
return firstSegments.every((parts) => parts.length > 1 && parts[0] === candidate)
|
||||
? candidate
|
||||
: null;
|
||||
}
|
||||
|
||||
function bytesToPortableFileEntry(pathValue: string, bytes: Uint8Array): CompanyPortabilityFileEntry {
|
||||
const contentType = binaryContentTypeByExtension[path.extname(pathValue).toLowerCase()];
|
||||
if (!contentType) return textDecoder.decode(bytes);
|
||||
return {
|
||||
encoding: "base64",
|
||||
data: Buffer.from(bytes).toString("base64"),
|
||||
contentType,
|
||||
};
|
||||
}
|
||||
|
||||
async function inflateZipEntry(compressionMethod: number, bytes: Uint8Array) {
|
||||
if (compressionMethod === 0) return bytes;
|
||||
if (compressionMethod !== 8) {
|
||||
throw new Error("Unsupported zip archive: only STORE and DEFLATE entries are supported.");
|
||||
}
|
||||
return new Uint8Array(inflateRawSync(bytes));
|
||||
}
|
||||
|
||||
export async function readZipArchive(source: ArrayBuffer | Uint8Array): Promise<{
|
||||
rootPath: string | null;
|
||||
files: Record<string, CompanyPortabilityFileEntry>;
|
||||
}> {
|
||||
const bytes = source instanceof Uint8Array ? source : new Uint8Array(source);
|
||||
const entries: Array<{ path: string; body: CompanyPortabilityFileEntry }> = [];
|
||||
let offset = 0;
|
||||
|
||||
while (offset + 4 <= bytes.length) {
|
||||
const signature = readUint32(bytes, offset);
|
||||
if (signature === 0x02014b50 || signature === 0x06054b50) break;
|
||||
if (signature !== 0x04034b50) {
|
||||
throw new Error("Invalid zip archive: unsupported local file header.");
|
||||
}
|
||||
|
||||
if (offset + 30 > bytes.length) {
|
||||
throw new Error("Invalid zip archive: truncated local file header.");
|
||||
}
|
||||
|
||||
const generalPurposeFlag = readUint16(bytes, offset + 6);
|
||||
const compressionMethod = readUint16(bytes, offset + 8);
|
||||
const compressedSize = readUint32(bytes, offset + 18);
|
||||
const fileNameLength = readUint16(bytes, offset + 26);
|
||||
const extraFieldLength = readUint16(bytes, offset + 28);
|
||||
|
||||
if ((generalPurposeFlag & 0x0008) !== 0) {
|
||||
throw new Error("Unsupported zip archive: data descriptors are not supported.");
|
||||
}
|
||||
|
||||
const nameOffset = offset + 30;
|
||||
const bodyOffset = nameOffset + fileNameLength + extraFieldLength;
|
||||
const bodyEnd = bodyOffset + compressedSize;
|
||||
if (bodyEnd > bytes.length) {
|
||||
throw new Error("Invalid zip archive: truncated file contents.");
|
||||
}
|
||||
|
||||
const rawArchivePath = textDecoder.decode(bytes.slice(nameOffset, nameOffset + fileNameLength));
|
||||
const archivePath = normalizeArchivePath(rawArchivePath);
|
||||
const isDirectoryEntry = /\/$/.test(rawArchivePath.replace(/\\/g, "/"));
|
||||
if (archivePath && !isDirectoryEntry) {
|
||||
const entryBytes = await inflateZipEntry(compressionMethod, bytes.slice(bodyOffset, bodyEnd));
|
||||
entries.push({
|
||||
path: archivePath,
|
||||
body: bytesToPortableFileEntry(archivePath, entryBytes),
|
||||
});
|
||||
}
|
||||
|
||||
offset = bodyEnd;
|
||||
}
|
||||
|
||||
const rootPath = sharedArchiveRoot(entries.map((entry) => entry.path));
|
||||
const files: Record<string, CompanyPortabilityFileEntry> = {};
|
||||
for (const entry of entries) {
|
||||
const normalizedPath =
|
||||
rootPath && entry.path.startsWith(`${rootPath}/`)
|
||||
? entry.path.slice(rootPath.length + 1)
|
||||
: entry.path;
|
||||
if (!normalizedPath) continue;
|
||||
files[normalizedPath] = entry.body;
|
||||
}
|
||||
|
||||
return { rootPath, files };
|
||||
}
|
||||
// The node-side portability zip reader lives in @paperclipai/shared so the
|
||||
// server can consume the same codec (a raw uploaded zip is unzipped into the
|
||||
// exact `{ rootPath, files }` bundle the inline import source carries). This
|
||||
// module re-exports it to keep the CLI's existing import paths stable.
|
||||
export {
|
||||
binaryContentTypeByExtension,
|
||||
bytesToPortableFileEntry,
|
||||
isBlobStorePath,
|
||||
readZipArchive,
|
||||
} from "@paperclipai/shared/portability-zip";
|
||||
@@ -197,7 +197,7 @@ export function buildWorktreeConfig(input: {
|
||||
embeddedPostgresDataDir: paths.embeddedPostgresDataDir,
|
||||
embeddedPostgresPort: databasePort,
|
||||
backup: {
|
||||
enabled: source?.database.backup.enabled ?? true,
|
||||
enabled: false,
|
||||
intervalMinutes: source?.database.backup.intervalMinutes ?? 60,
|
||||
retentionDays: source?.database.backup.retentionDays ?? 30,
|
||||
dir: paths.backupDir,
|
||||
@@ -258,6 +258,7 @@ export function buildWorktreeEnvEntries(
|
||||
PAPERCLIP_CONFIG: paths.configPath,
|
||||
PAPERCLIP_CONTEXT: paths.contextPath,
|
||||
PAPERCLIP_IN_WORKTREE: "true",
|
||||
PAPERCLIP_DB_BACKUP_ENABLED: "false",
|
||||
...(branding?.name ? { PAPERCLIP_WORKTREE_NAME: branding.name } : {}),
|
||||
...(branding?.color ? { PAPERCLIP_WORKTREE_COLOR: branding.color } : {}),
|
||||
};
|
||||
|
||||
@@ -22,7 +22,6 @@ import { registerRoutineCommands } from "./commands/routines.js";
|
||||
import { registerPipelineCommands } from "./commands/pipelines.js";
|
||||
import { registerFeedbackCommands } from "./commands/client/feedback.js";
|
||||
import { registerSecretCommands } from "./commands/client/secrets.js";
|
||||
import { registerCloudCommands } from "./commands/client/cloud.js";
|
||||
import { registerSkillsCommands } from "./commands/client/skills.js";
|
||||
import { registerTeamCommands } from "./commands/client/teams.js";
|
||||
import { applyDataDirOverride, type DataDirOptionLike } from "./config/data-dir.js";
|
||||
@@ -181,7 +180,6 @@ registerRoutineCommands(program);
|
||||
registerPipelineCommands(program);
|
||||
registerFeedbackCommands(program);
|
||||
registerSecretCommands(program);
|
||||
registerCloudCommands(program);
|
||||
registerSkillsCommands(program);
|
||||
registerTeamCommands(program);
|
||||
registerWorktreeCommands(program);
|
||||
|
||||
@@ -355,6 +355,7 @@ This command:
|
||||
- creates an isolated instance under `~/.paperclip-worktrees/instances/<worktree-id>/`
|
||||
- when run inside a linked git worktree, mirrors the effective git hooks into that worktree's private git dir
|
||||
- picks a free app port and embedded PostgreSQL port
|
||||
- disables automatic database backups for the isolated instance
|
||||
- by default seeds the isolated DB in `minimal` mode from the current effective Paperclip instance/config (repo-local worktree config when present, otherwise the default instance) via a logical SQL snapshot
|
||||
|
||||
Seed modes:
|
||||
@@ -374,6 +375,7 @@ Provisioned git worktrees also pause seeded routines that still have enabled sch
|
||||
That repo-local env also sets:
|
||||
|
||||
- `PAPERCLIP_IN_WORKTREE=true`
|
||||
- `PAPERCLIP_DB_BACKUP_ENABLED=false`
|
||||
- `PAPERCLIP_WORKTREE_NAME=<worktree-name>`
|
||||
- `PAPERCLIP_WORKTREE_COLOR=<hex-color>`
|
||||
|
||||
@@ -655,6 +657,11 @@ schemas. Defaults:
|
||||
- retain 30 days
|
||||
- backup dir: `~/.paperclip/instances/default/data/backups`
|
||||
|
||||
Automatic backups are disabled for isolated worktree instances created with
|
||||
`paperclipai worktree init` or `paperclipai worktree:make`. Existing worktree
|
||||
configs are migrated to the disabled setting when their server next starts. The
|
||||
main/default instance keeps the normal enabled-by-default behavior.
|
||||
|
||||
Configure these in:
|
||||
|
||||
```sh
|
||||
|
||||
@@ -21,6 +21,14 @@ Container images for running coding-agent harnesses in sandboxed environments (f
|
||||
- tini (PID-1 init, ensures signal propagation)
|
||||
- Non-root user `paperclip` (uid/gid 1000)
|
||||
|
||||
The NodeSource install puts `node` on the default `PATH`. The agent shim in this
|
||||
image runs the harness directly with that `PATH`. The shim does not source a
|
||||
login profile, and the runtime never writes a profile or an rc file. Some
|
||||
sandbox providers instead wrap each command in a login shell. That shell sources
|
||||
`/etc/profile` and the user profile files to read an owner-supplied `PATH`. No
|
||||
exec path sources `nvm`. For the full exec-path contract, see
|
||||
`packages/plugins/sandbox-providers/SANDBOX-REQUIREMENTS.md`.
|
||||
|
||||
**Paperclip Binaries:**
|
||||
- `/usr/local/bin/paperclip-agent-shim`: Go binary compiled from `tools/agent-shim/`. Reads `/run/paperclip/runtime-command.json` and `syscall.Exec`s the harness CLI.
|
||||
|
||||
|
||||
+1
-1
@@ -62,7 +62,7 @@ Headers: X-Paperclip-Run-Id: {runId}
|
||||
}
|
||||
```
|
||||
|
||||
The optional `comment` field adds a comment in the same call.
|
||||
The optional `comment` field adds a comment in the same call. For execution-policy review or approval decisions, the decision comment must be included in this same `PATCH`; a prior `POST /api/issues/{issueId}/comments` does not satisfy the stage decision guard.
|
||||
|
||||
Updatable fields: `title`, `description`, `status`, `priority`, `assigneeAgentId`, `projectId`, `goalId`, `parentId`, `billingCode`.
|
||||
|
||||
|
||||
@@ -19,6 +19,8 @@ PATCH /api/issues/{issueId}
|
||||
{ "status": "done", "comment": "Implemented login endpoint with JWT auth." }
|
||||
```
|
||||
|
||||
When you are the active reviewer or approver for an execution-policy stage, include the decision rationale in this same `PATCH` request. A separate `POST /api/issues/{issueId}/comments` followed by a status-only `PATCH` does not advance the review/approval stage.
|
||||
|
||||
## Comment Style
|
||||
|
||||
Use concise markdown with:
|
||||
|
||||
@@ -29,6 +29,14 @@ my-company/
|
||||
- **SKILL.md** files are compatible with the Agent Skills ecosystem.
|
||||
- **.paperclip.yaml** holds Paperclip-specific config (adapter types, env inputs, budgets) as an optional sidecar.
|
||||
|
||||
## Export & Import in the App
|
||||
|
||||
Both flows are also available in the web UI as company settings pages: **Export** and **Import** appear in the company settings navigation.
|
||||
|
||||
The **Export** page lets you pick exactly which files go into the bundle before downloading it. Above the file tree it shows a **"Not included in this export"** panel — the export fidelity report — listing data the bundle will not carry (for example attachments, approvals, cost history, or activity log entries), with blocking issues highlighted.
|
||||
|
||||
The **Import** page previews the package, lets you resolve name collisions and adapter assignments, and applies the import. A **"Start imported agents and routines paused"** checkbox (on by default) makes imported agents and routines land paused instead of live. After the import finishes, an **"Activate imported agents and routines"** panel lists everything that was imported paused so you can resume the agents and activate the routines you select — nothing starts running until you say so.
|
||||
|
||||
## Exporting a Company
|
||||
|
||||
Export a company into a portable folder:
|
||||
@@ -142,6 +150,8 @@ The preview shows:
|
||||
|
||||
Imported agents always land with timer heartbeats disabled. Assignment/on-demand wake behavior from the package is preserved, but scheduled runs stay off until a board operator re-enables them.
|
||||
|
||||
Imports can additionally request `pauseAutomations` (the default in the app's Import page) so imported agents and routines land fully paused. Use the post-import activation panel — or resume the agents and activate the routines individually — when you are ready for them to run.
|
||||
|
||||
### Common Workflows
|
||||
|
||||
**Clone a company template from GitHub:**
|
||||
@@ -184,11 +194,14 @@ The CLI commands use these API endpoints under the hood:
|
||||
| Action | Endpoint |
|
||||
|--------|----------|
|
||||
| Export company | `POST /api/companies/{companyId}/export` |
|
||||
| Export fidelity report | `GET /api/companies/{companyId}/export/fidelity` |
|
||||
| Preview import (existing company) | `POST /api/companies/{companyId}/imports/preview` |
|
||||
| Apply import (existing company) | `POST /api/companies/{companyId}/imports/apply` |
|
||||
| Preview import (new company) | `POST /api/companies/import/preview` |
|
||||
| Apply import (new company) | `POST /api/companies/import` |
|
||||
|
||||
Import apply requests accept `pauseAutomations: true` to create imported agents and routines in a paused state.
|
||||
|
||||
CEO agents can also use the safe import routes (`/imports/preview` and `/imports/apply`) which enforce non-destructive rules: `replace` is rejected, collisions resolve with `rename` or `skip`, and issues are always created as new.
|
||||
|
||||
## GitHub Sources
|
||||
|
||||
@@ -173,9 +173,26 @@ This prevents silent completions where an agent finishes work but leaves no trac
|
||||
- Only the **active reviewer/approver** (the `currentParticipant` in execution state) can advance or reject the current stage.
|
||||
- Non-participants who attempt to transition the issue receive a `422 Unprocessable Entity` error.
|
||||
- Both approvals and change requests **require a comment** — empty or whitespace-only comments are rejected.
|
||||
- The decision comment must be included in the same `PATCH /api/issues/{issueId}` request that changes the status. A prior `POST /api/issues/{issueId}/comments` entry remains normal discussion and does not satisfy the review/approval decision guard.
|
||||
|
||||
## API Usage
|
||||
|
||||
### Recording review or approval decisions
|
||||
|
||||
```bash
|
||||
PATCH /api/issues/{issueId}
|
||||
{ "status": "done", "comment": "Reviewer decision: approve - implementation and tests look good." }
|
||||
```
|
||||
|
||||
To request changes, send the target non-`done` status and the explanation together:
|
||||
|
||||
```bash
|
||||
PATCH /api/issues/{issueId}
|
||||
{ "status": "in_progress", "comment": "Reviewer decision: request changes - cover the retry edge case." }
|
||||
```
|
||||
|
||||
Do not split a decision into `POST /comments` followed by a status-only `PATCH`; the runtime only records the decision from the `comment` field on the same status update.
|
||||
|
||||
### Setting an execution policy on issue creation
|
||||
|
||||
```bash
|
||||
|
||||
@@ -3,7 +3,7 @@ import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import type { AcpRuntimeOptions } from "acpx/runtime";
|
||||
import type { AdapterRuntimeMcpAccess } from "@paperclipai/adapter-utils";
|
||||
import type { AdapterExecutionContext, AdapterRuntimeMcpAccess } from "@paperclipai/adapter-utils";
|
||||
import {
|
||||
DEFAULT_REMOTE_SANDBOX_ADAPTER_TIMEOUT_SEC,
|
||||
prepareAdapterExecutionTargetRuntime,
|
||||
@@ -167,6 +167,7 @@ async function runExecutor(
|
||||
executionTarget?: Record<string, unknown>;
|
||||
runtimeMcp?: AdapterRuntimeMcpAccess;
|
||||
prepareRemoteManagedHome?: AcpxEngineExecutorOptions["prepareRemoteManagedHome"];
|
||||
startupTraceContext?: AdapterExecutionContext["startupTraceContext"];
|
||||
} = {},
|
||||
) {
|
||||
const runtimeOptions: Record<string, unknown>[] = [];
|
||||
@@ -201,6 +202,7 @@ async function runExecutor(
|
||||
authToken: options.authToken,
|
||||
executionTarget: options.executionTarget,
|
||||
runtimeMcp: options.runtimeMcp,
|
||||
startupTraceContext: options.startupTraceContext,
|
||||
onLog: async (stream: "stdout" | "stderr", text: string) => {
|
||||
logs.push({ stream, text });
|
||||
},
|
||||
@@ -216,6 +218,77 @@ async function runExecutor(
|
||||
return { logs, meta, events, runtimeOptions, configOptions, sessionInputs, result };
|
||||
}
|
||||
|
||||
// A recording span, used only in tests. It captures the span name, the parent
|
||||
// span (resolved from the explicit parent-context token), the attribute map,
|
||||
// the terminal status, and whether the span ended. The engine treats it purely
|
||||
// through the structural `StartupSpan` contract.
|
||||
interface RecordingSpan {
|
||||
name: string;
|
||||
attributes: Record<string, string | number | boolean>;
|
||||
parent: RecordingSpan | null;
|
||||
status: { code: number } | null;
|
||||
ended: boolean;
|
||||
setAttribute(key: string, value: string | number | boolean): void;
|
||||
setStatus(status: { code: number; message?: string }): void;
|
||||
end(): void;
|
||||
}
|
||||
|
||||
// Build an in-memory startup trace context that records every span. It models
|
||||
// the real OTel parenting contract: `startSpan(name, options, context)` reads
|
||||
// the parent from the explicit `context` token that `contextWithSpan` produced,
|
||||
// so a test asserts the exact parent of each child without an OTel package or
|
||||
// ambient async-context propagation.
|
||||
function createRecordingStartupTrace() {
|
||||
const spans: RecordingSpan[] = [];
|
||||
const traceContext = {
|
||||
tracer: {
|
||||
startSpan(
|
||||
name: string,
|
||||
options?: { attributes?: Record<string, string | number | boolean> },
|
||||
context?: unknown,
|
||||
) {
|
||||
const parent =
|
||||
context && typeof context === "object" && "span" in context
|
||||
? ((context as { span: RecordingSpan }).span ?? null)
|
||||
: null;
|
||||
const span: RecordingSpan = {
|
||||
name,
|
||||
attributes: { ...(options?.attributes ?? {}) },
|
||||
parent,
|
||||
status: null,
|
||||
ended: false,
|
||||
setAttribute(key: string, value: string | number | boolean) {
|
||||
span.attributes[key] = value;
|
||||
},
|
||||
setStatus(status: { code: number }) {
|
||||
span.status = { code: status.code };
|
||||
},
|
||||
end() {
|
||||
span.ended = true;
|
||||
},
|
||||
};
|
||||
spans.push(span);
|
||||
return span;
|
||||
},
|
||||
},
|
||||
contextWithSpan(span: unknown) {
|
||||
return { span };
|
||||
},
|
||||
} satisfies AdapterExecutionContext["startupTraceContext"];
|
||||
return { traceContext, spans };
|
||||
}
|
||||
|
||||
// The closed span-attribute allowlist for a sandbox-start span (Phase 2 + 3).
|
||||
// A test asserts every recorded attribute key is in this set, so a command,
|
||||
// path, id, or error-text key can never ride a span.
|
||||
const ALLOWED_STARTUP_SPAN_ATTRIBUTE_KEYS = new Set([
|
||||
"step",
|
||||
"provider",
|
||||
"roundTrips",
|
||||
"providerExecMs",
|
||||
"providerGetMs",
|
||||
]);
|
||||
|
||||
describe("shared ACPX engine runtime behavior", () => {
|
||||
it("sets Codex model, effort, and fast mode through CODEX_CONFIG without session config calls", async () => {
|
||||
const { configOptions, meta } = await runExecutor({
|
||||
@@ -798,6 +871,155 @@ describe("shared ACPX engine runtime behavior", () => {
|
||||
expect(fp(rotatedKey)).not.toBe(fp(withKey));
|
||||
});
|
||||
|
||||
it("busts the session fingerprint when the referenced-project set or pinned checkout changes", async () => {
|
||||
const root = await makeTempRoot();
|
||||
const stateDir = path.join(root, "state");
|
||||
const cwd = path.join(root, "workspace");
|
||||
const baseConfig = { agentCommand: "node ./fake-acp.js", stateDir };
|
||||
|
||||
// A compatible resume reuses the already-staged referenced-project trees, so
|
||||
// the fingerprint must fold in the referenced-project set. Otherwise a resume
|
||||
// reuses a stale tree after the set or a project's pinned checkout changes.
|
||||
const withReferenced = (additional: unknown[]) => ({
|
||||
context: {
|
||||
taskId: "issue-1",
|
||||
wakeReason: "issue_assigned",
|
||||
paperclipWorkspace: { cwd, realization: { additional } },
|
||||
},
|
||||
});
|
||||
const projectA = {
|
||||
path: "/host/project-a",
|
||||
projectId: "a",
|
||||
projectWorkspaceId: "ws-a",
|
||||
repoUrl: "https://example.test/a.git",
|
||||
repoRef: "ref-a-1",
|
||||
};
|
||||
const projectB = {
|
||||
path: "/host/project-b",
|
||||
projectId: "b",
|
||||
projectWorkspaceId: "ws-b",
|
||||
repoUrl: "https://example.test/b.git",
|
||||
repoRef: "ref-b-1",
|
||||
};
|
||||
|
||||
const one = await runExecutor(baseConfig, withReferenced([projectA]));
|
||||
const two = await runExecutor(baseConfig, withReferenced([projectA, projectB]));
|
||||
const reordered = await runExecutor(baseConfig, withReferenced([projectB, projectA]));
|
||||
const repointed = await runExecutor(
|
||||
baseConfig,
|
||||
withReferenced([projectA, { ...projectB, repoRef: "ref-b-2" }]),
|
||||
);
|
||||
|
||||
const fp = (r: { result: { sessionParams?: unknown } }) =>
|
||||
(r.result.sessionParams as { configFingerprint?: string } | undefined)?.configFingerprint;
|
||||
|
||||
// Adding a referenced project changes the set identity, so the fingerprint
|
||||
// busts and the next launch stages the current set.
|
||||
expect(fp(one)).toBeDefined();
|
||||
expect(fp(two)).not.toBe(fp(one));
|
||||
// The identity depends on the set, not the order the records arrive in.
|
||||
expect(fp(reordered)).toBe(fp(two));
|
||||
// Re-pinning one project's checkout ref busts the fingerprint, so the resume
|
||||
// re-stages instead of reusing a stale tree.
|
||||
expect(fp(repointed)).not.toBe(fp(two));
|
||||
});
|
||||
|
||||
it("busts the session fingerprint when referenced-project files change at the same host path", async () => {
|
||||
const root = await makeTempRoot();
|
||||
const stateDir = path.join(root, "state");
|
||||
const cwd = path.join(root, "workspace");
|
||||
const projectDir = path.join(root, "project-a");
|
||||
await fs.mkdir(path.join(projectDir, "src"), { recursive: true });
|
||||
await fs.writeFile(path.join(projectDir, "src", "index.ts"), "export const value = 1;\n");
|
||||
const baseConfig = { agentCommand: "node ./fake-acp.js", stateDir };
|
||||
|
||||
// Same host path and same metadata across runs. Only the file bytes change,
|
||||
// which mirrors a branch that moved or a re-checkout in place. The metadata
|
||||
// identity stays constant, so without the content signature a compatible
|
||||
// resume would reuse a stale staged tree.
|
||||
const referencedProject = {
|
||||
path: projectDir,
|
||||
projectId: "a",
|
||||
projectWorkspaceId: "ws-a",
|
||||
repoUrl: "https://example.test/a.git",
|
||||
repoRef: "main",
|
||||
};
|
||||
const withProject = () => ({
|
||||
context: {
|
||||
taskId: "issue-1",
|
||||
wakeReason: "issue_assigned",
|
||||
paperclipWorkspace: { cwd, realization: { additional: [referencedProject] } },
|
||||
},
|
||||
});
|
||||
const fp = (r: { result: { sessionParams?: unknown } }) =>
|
||||
(r.result.sessionParams as { configFingerprint?: string } | undefined)?.configFingerprint;
|
||||
|
||||
const before = await runExecutor(baseConfig, withProject());
|
||||
// Change the tree content and its size while the host path and metadata stay
|
||||
// identical.
|
||||
await fs.writeFile(path.join(projectDir, "src", "index.ts"), "export const value = 2; // changed\n");
|
||||
const afterEdit = await runExecutor(baseConfig, withProject());
|
||||
// Add a new file to the tree.
|
||||
await fs.writeFile(path.join(projectDir, "src", "extra.ts"), "export const extra = true;\n");
|
||||
const afterAdd = await runExecutor(baseConfig, withProject());
|
||||
|
||||
expect(fp(before)).toBeDefined();
|
||||
// A content-only edit busts the fingerprint, so the resume re-stages.
|
||||
expect(fp(afterEdit)).not.toBe(fp(before));
|
||||
// A new file in the tree busts the fingerprint too.
|
||||
expect(fp(afterAdd)).not.toBe(fp(afterEdit));
|
||||
});
|
||||
|
||||
it("busts the session fingerprint when referenced-project bytes change but size and mtime stay equal", async () => {
|
||||
const root = await makeTempRoot();
|
||||
const stateDir = path.join(root, "state");
|
||||
const cwd = path.join(root, "workspace");
|
||||
const projectDir = path.join(root, "project-a");
|
||||
const filePath = path.join(projectDir, "src", "index.ts");
|
||||
await fs.mkdir(path.join(projectDir, "src"), { recursive: true });
|
||||
// The two contents have the same byte length, so a stat-only signature (size
|
||||
// and modification time) would collide once the modification time also matches.
|
||||
const firstContent = "export const value = 1;\n";
|
||||
const secondContent = "export const value = 2;\n";
|
||||
expect(secondContent.length).toBe(firstContent.length);
|
||||
await fs.writeFile(filePath, firstContent);
|
||||
|
||||
// A re-checkout can restore the same modification time, so pin it explicitly
|
||||
// and reapply it after the edit. Only the file bytes change between runs.
|
||||
const pinnedTime = new Date("2026-01-01T00:00:00.000Z");
|
||||
await fs.utimes(filePath, pinnedTime, pinnedTime);
|
||||
|
||||
const referencedProject = {
|
||||
path: projectDir,
|
||||
projectId: "a",
|
||||
projectWorkspaceId: "ws-a",
|
||||
repoUrl: "https://example.test/a.git",
|
||||
repoRef: "main",
|
||||
};
|
||||
const withProject = () => ({
|
||||
context: {
|
||||
taskId: "issue-1",
|
||||
wakeReason: "issue_assigned",
|
||||
paperclipWorkspace: { cwd, realization: { additional: [referencedProject] } },
|
||||
},
|
||||
});
|
||||
const fp = (r: { result: { sessionParams?: unknown } }) =>
|
||||
(r.result.sessionParams as { configFingerprint?: string } | undefined)?.configFingerprint;
|
||||
|
||||
const baseConfig = { agentCommand: "node ./fake-acp.js", stateDir };
|
||||
const before = await runExecutor(baseConfig, withProject());
|
||||
// Replace the bytes with an equal-length string and restore the same
|
||||
// modification time, so path, size, and mtime all stay identical across the two runs.
|
||||
await fs.writeFile(filePath, secondContent);
|
||||
await fs.utimes(filePath, pinnedTime, pinnedTime);
|
||||
const after = await runExecutor(baseConfig, withProject());
|
||||
|
||||
expect(fp(before)).toBeDefined();
|
||||
// The content signature reads bytes, so an equal-size, equal-mtime edit busts
|
||||
// the fingerprint and the resume re-stages instead of reusing a stale tree.
|
||||
expect(fp(after)).not.toBe(fp(before));
|
||||
});
|
||||
|
||||
it("shapes ACPX session env for remote execution identities", async () => {
|
||||
const root = await makeTempRoot();
|
||||
const localCwd = path.join(root, "local");
|
||||
@@ -2780,6 +3002,226 @@ describe("ACPX engine remote session-lifecycle re-staging (PR 3: stage once / re
|
||||
});
|
||||
});
|
||||
|
||||
describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
async function remoteSandboxTarget(root: string) {
|
||||
const remoteCwd = path.join(root, "remote-workspace");
|
||||
await fs.mkdir(remoteCwd, { recursive: true });
|
||||
return {
|
||||
kind: "remote",
|
||||
transport: "sandbox",
|
||||
// A plugin-backed key (not a built-in family). It must never ride a span
|
||||
// as a raw attribute.
|
||||
providerKey: "fake-plugin",
|
||||
remoteCwd,
|
||||
runner: createLocalSandboxRunner(),
|
||||
};
|
||||
}
|
||||
|
||||
it("emits one root span with a child span per bring-up boundary, each parented to the root", async () => {
|
||||
const root = await makeTempRoot();
|
||||
const stateDir = path.join(root, "state");
|
||||
const localCwd = path.join(root, "worktree");
|
||||
const codexHome = path.join(root, "codex-home");
|
||||
await fs.mkdir(localCwd, { recursive: true });
|
||||
await fs.mkdir(codexHome, { recursive: true });
|
||||
const executionTarget = await remoteSandboxTarget(root);
|
||||
const { traceContext, spans } = createRecordingStartupTrace();
|
||||
|
||||
await runExecutor(
|
||||
{
|
||||
agent: "codex",
|
||||
agentCommand: "node ./fake-acp.js",
|
||||
stateDir,
|
||||
cwd: localCwd,
|
||||
env: { CODEX_HOME: codexHome },
|
||||
},
|
||||
{ authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext },
|
||||
);
|
||||
|
||||
// Exactly one root span, and it is the bring-up root.
|
||||
const roots = spans.filter((span) => span.parent === null);
|
||||
expect(roots).toHaveLength(1);
|
||||
const rootSpan = roots[0]!;
|
||||
expect(rootSpan.name).toBe("sandbox.startup");
|
||||
expect(rootSpan.ended).toBe(true);
|
||||
|
||||
// A codex bring-up over the remote sandbox lane crosses all 7 boundaries.
|
||||
const childNames = spans.filter((span) => span !== rootSpan).map((span) => span.name).sort();
|
||||
expect(childNames).toEqual(
|
||||
[
|
||||
"acp.handshake",
|
||||
"bridge.paperclip",
|
||||
"bridge.process-session",
|
||||
"codex-home.seed",
|
||||
"skills.reconcile",
|
||||
"stage.sync",
|
||||
"workspace.resolve",
|
||||
],
|
||||
);
|
||||
|
||||
// Every child parents to the one root and ends.
|
||||
for (const span of spans) {
|
||||
if (span === rootSpan) continue;
|
||||
expect(span.parent, `span "${span.name}" must parent to the root`).toBe(rootSpan);
|
||||
expect(span.ended, `span "${span.name}" must end`).toBe(true);
|
||||
}
|
||||
});
|
||||
|
||||
it("parents both concurrent bridge spans to the root (neither orphans)", async () => {
|
||||
const root = await makeTempRoot();
|
||||
const stateDir = path.join(root, "state");
|
||||
const localCwd = path.join(root, "worktree");
|
||||
await fs.mkdir(localCwd, { recursive: true });
|
||||
const executionTarget = await remoteSandboxTarget(root);
|
||||
const { traceContext, spans } = createRecordingStartupTrace();
|
||||
|
||||
await runExecutor(
|
||||
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
|
||||
{ authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext },
|
||||
);
|
||||
|
||||
const rootSpan = spans.find((span) => span.name === "sandbox.startup" && span.parent === null);
|
||||
expect(rootSpan).toBeTruthy();
|
||||
const paperclip = spans.find((span) => span.name === "bridge.paperclip");
|
||||
const processSession = spans.find((span) => span.name === "bridge.process-session");
|
||||
expect(paperclip?.parent).toBe(rootSpan);
|
||||
expect(processSession?.parent).toBe(rootSpan);
|
||||
});
|
||||
|
||||
it("keeps every span attribute inside the closed allowlist (no command/path/id keys)", async () => {
|
||||
const root = await makeTempRoot();
|
||||
const stateDir = path.join(root, "state");
|
||||
const localCwd = path.join(root, "worktree");
|
||||
await fs.mkdir(localCwd, { recursive: true });
|
||||
const executionTarget = await remoteSandboxTarget(root);
|
||||
const { traceContext, spans } = createRecordingStartupTrace();
|
||||
|
||||
await runExecutor(
|
||||
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
|
||||
{ authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext },
|
||||
);
|
||||
|
||||
expect(spans.length).toBeGreaterThan(0);
|
||||
for (const span of spans) {
|
||||
for (const [key, value] of Object.entries(span.attributes)) {
|
||||
expect(
|
||||
ALLOWED_STARTUP_SPAN_ATTRIBUTE_KEYS.has(key),
|
||||
`span "${span.name}" set a non-allowlisted attribute "${key}"`,
|
||||
).toBe(true);
|
||||
// No non-finite numeric attribute (no NaN, no Infinity).
|
||||
if (typeof value === "number") {
|
||||
expect(Number.isFinite(value), `attribute "${key}" must be finite`).toBe(true);
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
it("closes the root span with error status when the bring-up handshake fails", async () => {
|
||||
const root = await makeTempRoot();
|
||||
const stateDir = path.join(root, "state");
|
||||
const localCwd = path.join(root, "worktree");
|
||||
await fs.mkdir(localCwd, { recursive: true });
|
||||
const executionTarget = await remoteSandboxTarget(root);
|
||||
const { traceContext, spans } = createRecordingStartupTrace();
|
||||
|
||||
const execute = createAcpxEngineExecutor({
|
||||
createRuntime: () =>
|
||||
({
|
||||
ensureSession: async () => {
|
||||
throw new Error("handshake boom");
|
||||
},
|
||||
startTurn: () => ({
|
||||
events: (async function* () {})(),
|
||||
result: Promise.resolve({ status: "failed" }),
|
||||
cancel: async () => {},
|
||||
}),
|
||||
setConfigOption: async () => {},
|
||||
close: async () => {},
|
||||
}) as never,
|
||||
});
|
||||
|
||||
const result = await execute({
|
||||
runId: "run-handshake-fail",
|
||||
agent: { id: "agent-1", companyId: "company-1" },
|
||||
runtime: {},
|
||||
config: { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
|
||||
context: {},
|
||||
authToken: "real-run-jwt",
|
||||
executionTarget,
|
||||
startupTraceContext: traceContext,
|
||||
onLog: async () => {},
|
||||
onMeta: async () => {},
|
||||
onEvent: async () => {},
|
||||
} as never);
|
||||
|
||||
expect(result.exitCode).toBe(1);
|
||||
const rootSpan = spans.find((span) => span.name === "sandbox.startup" && span.parent === null);
|
||||
expect(rootSpan).toBeTruthy();
|
||||
expect(rootSpan!.ended).toBe(true);
|
||||
// `2` is `SpanStatusCode.ERROR`.
|
||||
expect(rootSpan!.status?.code).toBe(2);
|
||||
});
|
||||
|
||||
it("opens no span and does not throw when no trace context is injected", async () => {
|
||||
const root = await makeTempRoot();
|
||||
const stateDir = path.join(root, "state");
|
||||
const localCwd = path.join(root, "worktree");
|
||||
await fs.mkdir(localCwd, { recursive: true });
|
||||
const executionTarget = await remoteSandboxTarget(root);
|
||||
|
||||
// runExecutor asserts exitCode 0 internally; the run must complete with no
|
||||
// injected trace context (the default no-op path).
|
||||
const { result } = await runExecutor(
|
||||
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
|
||||
{ authToken: "real-run-jwt", executionTarget },
|
||||
);
|
||||
expect(result.exitCode).toBe(0);
|
||||
});
|
||||
|
||||
it("opens no span for a local run, even when a trace context is injected", async () => {
|
||||
const root = await makeTempRoot();
|
||||
const stateDir = path.join(root, "state");
|
||||
const localCwd = path.join(root, "worktree");
|
||||
await fs.mkdir(localCwd, { recursive: true });
|
||||
const { traceContext, spans } = createRecordingStartupTrace();
|
||||
|
||||
// A local run has no execution target. The `sandbox.startup` span names a
|
||||
// sandbox bring-up, so a local run must stay out of sandbox telemetry.
|
||||
const { result } = await runExecutor(
|
||||
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
|
||||
{ authToken: "real-run-jwt", startupTraceContext: traceContext },
|
||||
);
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(spans).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("opens no span for an SSH run, even when a trace context is injected", async () => {
|
||||
const root = await makeTempRoot();
|
||||
const stateDir = path.join(root, "state");
|
||||
const localCwd = path.join(root, "worktree");
|
||||
const remoteCwd = path.join(root, "remote-workspace");
|
||||
await fs.mkdir(localCwd, { recursive: true });
|
||||
const { traceContext, spans } = createRecordingStartupTrace();
|
||||
|
||||
// An SSH run is remote but is not a sandbox, so it also stays out of sandbox
|
||||
// telemetry.
|
||||
const { result } = await runExecutor(
|
||||
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
|
||||
{
|
||||
authToken: "real-run-jwt",
|
||||
executionTarget: { kind: "remote", transport: "ssh", remoteCwd },
|
||||
startupTraceContext: traceContext,
|
||||
},
|
||||
);
|
||||
expect(result.exitCode).toBe(0);
|
||||
expect(spans).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ACPX engine per-step startup timing (run.startup.step events)", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
|
||||
@@ -28,6 +28,7 @@ import {
|
||||
type AdapterExecutionTargetTimeoutResolution,
|
||||
type AdapterManagedRuntimeAsset,
|
||||
type PreparedAdapterExecutionTargetRuntime,
|
||||
type SandboxAdditionalSource,
|
||||
} from "@paperclipai/adapter-utils/execution-target";
|
||||
import {
|
||||
DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE,
|
||||
@@ -82,7 +83,15 @@ import {
|
||||
DEFAULT_ACP_ENGINE_TIMEOUT_SEC,
|
||||
DEFAULT_ACP_ENGINE_WARM_HANDLE_IDLE_MS,
|
||||
} from "./constants.js";
|
||||
import { measureStartupStep, type StartupStepMeasureOptions } from "./startup-timing.js";
|
||||
import {
|
||||
measureStartupStep,
|
||||
NOOP_STARTUP_SPAN,
|
||||
NOOP_STARTUP_TRACE_CONTEXT,
|
||||
type StartupSpan,
|
||||
type StartupSpanContext,
|
||||
type StartupStepMeasureOptions,
|
||||
type StartupTraceContext,
|
||||
} from "./startup-timing.js";
|
||||
import type { CommandManagedRuntimeRunner } from "../command-managed-runtime.js";
|
||||
|
||||
const defaultModuleDir = path.dirname(fileURLToPath(import.meta.url));
|
||||
@@ -418,6 +427,83 @@ function shortHash(value: unknown): string {
|
||||
return createHash("sha256").update(stableJson(value)).digest("hex").slice(0, 16);
|
||||
}
|
||||
|
||||
// Directory names the staging path never ships for a referenced project (heavy
|
||||
// build/cache output and git history). The content signature skips them so it
|
||||
// reflects only the staged tree and never reads their bytes. Keep this set equal
|
||||
// to the staging excludes in the sandbox and remote runtimes.
|
||||
const REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS = new Set([
|
||||
"node_modules",
|
||||
"vendor",
|
||||
"dist",
|
||||
"build",
|
||||
"out",
|
||||
"coverage",
|
||||
".next",
|
||||
".turbo",
|
||||
".cache",
|
||||
".git",
|
||||
]);
|
||||
|
||||
/**
|
||||
* Content signature of a referenced-project host tree for the session fingerprint.
|
||||
*
|
||||
* The staged-runtime cache reuses an already-staged referenced-project tree on a
|
||||
* compatible resume and does not re-sync it. Referenced-project metadata (id, host
|
||||
* path, workspace id, repo url, pinned ref) can stay identical while the files at
|
||||
* that host path change: a branch moved to a new commit, a re-checkout in place, or
|
||||
* a dirty worktree. So the metadata identity alone lets a resume serve a stale tree.
|
||||
* This signature folds the tree's own content state into the identity.
|
||||
*
|
||||
* The walk reads each file's relative path and bytes and folds them into the hash.
|
||||
* It reads bytes, not only file stats. A stat-only signature (size and modification
|
||||
* time) collides when an edit keeps the byte length and the modification time — a
|
||||
* re-checkout that restores the same size and timestamp. The byte hash busts on any
|
||||
* content change, so the fingerprint busts and the next launch stages the current
|
||||
* tree. The walk skips the heavy build, cache, and git directories the staging path
|
||||
* never ships, and records a symlink by its target text without following it. On a
|
||||
* read error the function returns a stable marker, so the fingerprint does not churn
|
||||
* while staging surfaces the real error. The walk runs only when the run carries
|
||||
* referenced projects (the multi-project sync path).
|
||||
*/
|
||||
async function referencedSourceContentSignature(localPath: string): Promise<string> {
|
||||
const hash = createHash("sha256");
|
||||
const walk = async (relative: string): Promise<void> => {
|
||||
const current = relative ? path.join(localPath, relative) : localPath;
|
||||
const dirents = await fs.readdir(current, { withFileTypes: true });
|
||||
dirents.sort((left, right) => (left.name < right.name ? -1 : left.name > right.name ? 1 : 0));
|
||||
for (const dirent of dirents) {
|
||||
const next = relative ? path.posix.join(relative, dirent.name) : dirent.name;
|
||||
if (dirent.isDirectory()) {
|
||||
if (REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS.has(dirent.name)) {
|
||||
continue;
|
||||
}
|
||||
await walk(next);
|
||||
continue;
|
||||
}
|
||||
const absolute = path.join(localPath, next);
|
||||
const stats = await fs.lstat(absolute);
|
||||
if (stats.isSymbolicLink()) {
|
||||
const target = await fs.readlink(absolute);
|
||||
hash.update(`symlink:${next}:${target}\n`);
|
||||
continue;
|
||||
}
|
||||
if (!stats.isFile()) {
|
||||
hash.update(`other:${next}:${stats.mode}\n`);
|
||||
continue;
|
||||
}
|
||||
hash.update(`file:${next}:${stats.size}\n`);
|
||||
hash.update(await fs.readFile(absolute));
|
||||
hash.update("\n");
|
||||
}
|
||||
};
|
||||
try {
|
||||
await walk("");
|
||||
} catch (error) {
|
||||
return `unreadable:${String(error)}`;
|
||||
}
|
||||
return hash.digest("hex").slice(0, 16);
|
||||
}
|
||||
|
||||
function defaultPaperclipInstanceDir(): string {
|
||||
const home = process.env.PAPERCLIP_HOME?.trim() || path.join(os.homedir(), ".paperclip");
|
||||
const instanceId = process.env.PAPERCLIP_INSTANCE_ID?.trim() || "default";
|
||||
@@ -1209,6 +1295,11 @@ async function stageAcpRemoteRuntime(input: {
|
||||
workspaceRemoteDir?: string;
|
||||
timeoutSec: number;
|
||||
assets?: AdapterManagedRuntimeAsset[];
|
||||
// Referenced (additional) projects to stage into the sandbox as plain,
|
||||
// read-only trees alongside the anchor workspace. Empty unless run prep
|
||||
// resolved referenced projects (gated upstream), so the anchor-only path is
|
||||
// unchanged.
|
||||
additionalSources?: SandboxAdditionalSource[];
|
||||
onLog: AdapterExecutionContext["onLog"];
|
||||
onRuntimeProgress: AdapterExecutionContext["onRuntimeProgress"];
|
||||
}): Promise<PreparedAdapterExecutionTargetRuntime> {
|
||||
@@ -1224,6 +1315,9 @@ async function stageAcpRemoteRuntime(input: {
|
||||
workspaceLocalDir: input.workspaceLocalDir,
|
||||
...(input.workspaceRemoteDir ? { workspaceRemoteDir: input.workspaceRemoteDir } : {}),
|
||||
...(input.assets && input.assets.length > 0 ? { assets: input.assets } : {}),
|
||||
...(input.additionalSources && input.additionalSources.length > 0
|
||||
? { additionalSources: input.additionalSources }
|
||||
: {}),
|
||||
onProgress: (line) => input.onLog("stdout", line),
|
||||
onRuntimeProgress: input.onRuntimeProgress,
|
||||
});
|
||||
@@ -1249,6 +1343,10 @@ async function buildRuntime(input: {
|
||||
ctx: AdapterExecutionContext;
|
||||
engine: AcpxEngineSettings;
|
||||
deps: AcpxEngineExecutorOptions;
|
||||
// The injected tracer plus the root-span parent-context token. Merged into
|
||||
// every startup-step option set, so each boundary span parents to the one
|
||||
// root span (`sandbox.startup`) that the executor opens.
|
||||
spanParent: Pick<StartupStepMeasureOptions, "tracer" | "parentContext">;
|
||||
}): Promise<AcpxPreparedRuntime> {
|
||||
const { runId, agent, config, context, authToken } = input.ctx;
|
||||
// Injectable monotonic clock for per-step startup timing. Hoisted above the
|
||||
@@ -1271,6 +1369,43 @@ async function buildRuntime(input: {
|
||||
const useConfiguredInsteadOfAgentHome = workspaceSource === "agent_home" && configuredCwd.length > 0;
|
||||
const effectiveWorkspaceCwd = useConfiguredInsteadOfAgentHome ? "" : workspaceCwd;
|
||||
const cwd = effectiveWorkspaceCwd || configuredCwd || process.cwd();
|
||||
// Referenced (additional) projects to stage into the sandbox alongside the
|
||||
// anchor workspace, read from the workspace realization record. The list is
|
||||
// empty unless run prep resolved referenced projects — gated upstream by the
|
||||
// multi-project workspace-sync kill-switch — so the anchor-only path is
|
||||
// unchanged.
|
||||
const realizationContext = parseObject(workspaceContext.realization);
|
||||
const additionalSourceRecords = (
|
||||
Array.isArray(realizationContext.additional) ? realizationContext.additional : []
|
||||
).map((entry) => parseObject(entry));
|
||||
const additionalSources: SandboxAdditionalSource[] = additionalSourceRecords
|
||||
.map((entry) => ({ localPath: asString(entry.path, ""), projectId: asString(entry.projectId, "") }))
|
||||
.filter((entry) => entry.localPath.length > 0 && entry.projectId.length > 0);
|
||||
// Stable identity of the referenced-project set for the session fingerprint.
|
||||
// The staged-runtime cache reuses already-staged referenced-project trees on a
|
||||
// compatible resume, so the fingerprint must change when the set OR a project's
|
||||
// pinned checkout changes. Without this, a resume reuses a stale staged tree.
|
||||
// Fold in each project's id, host path, workspace id, and pinned ref; sort by
|
||||
// projectId so the identity depends on the set, not the record order.
|
||||
const additionalSourcesIdentityBase = additionalSourceRecords
|
||||
.map((entry) => ({
|
||||
projectId: asString(entry.projectId, ""),
|
||||
localPath: asString(entry.path, ""),
|
||||
projectWorkspaceId: asString(entry.projectWorkspaceId, ""),
|
||||
repoUrl: asString(entry.repoUrl, ""),
|
||||
repoRef: asString(entry.repoRef, ""),
|
||||
}))
|
||||
.filter((entry) => entry.localPath.length > 0 && entry.projectId.length > 0)
|
||||
.sort((a, b) => (a.projectId < b.projectId ? -1 : a.projectId > b.projectId ? 1 : 0));
|
||||
// Metadata alone does not change on a content-only checkout change (same host
|
||||
// path and pinned ref, new file bytes). Fold in each tree's content signature so
|
||||
// a file add, remove, or edit busts the fingerprint and the resume re-stages.
|
||||
const additionalSourcesIdentity = await Promise.all(
|
||||
additionalSourcesIdentityBase.map(async (entry) => ({
|
||||
...entry,
|
||||
contentSignature: await referencedSourceContentSignature(entry.localPath),
|
||||
})),
|
||||
);
|
||||
const executionTarget = readAdapterExecutionTarget({
|
||||
executionTarget: input.ctx.executionTarget,
|
||||
legacyRemoteExecution: input.ctx.executionTransport?.remoteExecution,
|
||||
@@ -1287,17 +1422,23 @@ async function buildRuntime(input: {
|
||||
// and emits the per-step delta. Empty when there is no runner (local runs,
|
||||
// the runner-less ACP→CLI fallback, or an SSH runner that does not
|
||||
// instrument the seam), so those steps simply omit the fields.
|
||||
const stepMetrics = buildStartupStepMetrics(
|
||||
executionTarget?.kind === "remote" && executionTarget.transport === "sandbox"
|
||||
? executionTarget.runner
|
||||
: undefined,
|
||||
);
|
||||
// Merge the injected tracer + root parent-context into every step option set,
|
||||
// so each boundary span parents to the root span. With no injected trace
|
||||
// context both fields are no-ops and the span path stays inert.
|
||||
const stepMetrics: StartupStepMeasureOptions = {
|
||||
...buildStartupStepMetrics(
|
||||
executionTarget?.kind === "remote" && executionTarget.transport === "sandbox"
|
||||
? executionTarget.runner
|
||||
: undefined,
|
||||
),
|
||||
...input.spanParent,
|
||||
};
|
||||
// The two bridge-start steps intentionally overlap, so their runner counters
|
||||
// would double-count each other if we sampled them here. Keep the shared
|
||||
// counter attribution on the sequential startup phases only; the concurrent
|
||||
// bridge steps still emit duration telemetry, just not misleading per-step
|
||||
// round-trip/provider deltas.
|
||||
const concurrentBridgeStepMetrics: StartupStepMeasureOptions = {};
|
||||
// bridge steps still emit duration telemetry (and a span), just not
|
||||
// misleading per-step round-trip/provider deltas.
|
||||
const concurrentBridgeStepMetrics: StartupStepMeasureOptions = { ...input.spanParent };
|
||||
const shapedWorkspaceEnv = shapePaperclipWorkspaceEnvForExecution({
|
||||
workspaceCwd: effectiveWorkspaceCwd,
|
||||
workspaceWorktreePath,
|
||||
@@ -1553,6 +1694,11 @@ async function buildRuntime(input: {
|
||||
requestedThinkingEffort,
|
||||
fastMode,
|
||||
remoteExecutionIdentity,
|
||||
// Referenced-project set + pinned-checkout identity. A change here (a project
|
||||
// added, removed, or re-pinned) invalidates a warm/resumable session so the
|
||||
// next launch stages the current referenced-project trees instead of reusing
|
||||
// a stale staged tree.
|
||||
additionalSourcesIdentity,
|
||||
skillsIdentity,
|
||||
skillPromptInstructions,
|
||||
paperclipClaudeSettings: paperclipClaudeSettings
|
||||
@@ -1681,6 +1827,7 @@ async function buildRuntime(input: {
|
||||
workspaceRemoteDir: sessionCwd,
|
||||
timeoutSec,
|
||||
assets,
|
||||
additionalSources,
|
||||
onLog: input.ctx.onLog,
|
||||
onRuntimeProgress: input.ctx.onRuntimeProgress,
|
||||
});
|
||||
@@ -2658,6 +2805,54 @@ function warmHandleMatches(
|
||||
return entry !== undefined && entry.runtime === runtime && entry.handle === handle;
|
||||
}
|
||||
|
||||
/** The stable name of the one root span for a sandbox bring-up. It is a fixed
|
||||
* low-cardinality constant, never derived from run/user data. */
|
||||
const STARTUP_ROOT_SPAN_NAME = "sandbox.startup";
|
||||
|
||||
/**
|
||||
* Open the one root span for a sandbox bring-up and return its parent-context
|
||||
* token plus a guarded `end`. The span parents every startup boundary span:
|
||||
* the engine forwards `parentContext` to each `measureStartupStep` call. The
|
||||
* `end` closure runs at most once (bring-up complete OR a bring-up failure) and
|
||||
* swallows every tracer error, so observability never changes startup control
|
||||
* flow. With no injected trace context, the tracer is a no-op and the span is
|
||||
* a no-op.
|
||||
*/
|
||||
function openStartupRootSpan(tracing: StartupTraceContext): {
|
||||
parentContext: StartupSpanContext;
|
||||
end: (failed: boolean) => void;
|
||||
} {
|
||||
let span: StartupSpan;
|
||||
try {
|
||||
span = tracing.tracer.startSpan(STARTUP_ROOT_SPAN_NAME);
|
||||
} catch {
|
||||
span = NOOP_STARTUP_SPAN;
|
||||
}
|
||||
let parentContext: StartupSpanContext;
|
||||
try {
|
||||
parentContext = tracing.contextWithSpan(span);
|
||||
} catch {
|
||||
parentContext = undefined;
|
||||
}
|
||||
let ended = false;
|
||||
return {
|
||||
parentContext,
|
||||
end: (failed: boolean) => {
|
||||
if (ended) return;
|
||||
ended = true;
|
||||
try {
|
||||
// `2` is `SpanStatusCode.ERROR`. `adapter-utils` stays OTel-free, so it
|
||||
// uses the numeric value that a real injected span reads as the error
|
||||
// status.
|
||||
if (failed) span.setStatus({ code: 2 });
|
||||
span.end();
|
||||
} catch {
|
||||
// Observability must not change startup control flow.
|
||||
}
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
|
||||
const createRuntime = deps.createRuntime ?? createAcpRuntime;
|
||||
const now = deps.now ?? (() => Date.now());
|
||||
@@ -2688,7 +2883,39 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
|
||||
now,
|
||||
idleMs: warmIdleMs,
|
||||
});
|
||||
const prepared = await buildRuntime({ ctx, engine, deps });
|
||||
// The `sandbox.startup` span names a sandbox bring-up. It must not cover a
|
||||
// local or SSH run: those runs have no sandbox, so they stay out of sandbox
|
||||
// telemetry. Open the real root span only when the target is a remote
|
||||
// sandbox; every other target forces the no-op trace context, so the whole
|
||||
// startup span path stays inert regardless of the injected context.
|
||||
const startupExecutionTarget = readAdapterExecutionTarget({
|
||||
executionTarget: ctx.executionTarget,
|
||||
legacyRemoteExecution: ctx.executionTransport?.remoteExecution,
|
||||
});
|
||||
const targetsRemoteSandbox =
|
||||
startupExecutionTarget?.kind === "remote" && startupExecutionTarget.transport === "sandbox";
|
||||
// Open the one root span for this bring-up. It spans `buildRuntime` through
|
||||
// `acp.handshake`, so every startup boundary span parents to it. `spanParent`
|
||||
// carries the injected tracer + the root parent-context token into each
|
||||
// `measureStartupStep` call. With no injected trace context the whole path
|
||||
// is a no-op. `endRootSpan` runs exactly once — at bring-up completion or on
|
||||
// a bring-up failure.
|
||||
const tracing =
|
||||
targetsRemoteSandbox && ctx.startupTraceContext
|
||||
? ctx.startupTraceContext
|
||||
: NOOP_STARTUP_TRACE_CONTEXT;
|
||||
const rootSpan = openStartupRootSpan(tracing);
|
||||
const spanParent: Pick<StartupStepMeasureOptions, "tracer" | "parentContext"> = {
|
||||
tracer: tracing.tracer,
|
||||
parentContext: rootSpan.parentContext,
|
||||
};
|
||||
let prepared: AcpxPreparedRuntime;
|
||||
try {
|
||||
prepared = await buildRuntime({ ctx, engine, deps, spanParent });
|
||||
} catch (err) {
|
||||
rootSpan.end(true);
|
||||
throw err;
|
||||
}
|
||||
// State the effective wall-clock timeout and its source up front so a
|
||||
// later timeout is diagnosable from the run log alone. Goes to stderr:
|
||||
// the acpx stdout log stream carries JSON acpx.* event payloads and must
|
||||
@@ -2816,6 +3043,8 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
// Bring-up failed at the handshake — close the root span with error status.
|
||||
rootSpan.end(true);
|
||||
const { classified, message } = await emitAcpxFailure({
|
||||
ctx,
|
||||
prepared,
|
||||
@@ -2839,6 +3068,8 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
|
||||
}
|
||||
|
||||
if (!handle) {
|
||||
// Bring-up produced no session handle — close the root span with error status.
|
||||
rootSpan.end(true);
|
||||
await discardStagedRuntime({ handles: stagedRuntimes, prepared });
|
||||
await cleanupRemoteBridges(prepared);
|
||||
return {
|
||||
@@ -2853,6 +3084,10 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
|
||||
summary: "ACPX did not return a runtime session handle.",
|
||||
};
|
||||
}
|
||||
// Bring-up is complete: the session handle is established. Close the root
|
||||
// span here, so it covers `buildRuntime` through `acp.handshake` and no
|
||||
// further. The agent turn runs after and is out of the startup root's scope.
|
||||
rootSpan.end(false);
|
||||
const sessionHandle = handle;
|
||||
try {
|
||||
await applySessionConfigOptions({
|
||||
|
||||
@@ -1,6 +1,49 @@
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import type { AdapterRuntimeEvent } from "../types.js";
|
||||
import { measureStartupStep } from "./startup-timing.js";
|
||||
import type { StartupSpan, StartupTracer } from "./startup-timing.js";
|
||||
import { measureStartupStep, normalizeProviderFamily } from "./startup-timing.js";
|
||||
|
||||
/**
|
||||
* A recording span for the mock tracer. It captures the attribute set, the
|
||||
* status, and the end count so a test can assert the emitted span shape.
|
||||
*/
|
||||
class MockSpan implements StartupSpan {
|
||||
readonly attributes: Record<string, string | number | boolean> = {};
|
||||
status: { code: number; message?: string } | undefined;
|
||||
endCount = 0;
|
||||
|
||||
constructor(
|
||||
readonly name: string,
|
||||
initial: Record<string, string | number | boolean> | undefined,
|
||||
) {
|
||||
if (initial) Object.assign(this.attributes, initial);
|
||||
}
|
||||
|
||||
setAttribute(key: string, value: string | number | boolean): void {
|
||||
this.attributes[key] = value;
|
||||
}
|
||||
|
||||
setStatus(status: { code: number; message?: string }): void {
|
||||
this.status = status;
|
||||
}
|
||||
|
||||
end(): void {
|
||||
this.endCount += 1;
|
||||
}
|
||||
}
|
||||
|
||||
/** A recording tracer that keeps every span it opens for assertions. */
|
||||
function makeMockTracer(): { tracer: StartupTracer; spans: MockSpan[] } {
|
||||
const spans: MockSpan[] = [];
|
||||
const tracer: StartupTracer = {
|
||||
startSpan(name, options) {
|
||||
const span = new MockSpan(name, options?.attributes);
|
||||
spans.push(span);
|
||||
return span;
|
||||
},
|
||||
};
|
||||
return { tracer, spans };
|
||||
}
|
||||
|
||||
describe("measureStartupStep", () => {
|
||||
it("emits one run.startup.step event with the step name and measured durationMs", async () => {
|
||||
@@ -193,4 +236,165 @@ describe("measureStartupStep", () => {
|
||||
}),
|
||||
).rejects.toBe(boom);
|
||||
});
|
||||
|
||||
it("opens one span and ends it once for a normal step", async () => {
|
||||
const { tracer, spans } = makeMockTracer();
|
||||
const onEvent = vi.fn(async () => {});
|
||||
|
||||
await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", {
|
||||
tracer,
|
||||
});
|
||||
|
||||
expect(spans).toHaveLength(1);
|
||||
expect(spans[0]!.name).toBe("stage.sync");
|
||||
expect(spans[0]!.attributes.step).toBe("stage.sync");
|
||||
expect(spans[0]!.endCount).toBe(1);
|
||||
});
|
||||
|
||||
it("ends the span and sets an error status when fn throws, then re-throws", async () => {
|
||||
const { tracer, spans } = makeMockTracer();
|
||||
const onEvent = vi.fn(async () => {});
|
||||
const boom = new Error("step failed");
|
||||
|
||||
await expect(
|
||||
measureStartupStep({ onEvent }, () => 0, "acp.handshake", async () => {
|
||||
throw boom;
|
||||
}, { tracer }),
|
||||
).rejects.toBe(boom);
|
||||
|
||||
expect(spans).toHaveLength(1);
|
||||
expect(spans[0]!.endCount).toBe(1);
|
||||
// SpanStatusCode.ERROR === 2 in @opentelemetry/api.
|
||||
expect(spans[0]!.status?.code).toBe(2);
|
||||
});
|
||||
|
||||
it("sets the same roundTrips / providerExecMs / providerGetMs deltas on the payload and the span", async () => {
|
||||
let t = 0;
|
||||
const now = () => t;
|
||||
let execCount = 5;
|
||||
let execMs = 100;
|
||||
let getMs = 40;
|
||||
const events: AdapterRuntimeEvent[] = [];
|
||||
const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => {
|
||||
events.push(event);
|
||||
});
|
||||
const { tracer, spans } = makeMockTracer();
|
||||
|
||||
await measureStartupStep({ onEvent }, now, "stage.sync", async () => {
|
||||
t = 90;
|
||||
execCount += 3;
|
||||
execMs += 600;
|
||||
getMs += 15;
|
||||
return "ok";
|
||||
}, {
|
||||
tracer,
|
||||
roundTrips: () => execCount,
|
||||
providerExecMs: () => execMs,
|
||||
providerGetMs: () => getMs,
|
||||
});
|
||||
|
||||
const payload = events[0]!.payload as Record<string, unknown>;
|
||||
expect(payload.roundTrips).toBe(3);
|
||||
expect(payload.providerExecMs).toBe(600);
|
||||
expect(payload.providerGetMs).toBe(15);
|
||||
// The span carries the identical deltas — one build block feeds both.
|
||||
expect(spans[0]!.attributes.roundTrips).toBe(3);
|
||||
expect(spans[0]!.attributes.providerExecMs).toBe(600);
|
||||
expect(spans[0]!.attributes.providerGetMs).toBe(15);
|
||||
});
|
||||
|
||||
it("sets no span attribute (and no payload field) when a reader returns undefined", async () => {
|
||||
const events: AdapterRuntimeEvent[] = [];
|
||||
const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => {
|
||||
events.push(event);
|
||||
});
|
||||
const { tracer, spans } = makeMockTracer();
|
||||
|
||||
await measureStartupStep({ onEvent }, () => 0, "workspace.resolve", async () => "ok", {
|
||||
tracer,
|
||||
// A reader may return undefined when the counter is unavailable. The guard
|
||||
// must omit the attribute rather than emit NaN or 0.
|
||||
roundTrips: () => undefined as unknown as number,
|
||||
providerExecMs: () => undefined as unknown as number,
|
||||
});
|
||||
|
||||
expect(spans[0]!.attributes).not.toHaveProperty("roundTrips");
|
||||
expect(spans[0]!.attributes).not.toHaveProperty("providerExecMs");
|
||||
expect(Object.values(spans[0]!.attributes).some((v) => Number.isNaN(v))).toBe(false);
|
||||
const payload = events[0]!.payload as Record<string, unknown>;
|
||||
expect(payload).not.toHaveProperty("roundTrips");
|
||||
expect(payload).not.toHaveProperty("providerExecMs");
|
||||
});
|
||||
|
||||
it("normalizes a plugin-backed provider key to plugin and keeps a built-in family as-is", async () => {
|
||||
const onEvent = vi.fn(async () => {});
|
||||
|
||||
const custom = makeMockTracer();
|
||||
await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", {
|
||||
tracer: custom.tracer,
|
||||
provider: "acme-cloud-runner",
|
||||
});
|
||||
expect(custom.spans[0]!.attributes.provider).toBe("plugin");
|
||||
|
||||
const builtIn = makeMockTracer();
|
||||
await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", {
|
||||
tracer: builtIn.tracer,
|
||||
provider: "daytona",
|
||||
});
|
||||
expect(builtIn.spans[0]!.attributes.provider).toBe("daytona");
|
||||
});
|
||||
|
||||
it("normalizeProviderFamily maps every non-built-in key to plugin", () => {
|
||||
for (const key of ["daytona", "kubernetes", "e2b", "cloudflare", "exe-dev", "modal", "novita"]) {
|
||||
expect(normalizeProviderFamily(key)).toBe(key);
|
||||
}
|
||||
for (const key of ["acme", "my-plugin", "", "DAYTONA", undefined]) {
|
||||
expect(normalizeProviderFamily(key)).toBe("plugin");
|
||||
}
|
||||
});
|
||||
|
||||
it("emits exactly the allowlisted span-attribute key set and no command / path / ID / error-text key", async () => {
|
||||
const onEvent = vi.fn(async () => {});
|
||||
const { tracer, spans } = makeMockTracer();
|
||||
|
||||
await measureStartupStep({ onEvent }, () => 0, "acp.handshake", async () => "ok", {
|
||||
tracer,
|
||||
provider: "daytona",
|
||||
roundTrips: () => 3,
|
||||
providerExecMs: () => 600,
|
||||
providerGetMs: () => 15,
|
||||
// extra() carries caller-measured numbers into the EVENT payload only.
|
||||
// It must never widen the span-attribute set.
|
||||
extra: () => ({ createRuntimeMs: 12, ensureSessionMs: 6988 }),
|
||||
});
|
||||
|
||||
expect(Object.keys(spans[0]!.attributes).sort()).toEqual(
|
||||
["provider", "providerExecMs", "providerGetMs", "roundTrips", "step"],
|
||||
);
|
||||
// extra() keys stay off the span.
|
||||
expect(spans[0]!.attributes).not.toHaveProperty("createRuntimeMs");
|
||||
expect(spans[0]!.attributes).not.toHaveProperty("ensureSessionMs");
|
||||
// No free-form identifier / command / path key leaks in. The pattern uses
|
||||
// no `i` flag, so the camelCase `Id` matches `runId` / `userId` but not the
|
||||
// "id" inside the allowlisted `provider`.
|
||||
for (const key of Object.keys(spans[0]!.attributes)) {
|
||||
expect(key).not.toMatch(/command|args|env|stdout|stderr|path|url|repo|ref|branch|Id|_id|error|message/);
|
||||
}
|
||||
});
|
||||
|
||||
it("uses a no-op tracer by default so a call without a tracer changes nothing", async () => {
|
||||
const events: AdapterRuntimeEvent[] = [];
|
||||
const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => {
|
||||
events.push(event);
|
||||
});
|
||||
|
||||
// No tracer supplied. The helper must still emit the event and return the
|
||||
// value without throwing.
|
||||
const result = await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", {
|
||||
roundTrips: () => 3,
|
||||
});
|
||||
|
||||
expect(result).toBe("ok");
|
||||
expect(events[0]!.payload).toMatchObject({ step: "stage.sync" });
|
||||
});
|
||||
});
|
||||
@@ -9,6 +9,156 @@ import type { AdapterExecutionContext, AdapterRuntimeEvent } from "../types.js";
|
||||
*/
|
||||
export const RUN_STARTUP_STEP_EVENT_TYPE = "run.startup.step";
|
||||
|
||||
/**
|
||||
* The public built-in sandbox provider families. A key in this set is safe to
|
||||
* emit as a low-cardinality span attribute. Any other key is operator-defined
|
||||
* (plugin-backed) and unbounded, so `normalizeProviderFamily` maps it to the
|
||||
* generic value `plugin`. Keep this list closed and small.
|
||||
*/
|
||||
const BUILT_IN_PROVIDER_FAMILIES: ReadonlySet<string> = new Set([
|
||||
"daytona",
|
||||
"kubernetes",
|
||||
"e2b",
|
||||
"cloudflare",
|
||||
"exe-dev",
|
||||
"modal",
|
||||
"novita",
|
||||
]);
|
||||
|
||||
/** The generic family for any provider key outside the built-in list. */
|
||||
const PLUGIN_PROVIDER_FAMILY = "plugin";
|
||||
|
||||
/**
|
||||
* Map a raw provider key to a low-cardinality public family. Return the key
|
||||
* unchanged when it is a built-in family. Return `plugin` for every other
|
||||
* value, so an operator-defined plugin key never becomes an unbounded span
|
||||
* attribute. A missing or empty key also maps to `plugin`.
|
||||
*/
|
||||
export function normalizeProviderFamily(key: string | undefined): string {
|
||||
if (key && BUILT_IN_PROVIDER_FAMILIES.has(key)) return key;
|
||||
return PLUGIN_PROVIDER_FAMILY;
|
||||
}
|
||||
|
||||
/**
|
||||
* The value of `SpanStatusCode.ERROR` in `@opentelemetry/api`. `adapter-utils`
|
||||
* stays OTel-free, so the timing helper uses the numeric value directly. A real
|
||||
* injected OTel span reads it as the error status.
|
||||
*/
|
||||
const SPAN_STATUS_CODE_ERROR = 2;
|
||||
|
||||
/**
|
||||
* A minimal, OTel-free span contract. The server injects a real
|
||||
* `@opentelemetry/api` span, which satisfies this shape structurally. The
|
||||
* default is a no-op span, so a step with no injected tracer changes nothing.
|
||||
*/
|
||||
export interface StartupSpan {
|
||||
setAttribute(key: string, value: string | number | boolean): void;
|
||||
setStatus(status: { code: number; message?: string }): void;
|
||||
end(): void;
|
||||
}
|
||||
|
||||
/**
|
||||
* An opaque parent-context token. The server builds it from the OTel
|
||||
* `@opentelemetry/api` `context` / `trace` helpers. `adapter-utils` never reads
|
||||
* it; it only forwards it to `startSpan`, so this package stays OTel-free. A
|
||||
* child span opened with this token parents to the span the token carries.
|
||||
*/
|
||||
export type StartupSpanContext = unknown;
|
||||
|
||||
/**
|
||||
* A minimal, OTel-free tracer contract. The server injects a real
|
||||
* `@opentelemetry/api` tracer, which satisfies this shape structurally. The
|
||||
* `startSpan` signature is a subset of the OTel one, so a real tracer is
|
||||
* assignable here. The optional third argument is the explicit parent context:
|
||||
* a real OTel `startSpan(name, options, context)` parents the new span to the
|
||||
* span that `context` carries. `adapter-utils` passes it through as an opaque
|
||||
* token, so parenting never depends on ambient async-context propagation.
|
||||
*/
|
||||
export interface StartupTracer {
|
||||
startSpan(
|
||||
name: string,
|
||||
options?: { attributes?: Record<string, string | number | boolean> },
|
||||
context?: StartupSpanContext,
|
||||
): StartupSpan;
|
||||
}
|
||||
|
||||
/**
|
||||
* The injected tracer plus the one context helper the engine needs to build a
|
||||
* parent-context token from the root span. The server binds these to
|
||||
* `@opentelemetry/api` (`trace.getTracer`, `trace.setSpan` over
|
||||
* `context.active()`). The default is a no-op, so the whole span path stays a
|
||||
* no-op until the server injects a real implementation.
|
||||
*/
|
||||
export interface StartupTraceContext {
|
||||
readonly tracer: StartupTracer;
|
||||
/**
|
||||
* Return a parent-context token whose active span is `span`. A child span
|
||||
* opened with this token parents to `span`. The token is opaque to
|
||||
* `adapter-utils`.
|
||||
*/
|
||||
contextWithSpan(span: StartupSpan): StartupSpanContext;
|
||||
}
|
||||
|
||||
/** A shared no-op span. It implements the structural span contract and does
|
||||
* nothing, so a caller with no injected tracer changes no behavior. */
|
||||
export const NOOP_STARTUP_SPAN: StartupSpan = {
|
||||
setAttribute() {},
|
||||
setStatus() {},
|
||||
end() {},
|
||||
};
|
||||
|
||||
const NOOP_SPAN = NOOP_STARTUP_SPAN;
|
||||
|
||||
/**
|
||||
* The default tracer. It opens no real span, so `measureStartupStep` behaves
|
||||
* exactly as before when the caller injects no tracer.
|
||||
*/
|
||||
const NOOP_TRACER: StartupTracer = {
|
||||
startSpan: () => NOOP_SPAN,
|
||||
};
|
||||
|
||||
/**
|
||||
* The default trace context. Its tracer is a no-op and it produces no parent
|
||||
* token, so the engine emits no spans until the server injects a real
|
||||
* implementation.
|
||||
*/
|
||||
export const NOOP_STARTUP_TRACE_CONTEXT: StartupTraceContext = {
|
||||
tracer: NOOP_TRACER,
|
||||
contextWithSpan: () => undefined,
|
||||
};
|
||||
|
||||
/**
|
||||
* Set a numeric span attribute only when the value is a finite number. A reader
|
||||
* that returns `undefined` (the counter is unavailable) yields no attribute,
|
||||
* never `NaN` and never a misleading `0`. This mirrors the host counter guard
|
||||
* at `environment-execution-target.ts`.
|
||||
*/
|
||||
function setFiniteNumberAttr(
|
||||
span: StartupSpan,
|
||||
key: string,
|
||||
value: number | undefined,
|
||||
): void {
|
||||
if (typeof value === "number" && Number.isFinite(value)) {
|
||||
span.setAttribute(key, value);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute a counter delta from a reader. Return `undefined` when the reader is
|
||||
* absent, or when either the start or the end snapshot is not a finite number.
|
||||
* A `undefined` result yields no payload field and no span attribute.
|
||||
*/
|
||||
function finiteDelta(
|
||||
read: (() => number) | undefined,
|
||||
start: number | undefined,
|
||||
): number | undefined {
|
||||
if (!read) return undefined;
|
||||
const end = read();
|
||||
if (typeof end !== "number" || !Number.isFinite(end)) return undefined;
|
||||
const base = typeof start === "number" && Number.isFinite(start) ? start : 0;
|
||||
return end - base;
|
||||
}
|
||||
|
||||
/**
|
||||
* Optional per-step attribution attached to a `run.startup.step` event, all
|
||||
* additive to the free-form jsonb payload (no schema change). Each reader is a
|
||||
@@ -26,12 +176,28 @@ export const RUN_STARTUP_STEP_EVENT_TYPE = "run.startup.step";
|
||||
* any additional numeric fields to merge into the payload; used by
|
||||
* `acp.handshake` to carry its `createRuntimeMs` / `ensureSessionMs` sub-split
|
||||
* (Open Q2), which are measured by the caller rather than read from a counter.
|
||||
* The `extra` map feeds the EVENT payload only. Its keys never become span
|
||||
* attributes, so a free-form key cannot widen the closed span allowlist.
|
||||
* - `tracer` — an injected structural tracer. It defaults to a no-op, so the
|
||||
* span path changes no runtime behavior until the server injects a real
|
||||
* tracer. The span carries only the closed attribute allowlist (`step`, the
|
||||
* normalized `provider`, and the finite counter deltas).
|
||||
* - `parentContext` — an opaque parent-context token from the root span. When
|
||||
* set, the step's span parents to that root. `measureStartupStep` forwards it
|
||||
* to `startSpan` and never inspects it, so parenting stays explicit and does
|
||||
* not depend on ambient async-context propagation.
|
||||
* - `provider` — the raw provider key for the step. `measureStartupStep`
|
||||
* normalizes it through `normalizeProviderFamily` before it sets the
|
||||
* low-cardinality `provider` span attribute. It never sets the raw key.
|
||||
*/
|
||||
export interface StartupStepMeasureOptions {
|
||||
roundTrips?: () => number;
|
||||
providerExecMs?: () => number;
|
||||
providerGetMs?: () => number;
|
||||
extra?: () => Record<string, number>;
|
||||
tracer?: StartupTracer;
|
||||
parentContext?: StartupSpanContext;
|
||||
provider?: string;
|
||||
}
|
||||
|
||||
function buildStepEvent(payload: Record<string, unknown>): AdapterRuntimeEvent {
|
||||
@@ -55,6 +221,16 @@ function buildStepEvent(payload: Record<string, unknown>): AdapterRuntimeEvent {
|
||||
* `ctx.onEvent` is optional — a missing sink is a no-op that neither throws nor
|
||||
* swallows `fn`'s return value or error. A step skipped by a warm cache never
|
||||
* calls this helper, so it emits no event (never a zero).
|
||||
*
|
||||
* When `options.tracer` is injected, the helper also opens one span at `start`
|
||||
* and ends it in the `finally`. The span carries a closed attribute allowlist:
|
||||
* `step`, the normalized `provider`, and the finite counter deltas
|
||||
* (`roundTrips` / `providerExecMs` / `providerGetMs`). A throwing `fn` sets the
|
||||
* span error status before the span ends. The span build reuses the same delta
|
||||
* values as the event payload, so the two paths never drift. The tracer
|
||||
* defaults to a no-op, so a caller with no tracer changes nothing. Every span
|
||||
* call sits inside the same error swallow as the event sink, so a throwing
|
||||
* tracer never changes startup control flow.
|
||||
*/
|
||||
export async function measureStartupStep<T>(
|
||||
ctx: Pick<AdapterExecutionContext, "onEvent">,
|
||||
@@ -67,23 +243,58 @@ export async function measureStartupStep<T>(
|
||||
const roundTripsStart = options.roundTrips?.();
|
||||
const providerExecStart = options.providerExecMs?.();
|
||||
const providerGetStart = options.providerGetMs?.();
|
||||
|
||||
// Open the span with only the low-cardinality allowlisted attributes known at
|
||||
// the start: the step name and the normalized provider family.
|
||||
const tracer = options.tracer ?? NOOP_TRACER;
|
||||
const startAttributes: Record<string, string> = { step };
|
||||
if (options.provider !== undefined) {
|
||||
startAttributes.provider = normalizeProviderFamily(options.provider);
|
||||
}
|
||||
let span: StartupSpan;
|
||||
try {
|
||||
span = tracer.startSpan(step, { attributes: startAttributes }, options.parentContext);
|
||||
} catch {
|
||||
// A throwing tracer must not change startup control flow.
|
||||
span = NOOP_SPAN;
|
||||
}
|
||||
|
||||
let stepFailed = false;
|
||||
try {
|
||||
return await fn();
|
||||
} catch (err) {
|
||||
stepFailed = true;
|
||||
throw err;
|
||||
} finally {
|
||||
const durationMs = now() - start;
|
||||
|
||||
// One attribute-build block feeds both the event payload and the span, so
|
||||
// the two paths never drift. `undefined` deltas produce neither a payload
|
||||
// field nor a span attribute (fail open — never `NaN`, never `0`).
|
||||
const roundTrips = finiteDelta(options.roundTrips, roundTripsStart);
|
||||
const providerExecMs = finiteDelta(options.providerExecMs, providerExecStart);
|
||||
const providerGetMs = finiteDelta(options.providerGetMs, providerGetStart);
|
||||
|
||||
const payload: Record<string, unknown> = { step, durationMs };
|
||||
if (options.roundTrips) {
|
||||
payload.roundTrips = options.roundTrips() - (roundTripsStart ?? 0);
|
||||
}
|
||||
if (options.providerExecMs) {
|
||||
payload.providerExecMs = options.providerExecMs() - (providerExecStart ?? 0);
|
||||
}
|
||||
if (options.providerGetMs) {
|
||||
payload.providerGetMs = options.providerGetMs() - (providerGetStart ?? 0);
|
||||
}
|
||||
if (roundTrips !== undefined) payload.roundTrips = roundTrips;
|
||||
if (providerExecMs !== undefined) payload.providerExecMs = providerExecMs;
|
||||
if (providerGetMs !== undefined) payload.providerGetMs = providerGetMs;
|
||||
if (options.extra) {
|
||||
// `extra` feeds the EVENT payload only. Its keys never become span
|
||||
// attributes, so it cannot widen the closed span allowlist.
|
||||
Object.assign(payload, options.extra());
|
||||
}
|
||||
|
||||
try {
|
||||
if (stepFailed) span.setStatus({ code: SPAN_STATUS_CODE_ERROR });
|
||||
setFiniteNumberAttr(span, "roundTrips", roundTrips);
|
||||
setFiniteNumberAttr(span, "providerExecMs", providerExecMs);
|
||||
setFiniteNumberAttr(span, "providerGetMs", providerGetMs);
|
||||
span.end();
|
||||
} catch {
|
||||
// Observability must not change startup control flow.
|
||||
}
|
||||
|
||||
try {
|
||||
await ctx.onEvent?.(buildStepEvent(payload));
|
||||
} catch {
|
||||
|
||||
@@ -17,7 +17,7 @@ const execFile = promisify(execFileCallback);
|
||||
|
||||
interface SpawnRunnerHandle {
|
||||
runner: CommandManagedRuntimeRunner;
|
||||
calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string; noProfile?: boolean }>;
|
||||
calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string }>;
|
||||
}
|
||||
|
||||
// A runner that actually executes the shell scripts (piping stdin through a real
|
||||
@@ -27,7 +27,7 @@ function makeSpawnRunner(options: {
|
||||
supportsSingleStreamStdinProgress?: boolean;
|
||||
maxStdoutBytes?: number;
|
||||
} = {}): SpawnRunnerHandle {
|
||||
const calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string; noProfile?: boolean }> = [];
|
||||
const calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string }> = [];
|
||||
const runner: CommandManagedRuntimeRunner = {
|
||||
supportsSingleStreamStdinProgress: options.supportsSingleStreamStdinProgress,
|
||||
execute: async (input) =>
|
||||
@@ -37,7 +37,6 @@ function makeSpawnRunner(options: {
|
||||
args: input.args,
|
||||
cwd: input.cwd,
|
||||
stdin: input.stdin,
|
||||
noProfile: input.noProfile,
|
||||
});
|
||||
const startedAt = new Date().toISOString();
|
||||
const command =
|
||||
@@ -153,7 +152,6 @@ describe("command managed runtime", () => {
|
||||
env?: Record<string, string>;
|
||||
stdin?: string;
|
||||
timeoutMs?: number;
|
||||
noProfile?: boolean;
|
||||
}> = [];
|
||||
const runner = {
|
||||
execute: async (input: {
|
||||
@@ -163,7 +161,6 @@ describe("command managed runtime", () => {
|
||||
env?: Record<string, string>;
|
||||
stdin?: string;
|
||||
timeoutMs?: number;
|
||||
noProfile?: boolean;
|
||||
}): Promise<RunProcessResult> => {
|
||||
calls.push({ ...input });
|
||||
const startedAt = new Date().toISOString();
|
||||
@@ -236,7 +233,6 @@ describe("command managed runtime", () => {
|
||||
// The single-stream upload pipes the tarball through exactly one stdin-backed
|
||||
// process (the speed fix); nothing else streams stdin.
|
||||
expect(calls.filter((call) => call.stdin != null).length).toBe(1);
|
||||
expect(calls.some((call) => call.noProfile === true)).toBe(true);
|
||||
|
||||
await mkdir(path.join(remoteWorkspaceDir, ".paperclip-runtime"), { recursive: true });
|
||||
await writeFile(path.join(remoteWorkspaceDir, "README.md"), "remote workspace\n", "utf8");
|
||||
@@ -251,7 +247,6 @@ describe("command managed runtime", () => {
|
||||
// Restore streams the download through `base64`/onLog (no stdin), so the only
|
||||
// stdin-backed call remains the single upload from prepare.
|
||||
expect(calls.filter((call) => call.stdin != null).length).toBe(1);
|
||||
expect(calls.some((call) => call.noProfile === true)).toBe(true);
|
||||
});
|
||||
|
||||
it("stages runtime assets without replacing or restoring an in-place workspace", async () => {
|
||||
@@ -308,6 +303,59 @@ describe("command managed runtime", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("stages each additional project into an isolated dir on the base64/tar transport, one failure skipped", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-runtime-additional-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
|
||||
const localWorkspaceDir = path.join(rootDir, "local-workspace");
|
||||
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||
await mkdir(localWorkspaceDir, { recursive: true });
|
||||
await mkdir(remoteWorkspaceDir, { recursive: true });
|
||||
await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor\n", "utf8");
|
||||
|
||||
const goodOne = path.join(rootDir, "src-one");
|
||||
const goodTwo = path.join(rootDir, "src-two");
|
||||
await mkdir(goodOne, { recursive: true });
|
||||
await mkdir(path.join(goodTwo, "nested"), { recursive: true });
|
||||
await writeFile(path.join(goodOne, "one.txt"), "one body\n", "utf8");
|
||||
await writeFile(path.join(goodTwo, "nested", "two.txt"), "two body\n", "utf8");
|
||||
|
||||
// The `makeSpawnRunner` runner exposes no native syncIn, so staging rides the
|
||||
// base64/tar fallback. The middle source points at a missing directory, so
|
||||
// its tar build fails; failure isolation skips only it.
|
||||
const { runner } = makeSpawnRunner();
|
||||
const prepared = await prepareCommandManagedRuntime({
|
||||
runner,
|
||||
spec: {
|
||||
remoteCwd: remoteWorkspaceDir,
|
||||
timeoutMs: 30_000,
|
||||
},
|
||||
adapterKey: "claude",
|
||||
workspaceLocalDir: localWorkspaceDir,
|
||||
additionalSources: [
|
||||
{ localPath: goodOne, projectId: "one" },
|
||||
{ localPath: path.join(rootDir, "missing"), projectId: "broken" },
|
||||
{ localPath: goodTwo, projectId: "two" },
|
||||
],
|
||||
});
|
||||
|
||||
const runtimeRootDir = path.posix.join(remoteWorkspaceDir, ".paperclip-runtime", "claude");
|
||||
expect(Object.keys(prepared.additionalSourceDirs).sort()).toEqual(["one", "two"]);
|
||||
expect(prepared.additionalSourceDirs.one).toBe(path.posix.join(runtimeRootDir, "project-one"));
|
||||
expect(prepared.additionalSourceDirs.two).toBe(path.posix.join(runtimeRootDir, "project-two"));
|
||||
expect(prepared.additionalSourceDirs.broken).toBeUndefined();
|
||||
|
||||
// Each healthy project's tree materialized in its OWN dir (nested files kept).
|
||||
await expect(readFile(path.join(prepared.additionalSourceDirs.one, "one.txt"), "utf8")).resolves.toBe("one body\n");
|
||||
await expect(readFile(path.join(prepared.additionalSourceDirs.two, "nested", "two.txt"), "utf8")).resolves.toBe(
|
||||
"two body\n",
|
||||
);
|
||||
// The broken project's dir was never created.
|
||||
await expect(readFile(path.join(runtimeRootDir, "project-broken"), "utf8")).rejects.toMatchObject({
|
||||
code: "ENOENT",
|
||||
});
|
||||
});
|
||||
|
||||
it("keeps adapter detection on the profile-backed shell path", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-runtime-detect-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
@@ -330,10 +378,8 @@ describe("command managed runtime", () => {
|
||||
detectCommand: "sh",
|
||||
});
|
||||
|
||||
// The detection probe must be the first shell invocation and stay on the
|
||||
// default profile-sourcing path (noProfile !== true) so a CLI provided by
|
||||
// the login profile is discoverable before we decide whether to install.
|
||||
expect(calls[0]?.noProfile).not.toBe(true);
|
||||
// The detection probe must be the first shell invocation, so a CLI on the
|
||||
// sandbox default PATH is discoverable before we decide whether to install.
|
||||
expect(calls[0]?.args?.join(" ")).toContain("command -v 'sh'");
|
||||
// Detection succeeds here, so the install command must be skipped entirely;
|
||||
// the remaining calls are workspace staging, never the install command.
|
||||
@@ -588,15 +634,6 @@ describe("command managed runtime", () => {
|
||||
expect(untarIdx).toBeGreaterThan(uploadIdx);
|
||||
expect(cmd1Idx).toBeGreaterThan(untarIdx);
|
||||
expect(cmd2Idx).toBeGreaterThan(cmd1Idx);
|
||||
|
||||
// Fast path: the fixed internal transport helpers (tar upload + untar) ride
|
||||
// the no-profile shell — they are trusted, fixed commands that never need a
|
||||
// login-shell profile. The opaque post-upload commands stay profile-backed
|
||||
// (noProfile !== true) so any env a caller-supplied command relies on is present.
|
||||
expect(calls[uploadIdx]?.noProfile).toBe(true);
|
||||
expect(calls[untarIdx]?.noProfile).toBe(true);
|
||||
expect(calls[cmd1Idx]?.noProfile).not.toBe(true);
|
||||
expect(calls[cmd2Idx]?.noProfile).not.toBe(true);
|
||||
});
|
||||
|
||||
it("fallback syncIn runs a post-upload command under its own timeout, not the sync-client default", async () => {
|
||||
@@ -663,11 +700,6 @@ describe("command managed runtime", () => {
|
||||
expect(scripts[3]).toContain(targetFile);
|
||||
expect(scripts[4]).toContain("rm -rf");
|
||||
expect(scripts[4]).toContain(targetFile + ".paperclip-syncin.");
|
||||
|
||||
// Fast path: the staged-write helpers (chmod + rename) are fixed internal
|
||||
// commands, so they ride the no-profile shell alongside the upload/staging.
|
||||
expect(calls[2]?.noProfile).toBe(true); // chmod
|
||||
expect(calls[3]?.noProfile).toBe(true); // mv (rename into place)
|
||||
});
|
||||
|
||||
it("fallback syncIn cleans up a staged file when chmod fails before rename", async () => {
|
||||
|
||||
@@ -6,6 +6,7 @@ import {
|
||||
createTarballFromDirectory,
|
||||
prepareSandboxManagedRuntime,
|
||||
type PreparedSandboxManagedRuntime,
|
||||
type SandboxAdditionalSource,
|
||||
type SandboxManagedRuntimeAsset,
|
||||
type SandboxManagedRuntimeClient,
|
||||
type SandboxRemoteExecutionSpec,
|
||||
@@ -50,7 +51,6 @@ export interface CommandManagedRuntimeRunner {
|
||||
env?: Record<string, string>;
|
||||
stdin?: string;
|
||||
timeoutMs?: number;
|
||||
noProfile?: boolean;
|
||||
onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
|
||||
onSpawn?: (meta: { pid: number; startedAt: string }) => Promise<void>;
|
||||
}): Promise<RunProcessResult>;
|
||||
@@ -206,7 +206,6 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
opts: {
|
||||
stdin?: string;
|
||||
timeoutMs?: number;
|
||||
noProfile?: boolean;
|
||||
onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
|
||||
} = {},
|
||||
) => {
|
||||
@@ -216,7 +215,6 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
cwd: input.commandCwd,
|
||||
stdin: opts.stdin,
|
||||
timeoutMs: opts.timeoutMs ?? input.timeoutMs,
|
||||
noProfile: opts.noProfile === true,
|
||||
onLog: opts.onLog,
|
||||
});
|
||||
requireSuccessfulResult(result, script);
|
||||
@@ -225,7 +223,7 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
|
||||
const client: SandboxManagedRuntimeClient = {
|
||||
makeDir: async (remotePath) => {
|
||||
await runShell(`mkdir -p ${shellQuote(remotePath)}`, { noProfile: true });
|
||||
await runShell(`mkdir -p ${shellQuote(remotePath)}`);
|
||||
},
|
||||
writeFile: async (remotePath, bytes, options) => {
|
||||
const buffer = toBuffer(bytes);
|
||||
@@ -252,7 +250,7 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
`mkdir -p ${shellQuote(remoteDir)} && ` +
|
||||
`base64 -d > ${shellQuote(remoteTempPath)} && ` +
|
||||
`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`,
|
||||
{ stdin: body, noProfile: true },
|
||||
{ stdin: body },
|
||||
);
|
||||
await options?.onProgress?.(total, total);
|
||||
return;
|
||||
@@ -266,15 +264,14 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
await runShell(
|
||||
`mkdir -p ${shellQuote(remoteDir)} && ` +
|
||||
`rm -f ${shellQuote(remoteTempPath)} && : > ${shellQuote(remoteTempPath)}`,
|
||||
{ noProfile: true },
|
||||
);
|
||||
for (let offset = 0; offset < total; offset += REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE) {
|
||||
const end = Math.min(total, offset + REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE);
|
||||
const chunk = buffer.subarray(offset, end).toString("base64");
|
||||
await runShell(`base64 -d >> ${shellQuote(remoteTempPath)}`, { stdin: chunk, noProfile: true });
|
||||
await runShell(`base64 -d >> ${shellQuote(remoteTempPath)}`, { stdin: chunk });
|
||||
await options?.onProgress?.(end, total);
|
||||
}
|
||||
await runShell(`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`, { noProfile: true });
|
||||
await runShell(`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`);
|
||||
await options?.onProgress?.(total, total);
|
||||
} finally {
|
||||
await bestEffortRemoveRemotePath(client, remoteTempPath);
|
||||
@@ -284,7 +281,7 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
// Chunked reads intentionally query the remote size first, even without
|
||||
// a progress sink, so each sandbox RPC stays bounded and truncation is
|
||||
// detected without materializing the whole file as one stdout string.
|
||||
const sizeResult = await runShell(`wc -c < ${shellQuote(remotePath)}`, { noProfile: true });
|
||||
const sizeResult = await runShell(`wc -c < ${shellQuote(remotePath)}`);
|
||||
const totalBytes = Number.parseInt(sizeResult.stdout.trim(), 10);
|
||||
if (!Number.isFinite(totalBytes) || totalBytes < 0) {
|
||||
throw new Error(`Could not determine remote file size for ${remotePath}`);
|
||||
@@ -303,7 +300,6 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
for (let chunkIndex = 0; decodedSoFar < totalBytes; chunkIndex++) {
|
||||
const result = await runShell(
|
||||
`dd if=${shellQuote(remotePath)} bs=${REMOTE_READ_CHUNK_BYTES} skip=${chunkIndex} count=1 2>/dev/null | base64`,
|
||||
{ noProfile: true },
|
||||
);
|
||||
const chunk = Buffer.from(result.stdout.replace(/\s+/g, ""), "base64");
|
||||
if (chunk.byteLength === 0) break;
|
||||
@@ -326,7 +322,6 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
`basename "$entry"; ` +
|
||||
`done; ` +
|
||||
`fi`,
|
||||
{ noProfile: true },
|
||||
);
|
||||
return result.stdout
|
||||
.split(/\r?\n/)
|
||||
@@ -340,7 +335,6 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
args: shellCommandArgs(`rm -rf ${shellQuote(remotePath)}`),
|
||||
cwd: input.commandCwd,
|
||||
timeoutMs: input.timeoutMs,
|
||||
noProfile: true,
|
||||
});
|
||||
requireSuccessfulResult(result, `remove ${remotePath}`);
|
||||
},
|
||||
@@ -350,7 +344,6 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
args: shellCommandArgs(command),
|
||||
cwd: input.commandCwd,
|
||||
timeoutMs: options.timeoutMs,
|
||||
noProfile: options.noProfile === true,
|
||||
});
|
||||
requireSuccessfulResult(result, command);
|
||||
},
|
||||
@@ -390,7 +383,7 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
await client.writeFile(remoteTarPath, bufferToArrayBuffer(tarBytes));
|
||||
await client.run(
|
||||
buildSyncInExtractDirectoryCommand({ remoteTarPath, targetDir: mapping.targetPath }),
|
||||
{ timeoutMs: input.timeoutMs, noProfile: true },
|
||||
{ timeoutMs: input.timeoutMs },
|
||||
);
|
||||
bytesTransferred += tarBytes.byteLength;
|
||||
} else {
|
||||
@@ -403,11 +396,11 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
if (mapping.mode != null) {
|
||||
await client.run(
|
||||
buildSyncInChmodCommand({ mode: mapping.mode, targetPath: targetPathForWrite }),
|
||||
{ timeoutMs: input.timeoutMs, noProfile: true },
|
||||
{ timeoutMs: input.timeoutMs },
|
||||
);
|
||||
await client.run(
|
||||
buildSyncInRenameCommand({ sourcePath: targetPathForWrite, targetPath: mapping.targetPath }),
|
||||
{ timeoutMs: input.timeoutMs, noProfile: true },
|
||||
{ timeoutMs: input.timeoutMs },
|
||||
);
|
||||
}
|
||||
bytesTransferred += fileBytes.byteLength;
|
||||
@@ -475,6 +468,8 @@ export async function prepareCommandManagedRuntime(input: {
|
||||
workspaceExclude?: string[];
|
||||
preserveAbsentOnRestore?: string[];
|
||||
assets?: CommandManagedRuntimeAsset[];
|
||||
/** Referenced (additional) projects to stage into the sandbox as plain, read-only trees. */
|
||||
additionalSources?: SandboxAdditionalSource[];
|
||||
installCommand?: string | null;
|
||||
/** When provided alongside `installCommand`, skip the install if `command -v <detectCommand>` succeeds. */
|
||||
detectCommand?: string | null;
|
||||
@@ -530,6 +525,7 @@ export async function prepareCommandManagedRuntime(input: {
|
||||
workspaceExclude: mergeRuntimeExcludes(input.workspaceExclude),
|
||||
preserveAbsentOnRestore: input.preserveAbsentOnRestore,
|
||||
assets: input.assets,
|
||||
additionalSources: input.additionalSources,
|
||||
onProgress: input.onProgress,
|
||||
onRuntimeProgress: input.onRuntimeProgress,
|
||||
});
|
||||
@@ -567,6 +563,7 @@ export async function prepareCommandManagedRuntime(input: {
|
||||
workspaceExclude: mergeRuntimeExcludes(input.workspaceExclude),
|
||||
preserveAbsentOnRestore: input.preserveAbsentOnRestore,
|
||||
assets: input.assets,
|
||||
additionalSources: input.additionalSources,
|
||||
onProgress: input.onProgress,
|
||||
onRuntimeProgress: input.onRuntimeProgress,
|
||||
});
|
||||
|
||||
@@ -194,6 +194,60 @@ describe("sandbox adapter execution targets", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("creates the process session directories only in the launch exec, not in upfront makeDir execs", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-process-session-makedir-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const childPath = path.join(rootDir, "noop-acp-child.mjs");
|
||||
await writeFile(childPath, "process.stdin.on('data', () => {});\n", "utf8");
|
||||
|
||||
const delegate = createLocalSandboxRunner();
|
||||
const execScripts: string[] = [];
|
||||
const runner = {
|
||||
execute: vi.fn(async (input: Parameters<typeof delegate.execute>[0]) => {
|
||||
execScripts.push(input.args?.[1] ?? "");
|
||||
return delegate.execute(input);
|
||||
}),
|
||||
};
|
||||
const target: AdapterSandboxExecutionTarget = {
|
||||
kind: "remote",
|
||||
transport: "sandbox",
|
||||
providerKey: "local-test",
|
||||
remoteCwd: rootDir,
|
||||
timeoutMs: 30_000,
|
||||
runner,
|
||||
};
|
||||
|
||||
const bridge = await startAdapterExecutionTargetProcessSessionBridge({
|
||||
runId: "run-process-session-makedir",
|
||||
target,
|
||||
runtimeRootDir: path.posix.join(rootDir, ".paperclip-runtime", "acpx"),
|
||||
adapterKey: "acpx",
|
||||
command: process.execPath,
|
||||
args: [childPath],
|
||||
cwd: rootDir,
|
||||
env: {},
|
||||
timeoutSec: 5,
|
||||
onLog: async () => {},
|
||||
});
|
||||
expect(bridge).not.toBeNull();
|
||||
|
||||
try {
|
||||
// No standalone `mkdir -p '<dir>/stdin'` or `.../events` exec runs before launch.
|
||||
const standaloneSessionDirExecs = execScripts.filter((script) =>
|
||||
/^mkdir -p '[^']*\/(stdin|events)'\s*$/.test(script),
|
||||
);
|
||||
expect(standaloneSessionDirExecs).toEqual([]);
|
||||
|
||||
// The launch exec creates both directories in one `mkdir -p` line.
|
||||
const launchExecs = execScripts.filter(
|
||||
(script) => script.includes("nohup") && /mkdir -p [^\n]*\/stdin[^\n]*\/events/.test(script),
|
||||
);
|
||||
expect(launchExecs.length).toBe(1);
|
||||
} finally {
|
||||
await bridge?.stop();
|
||||
}
|
||||
});
|
||||
|
||||
it("bridges bidirectional sandbox process sessions through a local ACPX-spawnable proxy", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-process-session-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
|
||||
@@ -14,6 +14,8 @@ import {
|
||||
prepareRemoteManagedRuntime,
|
||||
remoteExecutionSessionMatches,
|
||||
} from "./remote-managed-runtime.js";
|
||||
import type { SandboxAdditionalSource } from "./sandbox-managed-runtime.js";
|
||||
export type { SandboxAdditionalSource } from "./sandbox-managed-runtime.js";
|
||||
import {
|
||||
createCommandManagedSandboxCallbackBridgeQueueClient,
|
||||
createSandboxCallbackBridgeAsset,
|
||||
@@ -114,6 +116,12 @@ export interface PreparedAdapterExecutionTargetRuntime {
|
||||
workspaceRemoteDir: string | null;
|
||||
runtimeRootDir: string | null;
|
||||
assetDirs: Record<string, string>;
|
||||
/**
|
||||
* Remote directory of each additional (referenced) project that staged
|
||||
* successfully, keyed by `projectId`. Empty for a local target or when no
|
||||
* additional sources were requested.
|
||||
*/
|
||||
additionalSourceDirs: Record<string, string>;
|
||||
restoreWorkspace(onProgress?: RuntimeProgressSink): Promise<void>;
|
||||
}
|
||||
|
||||
@@ -1107,6 +1115,8 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
|
||||
workspaceExclude?: string[];
|
||||
preserveAbsentOnRestore?: string[];
|
||||
assets?: AdapterManagedRuntimeAsset[];
|
||||
/** Referenced (additional) projects to stage into the sandbox as plain, read-only trees. */
|
||||
additionalSources?: SandboxAdditionalSource[];
|
||||
installCommand?: string | null;
|
||||
/** When provided alongside `installCommand`, skip the install if the binary is already on PATH. */
|
||||
detectCommand?: string | null;
|
||||
@@ -1124,6 +1134,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
|
||||
workspaceRemoteDir: null,
|
||||
runtimeRootDir: null,
|
||||
assetDirs: {},
|
||||
additionalSourceDirs: {},
|
||||
restoreWorkspace: async () => {},
|
||||
};
|
||||
}
|
||||
@@ -1137,6 +1148,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
|
||||
workspaceRemoteDir: input.workspaceRemoteDir,
|
||||
syncWorkspace: input.syncWorkspace,
|
||||
assets: input.assets,
|
||||
additionalSources: input.additionalSources,
|
||||
onProgress: input.onProgress,
|
||||
});
|
||||
return {
|
||||
@@ -1144,6 +1156,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
|
||||
workspaceRemoteDir: prepared.workspaceRemoteDir,
|
||||
runtimeRootDir: prepared.runtimeRootDir,
|
||||
assetDirs: prepared.assetDirs,
|
||||
additionalSourceDirs: prepared.additionalSourceDirs,
|
||||
restoreWorkspace: prepared.restoreWorkspace,
|
||||
};
|
||||
}
|
||||
@@ -1167,6 +1180,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
|
||||
workspaceExclude: input.workspaceExclude,
|
||||
preserveAbsentOnRestore: input.preserveAbsentOnRestore,
|
||||
assets: input.assets,
|
||||
additionalSources: input.additionalSources,
|
||||
installCommand: input.installCommand,
|
||||
detectCommand: input.detectCommand,
|
||||
onProgress: input.onProgress,
|
||||
@@ -1177,6 +1191,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
|
||||
workspaceRemoteDir: prepared.workspaceRemoteDir,
|
||||
runtimeRootDir: prepared.runtimeRootDir,
|
||||
assetDirs: prepared.assetDirs,
|
||||
additionalSourceDirs: prepared.additionalSourceDirs,
|
||||
restoreWorkspace: prepared.restoreWorkspace,
|
||||
};
|
||||
}
|
||||
@@ -1363,8 +1378,9 @@ export async function startAdapterExecutionTargetProcessSessionBridge(input: {
|
||||
shellCommand,
|
||||
});
|
||||
|
||||
await client.makeDir(stdinDir);
|
||||
await client.makeDir(eventsDir);
|
||||
// The launch exec below re-creates stdinDir and eventsDir with one `mkdir -p`,
|
||||
// and the remote script also creates them on start. No reader touches the two
|
||||
// directories before the launch exec runs, so upfront makeDir execs are redundant.
|
||||
await syncProcessSessionRemoteScript({
|
||||
runner,
|
||||
remoteCwd: target.remoteCwd,
|
||||
|
||||
@@ -15,7 +15,7 @@ const {
|
||||
stdout: Buffer.from('{"token":"remote"}\n').toString("base64"),
|
||||
stderr: "",
|
||||
})),
|
||||
syncDirectoryToSsh: vi.fn(async () => undefined),
|
||||
syncDirectoryToSsh: vi.fn(async (_input: { localDir: string }) => undefined),
|
||||
}));
|
||||
|
||||
vi.mock("./ssh.js", () => ({
|
||||
@@ -92,4 +92,95 @@ describe("remote managed runtime", () => {
|
||||
);
|
||||
expect(restoredAuth).toBe('{"token":"remote"}\n');
|
||||
});
|
||||
|
||||
it("stages each additional project into its own isolated SSH dir, isolating one failure", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-remote-runtime-additional-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const workspaceDir = path.join(rootDir, "workspace");
|
||||
const firstDir = path.join(rootDir, "referenced-first");
|
||||
const secondDir = path.join(rootDir, "referenced-second");
|
||||
const brokenDir = path.join(rootDir, "referenced-broken");
|
||||
await mkdir(workspaceDir, { recursive: true });
|
||||
|
||||
// The transfer rejects only for the broken project's directory.
|
||||
syncDirectoryToSsh.mockImplementation(async (input: { localDir: string }) => {
|
||||
if (input.localDir === brokenDir) throw new Error("ssh transfer failed");
|
||||
return undefined;
|
||||
});
|
||||
|
||||
const prepared = await prepareRemoteManagedRuntime({
|
||||
spec: {
|
||||
host: "127.0.0.1",
|
||||
port: 2222,
|
||||
username: "fixture",
|
||||
remoteWorkspacePath: "/app",
|
||||
remoteCwd: "/app",
|
||||
privateKey: "PRIVATE KEY",
|
||||
knownHosts: "KNOWN HOSTS",
|
||||
strictHostKeyChecking: true,
|
||||
},
|
||||
runId: "run-additional",
|
||||
adapterKey: "codex",
|
||||
workspaceLocalDir: workspaceDir,
|
||||
workspaceRemoteDir: "/app",
|
||||
syncWorkspace: false,
|
||||
additionalSources: [
|
||||
{ localPath: firstDir, projectId: "first" },
|
||||
{ localPath: brokenDir, projectId: "broken" },
|
||||
{ localPath: secondDir, projectId: "second" },
|
||||
],
|
||||
});
|
||||
|
||||
// Each healthy project staged into its OWN isolated dir under the runtime
|
||||
// root; the broken one is skipped, not fatal.
|
||||
expect(Object.keys(prepared.additionalSourceDirs).sort()).toEqual(["first", "second"]);
|
||||
expect(prepared.additionalSourceDirs.first).toBe("/app/.paperclip-runtime/codex/project-first");
|
||||
expect(prepared.additionalSourceDirs.second).toBe("/app/.paperclip-runtime/codex/project-second");
|
||||
expect(prepared.additionalSourceDirs.broken).toBeUndefined();
|
||||
expect(syncDirectoryToSsh).toHaveBeenCalledWith(expect.objectContaining({
|
||||
localDir: firstDir,
|
||||
remoteDir: "/app/.paperclip-runtime/codex/project-first",
|
||||
}));
|
||||
expect(syncDirectoryToSsh).toHaveBeenCalledWith(expect.objectContaining({
|
||||
localDir: secondDir,
|
||||
remoteDir: "/app/.paperclip-runtime/codex/project-second",
|
||||
}));
|
||||
});
|
||||
|
||||
it("skips an additional project whose localPath is not absolute", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-remote-runtime-relative-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const workspaceDir = path.join(rootDir, "workspace");
|
||||
const healthyDir = path.join(rootDir, "referenced-healthy");
|
||||
await mkdir(workspaceDir, { recursive: true });
|
||||
|
||||
const prepared = await prepareRemoteManagedRuntime({
|
||||
spec: {
|
||||
host: "127.0.0.1",
|
||||
port: 2222,
|
||||
username: "fixture",
|
||||
remoteWorkspacePath: "/app",
|
||||
remoteCwd: "/app",
|
||||
privateKey: "PRIVATE KEY",
|
||||
knownHosts: "KNOWN HOSTS",
|
||||
strictHostKeyChecking: true,
|
||||
},
|
||||
runId: "run-relative",
|
||||
adapterKey: "codex",
|
||||
workspaceLocalDir: workspaceDir,
|
||||
workspaceRemoteDir: "/app",
|
||||
syncWorkspace: false,
|
||||
additionalSources: [
|
||||
{ localPath: "relative/referenced", projectId: "relative" },
|
||||
{ localPath: healthyDir, projectId: "healthy" },
|
||||
],
|
||||
});
|
||||
|
||||
// The relative-path project never reaches the transfer and is skipped; the
|
||||
// absolute-path project still stages.
|
||||
expect(Object.keys(prepared.additionalSourceDirs)).toEqual(["healthy"]);
|
||||
expect(syncDirectoryToSsh).not.toHaveBeenCalledWith(expect.objectContaining({
|
||||
localDir: "relative/referenced",
|
||||
}));
|
||||
});
|
||||
});
|
||||
@@ -7,10 +7,26 @@ import {
|
||||
restoreWorkspaceFromSshExecution,
|
||||
syncDirectoryToSsh,
|
||||
} from "./ssh.js";
|
||||
import type { SandboxManagedRuntimeAssetRestoreContext } from "./sandbox-managed-runtime.js";
|
||||
import type {
|
||||
SandboxAdditionalSource,
|
||||
SandboxManagedRuntimeAssetRestoreContext,
|
||||
} from "./sandbox-managed-runtime.js";
|
||||
import { captureDirectorySnapshot } from "./workspace-restore-merge.js";
|
||||
import type { RuntimeProgressSink } from "./runtime-progress.js";
|
||||
|
||||
const REMOTE_ADDITIONAL_SOURCE_HEAVY_DIR_EXCLUDES = [
|
||||
"node_modules",
|
||||
"vendor",
|
||||
"dist",
|
||||
"build",
|
||||
"out",
|
||||
"coverage",
|
||||
".next",
|
||||
".turbo",
|
||||
".cache",
|
||||
".git",
|
||||
].flatMap((entry) => [entry, `${entry}/*`, `*/${entry}`, `*/${entry}/*`]);
|
||||
|
||||
export interface RemoteManagedRuntimeAsset {
|
||||
key: string;
|
||||
localDir: string;
|
||||
@@ -25,6 +41,12 @@ export interface PreparedRemoteManagedRuntime {
|
||||
workspaceRemoteDir: string;
|
||||
runtimeRootDir: string;
|
||||
assetDirs: Record<string, string>;
|
||||
/**
|
||||
* Remote directory of each additional (referenced) project that staged
|
||||
* successfully, keyed by `projectId`. A project whose staging failed is
|
||||
* absent (per-project failure isolation).
|
||||
*/
|
||||
additionalSourceDirs: Record<string, string>;
|
||||
restoreWorkspace(onProgress?: RuntimeProgressSink): Promise<void>;
|
||||
}
|
||||
|
||||
@@ -86,6 +108,8 @@ export async function prepareRemoteManagedRuntime(input: {
|
||||
workspaceRemoteDir?: string;
|
||||
syncWorkspace?: boolean;
|
||||
assets?: RemoteManagedRuntimeAsset[];
|
||||
/** Referenced (additional) projects to stage as plain, read-only trees. */
|
||||
additionalSources?: SandboxAdditionalSource[];
|
||||
// Upload progress sink. Threaded for the byte-counting transport rewrite; the
|
||||
// child task wires it into the workspace/asset transfers.
|
||||
onProgress?: RuntimeProgressSink;
|
||||
@@ -148,12 +172,50 @@ export async function prepareRemoteManagedRuntime(input: {
|
||||
throw error;
|
||||
}
|
||||
|
||||
// Stage each referenced (additional) project as a plain, read-only tree in its
|
||||
// OWN isolated remote directory (`project-<projectId>`). Additional sources
|
||||
// never get the anchor's git-history/overlay semantics. Per-project failure
|
||||
// isolation: one project's failure logs a warning and is skipped; the run and
|
||||
// the other projects continue (no workspace restore, unlike an asset failure).
|
||||
const additionalSourceDirs: Record<string, string> = {};
|
||||
for (const source of input.additionalSources ?? []) {
|
||||
const { localPath, projectId } = source;
|
||||
try {
|
||||
if (!path.posix.isAbsolute(localPath)) {
|
||||
throw new Error(`additional source localPath is not an absolute path: ${localPath}`);
|
||||
}
|
||||
if (
|
||||
projectId.length === 0 ||
|
||||
projectId.includes("/") ||
|
||||
projectId.includes("\\") ||
|
||||
projectId.includes("..")
|
||||
) {
|
||||
throw new Error(`additional source projectId is not a simple path segment: ${projectId}`);
|
||||
}
|
||||
const remoteDir = path.posix.join(runtimeRootDir, `project-${projectId}`);
|
||||
await syncDirectoryToSsh({
|
||||
spec: input.spec,
|
||||
localDir: localPath,
|
||||
remoteDir,
|
||||
exclude: REMOTE_ADDITIONAL_SOURCE_HEAVY_DIR_EXCLUDES,
|
||||
onProgress: input.onProgress,
|
||||
progressLabel: `project-${projectId}`,
|
||||
});
|
||||
additionalSourceDirs[projectId] = remoteDir;
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`[paperclip] Failed to stage referenced project ${projectId}; skipping it. ${String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
spec: input.spec,
|
||||
workspaceLocalDir: input.workspaceLocalDir,
|
||||
workspaceRemoteDir,
|
||||
runtimeRootDir,
|
||||
assetDirs,
|
||||
additionalSourceDirs,
|
||||
restoreWorkspace: async (onProgress?: RuntimeProgressSink) => {
|
||||
if (preparedWorkspace && baselineSnapshot) {
|
||||
await restoreWorkspaceFromSshExecution({
|
||||
|
||||
@@ -438,6 +438,7 @@ describe("sandbox callback bridge", () => {
|
||||
const worker = await startSandboxCallbackBridgeWorker({
|
||||
client: {
|
||||
makeDir: async () => {},
|
||||
makeDirs: async () => {},
|
||||
listJsonFiles: async () => {
|
||||
throw new Error(
|
||||
"list /remote/.paperclip-runtime/gemini/paperclip-bridge/queue/requests failed with exit code 255: kex_exchange_identification: read: Connection reset by peer",
|
||||
@@ -1114,4 +1115,110 @@ describe("sandbox callback bridge", () => {
|
||||
},
|
||||
}));
|
||||
});
|
||||
|
||||
it("creates the bridge queue directories in one directory-creation exec", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-bridge-makedirs-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
|
||||
const queueDir = path.posix.join(rootDir, "queue");
|
||||
const directories = sandboxCallbackBridgeDirectories(queueDir);
|
||||
const makeDir = vi.fn(async () => {});
|
||||
const makeDirs = vi.fn(async () => {});
|
||||
|
||||
const worker = await startSandboxCallbackBridgeWorker({
|
||||
client: {
|
||||
makeDir,
|
||||
makeDirs,
|
||||
listJsonFiles: async () => [],
|
||||
readTextFile: async () => {
|
||||
throw new Error("unexpected readTextFile");
|
||||
},
|
||||
writeTextFile: async () => {},
|
||||
rename: async () => {},
|
||||
remove: async () => {},
|
||||
},
|
||||
queueDir,
|
||||
authorizeRequest: async () => null,
|
||||
handleRequest: async () => ({ status: 200, body: "ok" }),
|
||||
});
|
||||
|
||||
await worker.stop();
|
||||
|
||||
expect(makeDir).not.toHaveBeenCalled();
|
||||
expect(makeDirs).toHaveBeenCalledTimes(1);
|
||||
expect(makeDirs).toHaveBeenCalledWith([
|
||||
directories.rootDir,
|
||||
directories.requestsDir,
|
||||
directories.responsesDir,
|
||||
directories.logsDir,
|
||||
]);
|
||||
});
|
||||
|
||||
it("falls back to sequential makeDir when the queue client omits makeDirs", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-bridge-makedir-fallback-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
|
||||
const queueDir = path.posix.join(rootDir, "queue");
|
||||
const directories = sandboxCallbackBridgeDirectories(queueDir);
|
||||
const makeDir = vi.fn(async (_remotePath: string) => {});
|
||||
|
||||
// A queue client that predates the batched makeDirs method. The worker
|
||||
// must still create every queue directory through sequential makeDir.
|
||||
const worker = await startSandboxCallbackBridgeWorker({
|
||||
client: {
|
||||
makeDir,
|
||||
listJsonFiles: async () => [],
|
||||
readTextFile: async () => {
|
||||
throw new Error("unexpected readTextFile");
|
||||
},
|
||||
writeTextFile: async () => {},
|
||||
rename: async () => {},
|
||||
remove: async () => {},
|
||||
},
|
||||
queueDir,
|
||||
authorizeRequest: async () => null,
|
||||
handleRequest: async () => ({ status: 200, body: "ok" }),
|
||||
});
|
||||
|
||||
await worker.stop();
|
||||
|
||||
expect(makeDir.mock.calls.map((call) => call[0])).toEqual([
|
||||
directories.rootDir,
|
||||
directories.requestsDir,
|
||||
directories.responsesDir,
|
||||
directories.logsDir,
|
||||
]);
|
||||
});
|
||||
|
||||
it("runs one mkdir -p exec for makeDirs on the command-managed queue client", async () => {
|
||||
const runner = {
|
||||
execute: vi.fn(async (_input: { args?: string[] }) => ({
|
||||
exitCode: 0,
|
||||
signal: null,
|
||||
timedOut: false,
|
||||
stdout: "",
|
||||
stderr: "",
|
||||
pid: null,
|
||||
startedAt: new Date().toISOString(),
|
||||
})),
|
||||
};
|
||||
|
||||
const client = createCommandManagedSandboxCallbackBridgeQueueClient({
|
||||
runner,
|
||||
remoteCwd: "/workspace",
|
||||
timeoutMs: 30_000,
|
||||
});
|
||||
|
||||
// The command-managed client always provides the batched makeDirs method.
|
||||
expect(client.makeDirs).toBeDefined();
|
||||
await client.makeDirs?.(["/workspace/a", "/workspace/b", "/workspace/c"]);
|
||||
|
||||
expect(runner.execute).toHaveBeenCalledTimes(1);
|
||||
const call = runner.execute.mock.calls[0][0];
|
||||
const script = call.args?.[call.args.length - 1] ?? "";
|
||||
expect(script).toContain("mkdir -p");
|
||||
expect(script).toContain("/workspace/a");
|
||||
expect(script).toContain("/workspace/b");
|
||||
expect(script).toContain("/workspace/c");
|
||||
});
|
||||
});
|
||||
@@ -149,6 +149,11 @@ export interface SandboxCallbackBridgeDirectories {
|
||||
|
||||
export interface SandboxCallbackBridgeQueueClient {
|
||||
makeDir(remotePath: string): Promise<void>;
|
||||
// Optional batched directory create. The built-in clients create every
|
||||
// queue directory in one remote exec. A client that predates this method
|
||||
// omits it; the worker falls back to sequential `makeDir` calls, so an
|
||||
// external implementation stays compatible without a change.
|
||||
makeDirs?(remotePaths: string[]): Promise<void>;
|
||||
listJsonFiles(remotePath: string): Promise<string[]>;
|
||||
readTextFile(remotePath: string): Promise<string>;
|
||||
writeTextFile(remotePath: string, body: string): Promise<void>;
|
||||
@@ -359,6 +364,11 @@ export function createFileSystemSandboxCallbackBridgeQueueClient(): SandboxCallb
|
||||
makeDir: async (remotePath) => {
|
||||
await fs.mkdir(remotePath, { recursive: true });
|
||||
},
|
||||
makeDirs: async (remotePaths) => {
|
||||
for (const remotePath of remotePaths) {
|
||||
await fs.mkdir(remotePath, { recursive: true });
|
||||
}
|
||||
},
|
||||
listJsonFiles: async (remotePath) => {
|
||||
const entries = await fs.readdir(remotePath, { withFileTypes: true }).catch(() => []);
|
||||
return entries
|
||||
@@ -470,6 +480,13 @@ export function createCommandManagedSandboxCallbackBridgeQueueClient(input: {
|
||||
makeDir: async (remotePath) => {
|
||||
await runChecked(`mkdir ${remotePath}`, `mkdir -p ${shellQuote(remotePath)}`);
|
||||
},
|
||||
makeDirs: async (remotePaths) => {
|
||||
if (remotePaths.length === 0) {
|
||||
return;
|
||||
}
|
||||
const quoted = remotePaths.map((remotePath) => shellQuote(remotePath));
|
||||
await runChecked(`mkdir ${remotePaths.join(" ")}`, `mkdir -p ${quoted.join(" ")}`);
|
||||
},
|
||||
listJsonFiles: async (remotePath) => {
|
||||
const result = await runShell(
|
||||
input.runner,
|
||||
@@ -606,10 +623,21 @@ export async function startSandboxCallbackBridgeWorker(input: {
|
||||
const pollIntervalMs = normalizeTimeoutMs(input.pollIntervalMs, DEFAULT_BRIDGE_POLL_INTERVAL_MS);
|
||||
const maxBodyBytes = normalizeTimeoutMs(input.maxBodyBytes, DEFAULT_BRIDGE_MAX_BODY_BYTES);
|
||||
const directories = sandboxCallbackBridgeDirectories(input.queueDir);
|
||||
await input.client.makeDir(directories.rootDir);
|
||||
await input.client.makeDir(directories.requestsDir);
|
||||
await input.client.makeDir(directories.responsesDir);
|
||||
await input.client.makeDir(directories.logsDir);
|
||||
const queueDirectories = [
|
||||
directories.rootDir,
|
||||
directories.requestsDir,
|
||||
directories.responsesDir,
|
||||
directories.logsDir,
|
||||
];
|
||||
if (input.client.makeDirs) {
|
||||
await input.client.makeDirs(queueDirectories);
|
||||
} else {
|
||||
// Backward-compatible fallback for a queue client that omits the batched
|
||||
// makeDirs method. Create each queue directory with a single makeDir.
|
||||
for (const directory of queueDirectories) {
|
||||
await input.client.makeDir(directory);
|
||||
}
|
||||
}
|
||||
|
||||
let stopping = false;
|
||||
let inFlight = 0;
|
||||
|
||||
@@ -231,6 +231,108 @@ describe("sandbox native file sync", () => {
|
||||
expect(await readFile(path.join(prepared.assetDirs.creds, "cred.txt"), "utf8")).toBe("secret\n");
|
||||
});
|
||||
|
||||
it("stages each additional project into its own isolated dir via a native directory syncIn", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-additional-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const localWorkspaceDir = path.join(rootDir, "local-workspace");
|
||||
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||
await mkdir(localWorkspaceDir, { recursive: true });
|
||||
await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor\n", "utf8");
|
||||
|
||||
// Three referenced projects, each with a distinctive file, staged as plain
|
||||
// read-only trees.
|
||||
const projects = [
|
||||
{ projectId: "alpha", localDir: path.join(rootDir, "src-alpha"), file: "alpha.txt", body: "alpha body\n" },
|
||||
{ projectId: "bravo", localDir: path.join(rootDir, "src-bravo"), file: "bravo.txt", body: "bravo body\n" },
|
||||
{ projectId: "charlie", localDir: path.join(rootDir, "src-charlie"), file: "charlie.txt", body: "charlie body\n" },
|
||||
];
|
||||
for (const project of projects) {
|
||||
await mkdir(project.localDir, { recursive: true });
|
||||
await writeFile(path.join(project.localDir, project.file), project.body, "utf8");
|
||||
}
|
||||
|
||||
const { client, syncInOps } = makeNativeClient();
|
||||
const prepared = await prepareSandboxManagedRuntime({
|
||||
spec: { transport: "sandbox", provider: "test", sandboxId: "s1", remoteCwd: remoteWorkspaceDir, timeoutMs: 30_000, apiKey: null },
|
||||
adapterKey: "test-adapter",
|
||||
client,
|
||||
workspaceLocalDir: localWorkspaceDir,
|
||||
additionalSources: projects.map((project) => ({ localPath: project.localDir, projectId: project.projectId })),
|
||||
});
|
||||
|
||||
// Each project lands in its OWN `project-<projectId>` directory under the
|
||||
// runtime root, and its file materializes there.
|
||||
const projectDirs = projects.map((project) => {
|
||||
const dir = prepared.additionalSourceDirs[project.projectId];
|
||||
expect(dir).toBe(path.posix.join(prepared.runtimeRootDir, `project-${project.projectId}`));
|
||||
return dir;
|
||||
});
|
||||
for (const [index, project] of projects.entries()) {
|
||||
expect(await readFile(path.join(projectDirs[index], project.file), "utf8")).toBe(project.body);
|
||||
}
|
||||
|
||||
// The target dirs are pairwise distinct and never nested inside one another.
|
||||
for (const outer of projectDirs) {
|
||||
for (const inner of projectDirs) {
|
||||
if (outer === inner) continue;
|
||||
expect(inner.startsWith(`${outer}/`)).toBe(false);
|
||||
}
|
||||
}
|
||||
|
||||
// Each project rides its own `syncIn` operation as a single `directory`
|
||||
// mapping, source = the host checkout dir, target = the isolated project dir.
|
||||
const inboundOps = syncInOps.flat();
|
||||
for (const [index, project] of projects.entries()) {
|
||||
const op = inboundOps.find((candidate) =>
|
||||
candidate.files.some((mapping) => mapping.targetPath === projectDirs[index]),
|
||||
);
|
||||
expect(op).toBeDefined();
|
||||
expect(op!.operationId).toMatch(/^sync-op-\d+$/);
|
||||
expect(op!.operationId).not.toContain(project.projectId);
|
||||
expect(op!.files).toHaveLength(1);
|
||||
expect(op!.files[0]).toMatchObject({
|
||||
sourcePath: project.localDir,
|
||||
targetPath: projectDirs[index],
|
||||
kind: "directory",
|
||||
});
|
||||
// Plain read-only tree — no post-upload extract/wipe/merge command.
|
||||
expect(op!.postUploadCommands ?? []).toHaveLength(0);
|
||||
}
|
||||
});
|
||||
|
||||
it("isolates one additional project's sync failure and stages the rest", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-additional-fail-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const localWorkspaceDir = path.join(rootDir, "local-workspace");
|
||||
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||
const goodDir = path.join(rootDir, "src-good");
|
||||
await mkdir(localWorkspaceDir, { recursive: true });
|
||||
await mkdir(goodDir, { recursive: true });
|
||||
await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor\n", "utf8");
|
||||
await writeFile(path.join(goodDir, "good.txt"), "good body\n", "utf8");
|
||||
|
||||
// The middle source points at a directory that does not exist, so its native
|
||||
// transfer fails. Failure isolation must skip only it and stage the rest.
|
||||
const { client } = makeNativeClient();
|
||||
const prepared = await prepareSandboxManagedRuntime({
|
||||
spec: { transport: "sandbox", provider: "test", sandboxId: "s1", remoteCwd: remoteWorkspaceDir, timeoutMs: 30_000, apiKey: null },
|
||||
adapterKey: "test-adapter",
|
||||
client,
|
||||
workspaceLocalDir: localWorkspaceDir,
|
||||
additionalSources: [
|
||||
{ localPath: goodDir, projectId: "good-a" },
|
||||
{ localPath: path.join(rootDir, "does-not-exist"), projectId: "broken" },
|
||||
{ localPath: goodDir, projectId: "good-b" },
|
||||
],
|
||||
});
|
||||
|
||||
// Both healthy projects staged; the broken one is absent, not fatal.
|
||||
expect(Object.keys(prepared.additionalSourceDirs).sort()).toEqual(["good-a", "good-b"]);
|
||||
expect(prepared.additionalSourceDirs.broken).toBeUndefined();
|
||||
expect(await readFile(path.join(prepared.additionalSourceDirs["good-a"], "good.txt"), "utf8")).toBe("good body\n");
|
||||
expect(await readFile(path.join(prepared.additionalSourceDirs["good-b"], "good.txt"), "utf8")).toBe("good body\n");
|
||||
});
|
||||
|
||||
it("dereferences symlinks only when followSymlinks is true (native honors the flag)", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-symlink-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
|
||||
@@ -2,18 +2,24 @@ import { promises as fsPromises } from "node:fs";
|
||||
import { lstat, mkdir, mkdtemp, readFile, readdir, rm, symlink, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { execFile as execFileCallback, spawn } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { resetLocalGitIndexToHead } from "./git-workspace-sync.js";
|
||||
|
||||
import {
|
||||
assertSyncOperationsConfined,
|
||||
mirrorDirectory,
|
||||
prepareSandboxManagedRuntime,
|
||||
type SandboxManagedRuntimeClient,
|
||||
type SandboxSyncOperation,
|
||||
type SandboxSyncResult,
|
||||
} from "./sandbox-managed-runtime.js";
|
||||
import {
|
||||
prepareCommandManagedRuntime,
|
||||
type CommandManagedRuntimeRunner,
|
||||
} from "./command-managed-runtime.js";
|
||||
import type { RunProcessResult } from "./server-utils.js";
|
||||
|
||||
function toArrayBuffer(bytes: Buffer): ArrayBuffer {
|
||||
return bytes.buffer.slice(bytes.byteOffset, bytes.byteOffset + bytes.byteLength) as ArrayBuffer;
|
||||
@@ -87,6 +93,41 @@ function attachNativeRecordingSyncIn(
|
||||
};
|
||||
}
|
||||
|
||||
// A capturing `syncIn` that records every operation for assertion and
|
||||
// materializes file AND directory mappings. A directory mapping (a referenced
|
||||
// project) uses `mirrorDirectory`, so a test can assert the advisory `access`
|
||||
// intent on directory mappings as well as file mappings.
|
||||
function attachCapturingSyncIn(
|
||||
client: SandboxManagedRuntimeClient,
|
||||
captured: SandboxSyncOperation[],
|
||||
): void {
|
||||
client.syncIn = async (operations: SandboxSyncOperation[]): Promise<SandboxSyncResult> => {
|
||||
const resultOperations: SandboxSyncResult["operations"] = [];
|
||||
for (const operation of operations) {
|
||||
captured.push(operation);
|
||||
let filesTransferred = 0;
|
||||
let bytesTransferred = 0;
|
||||
for (const mapping of operation.files) {
|
||||
await mkdir(path.posix.dirname(mapping.targetPath), { recursive: true });
|
||||
if (mapping.kind === "directory") {
|
||||
await mirrorDirectory(mapping.sourcePath, mapping.targetPath);
|
||||
} else {
|
||||
const bytes = await readFile(mapping.sourcePath);
|
||||
await writeFile(mapping.targetPath, bytes);
|
||||
if (mapping.mode != null) await fsPromises.chmod(mapping.targetPath, mapping.mode);
|
||||
bytesTransferred += bytes.byteLength;
|
||||
}
|
||||
filesTransferred += 1;
|
||||
}
|
||||
for (const command of operation.postUploadCommands ?? []) {
|
||||
await execFile("sh", ["-c", command.command], { maxBuffer: 32 * 1024 * 1024 });
|
||||
}
|
||||
resultOperations.push({ operationId: operation.operationId, filesTransferred, bytesTransferred });
|
||||
}
|
||||
return { operations: resultOperations };
|
||||
};
|
||||
}
|
||||
|
||||
vi.mock("node:fs", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("node:fs")>();
|
||||
return {
|
||||
@@ -438,9 +479,13 @@ describe("sandbox managed runtime", () => {
|
||||
"restore",
|
||||
"finalize",
|
||||
]));
|
||||
// Git history and workspace overlay sync as ONE merged operation, so a single
|
||||
// transfer-progress event rides the config_sync (workspace) phase. The git_sync
|
||||
// phase still emits its plain status message (asserted by the arrayContaining
|
||||
// check above).
|
||||
expect(runtimeStatuses.some((status) => (
|
||||
status.phase === "git_sync" &&
|
||||
/^Syncing git history to sandbox: 100% \(\d+\.\d\/\d+\.\d MB\)$/.test(status.message)
|
||||
status.phase === "config_sync" &&
|
||||
/^Syncing workspace to sandbox: 100% \(\d+\.\d\/\d+\.\d MB\)$/.test(status.message)
|
||||
))).toBe(true);
|
||||
expect(runtimeStatuses.some((status) => (
|
||||
status.phase === "export" &&
|
||||
@@ -1272,6 +1317,18 @@ describe("sandbox managed runtime", () => {
|
||||
`rm -rf ${q(path.posix.join(prepared.runtimeRootDir, "widget"))} && mkdir -p ${q(path.posix.join(prepared.runtimeRootDir, "widget"))}`,
|
||||
);
|
||||
|
||||
// The asset tar carries the asset directory as its read-write destination,
|
||||
// because the extract command fills that directory, not the staging archive.
|
||||
const assetTarMapping = assetOp!.files.find((mapping) => mapping.targetPath.endsWith("widget-upload.tar"));
|
||||
expect(assetTarMapping?.access).toBe("rw");
|
||||
expect(assetTarMapping?.writablePath).toBe(prepared.assetDirs.widget);
|
||||
|
||||
// The staged helper file is a read-only input that the command consumes, so
|
||||
// it is `access: "ro"` and never joins the writable set.
|
||||
const stageMapping = assetOp!.files.find((mapping) => mapping.targetPath.endsWith("widget-helper.sh"));
|
||||
expect(stageMapping?.access).toBe("ro");
|
||||
expect(stageMapping?.writablePath).toBeUndefined();
|
||||
|
||||
// The asset actually materialized through the native seam.
|
||||
await expect(readFile(path.join(prepared.assetDirs.widget, "seed.txt"), "utf8")).resolves.toBe("seed\n");
|
||||
});
|
||||
@@ -1337,21 +1394,18 @@ describe("sandbox managed runtime", () => {
|
||||
expect(directWrites).toEqual([]);
|
||||
expect(directRuns).toEqual([]);
|
||||
|
||||
// Two operations: git-workspace then workspace overlay. Each uploads a single
|
||||
// tar as a `file` mapping and carries its extract as an ordered post-command.
|
||||
expect(captured.length).toBeGreaterThanOrEqual(2);
|
||||
const gitOp = captured.find((op) =>
|
||||
op.files.some((mapping) => mapping.targetPath.endsWith("git-workspace-upload.tar")),
|
||||
);
|
||||
const workspaceOp = captured.find((op) =>
|
||||
op.files.some((mapping) => mapping.targetPath.endsWith("workspace-upload.tar")),
|
||||
);
|
||||
expect(gitOp).toBeDefined();
|
||||
expect(workspaceOp).toBeDefined();
|
||||
expect(gitOp!.files.every((mapping) => mapping.kind === "file")).toBe(true);
|
||||
// The git operation's post-upload command preserves `.paperclip-runtime` while
|
||||
// replacing the rest of the tree (wipe-except-preserved), then untars.
|
||||
const gitCommand = gitOp!.postUploadCommands![0].command;
|
||||
// One merged operation carries BOTH the git-history and workspace-overlay tars
|
||||
// as two `file` mappings, each with its extract as an ordered post-command.
|
||||
expect(captured).toHaveLength(1);
|
||||
const op = captured[0];
|
||||
const byBase = (base: string) =>
|
||||
op.files.find((mapping) => path.posix.basename(mapping.targetPath) === base);
|
||||
expect(byBase("git-workspace-upload.tar")).toBeDefined();
|
||||
expect(byBase("workspace-upload.tar")).toBeDefined();
|
||||
expect(op.files.every((mapping) => mapping.kind === "file")).toBe(true);
|
||||
// The first post-upload command extracts the git history and preserves
|
||||
// `.paperclip-runtime` while replacing the rest of the tree (wipe-except-preserved).
|
||||
const gitCommand = op.postUploadCommands![0].command;
|
||||
expect(gitCommand).toContain(".paperclip-runtime");
|
||||
expect(gitCommand).toContain("tar -xf");
|
||||
|
||||
@@ -1363,16 +1417,168 @@ describe("sandbox managed runtime", () => {
|
||||
expect(prepared.workspaceRemoteDir).toBe(remoteWorkspaceDir);
|
||||
});
|
||||
|
||||
// Regression lock: a representative `codex_local` start stages its inbound
|
||||
// bytes — git history, workspace overlay, and the managed Codex `home` asset
|
||||
// (auth.json merge) — as EXACTLY ONE `syncIn` operation each. Every inbound step
|
||||
// is routed through `client.syncIn` (one native `uploadFiles` round-trip per
|
||||
// operation, with the extract/merge carried as provider-executed
|
||||
// `postUploadCommands`), with no separate custom-provision diversion. Assert
|
||||
// the collapsed round-trip count so a future change that re-inlines a
|
||||
// `writeFile`+`run` sequence — or fans one staging step across multiple
|
||||
// operations — fails loudly here instead of silently regressing the start path.
|
||||
it("collapses a representative codex_local start to one syncIn round-trip per inbound staging step", async () => {
|
||||
it("issues one merged syncIn operation for a git-backed workspace stage-sync with two ordered extract commands", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-merged-git-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const sourceRepoDir = path.join(rootDir, "source-repo");
|
||||
const localWorkspaceDir = path.join(rootDir, "local-workspace");
|
||||
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||
await mkdir(sourceRepoDir, { recursive: true });
|
||||
await git(sourceRepoDir, ["init"]);
|
||||
await git(sourceRepoDir, ["checkout", "-b", "main"]);
|
||||
await git(sourceRepoDir, ["config", "user.name", "Paperclip Test"]);
|
||||
await git(sourceRepoDir, ["config", "user.email", "test@paperclip.dev"]);
|
||||
await writeFile(path.join(sourceRepoDir, "tracked.txt"), "tracked\n", "utf8");
|
||||
await git(sourceRepoDir, ["add", "tracked.txt"]);
|
||||
await git(sourceRepoDir, ["commit", "-m", "base"]);
|
||||
await git(sourceRepoDir, ["worktree", "add", "-b", "work", localWorkspaceDir, "HEAD"]);
|
||||
// Pre-seed the sandbox with a `.paperclip-runtime` dir that MUST survive.
|
||||
await mkdir(path.join(remoteWorkspaceDir, ".paperclip-runtime"), { recursive: true });
|
||||
await writeFile(path.join(remoteWorkspaceDir, ".paperclip-runtime", "keep.txt"), "keep\n", "utf8");
|
||||
|
||||
const client: SandboxManagedRuntimeClient = {
|
||||
makeDir: async (remotePath) => {
|
||||
await mkdir(remotePath, { recursive: true });
|
||||
},
|
||||
writeFile: async (remotePath, bytes) => {
|
||||
await mkdir(path.dirname(remotePath), { recursive: true });
|
||||
await writeFile(remotePath, Buffer.from(bytes));
|
||||
},
|
||||
readFile: async (remotePath) => await readFile(remotePath),
|
||||
listFiles: async () => [],
|
||||
remove: async (remotePath) => {
|
||||
await rm(remotePath, { recursive: true, force: true });
|
||||
},
|
||||
run: async (command) => {
|
||||
await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 });
|
||||
},
|
||||
};
|
||||
const captured: SandboxSyncOperation[] = [];
|
||||
attachNativeRecordingSyncIn(client, captured);
|
||||
// Count how many times `syncIn` is invoked so the merge collapses the two
|
||||
// workspace staging steps into a single native round trip.
|
||||
let syncInCallCount = 0;
|
||||
const recordingSyncIn = client.syncIn!;
|
||||
client.syncIn = async (operations) => {
|
||||
syncInCallCount += 1;
|
||||
return recordingSyncIn(operations);
|
||||
};
|
||||
|
||||
await prepareSandboxManagedRuntime({
|
||||
spec: {
|
||||
transport: "sandbox",
|
||||
provider: "test",
|
||||
sandboxId: "sandbox-1",
|
||||
remoteCwd: remoteWorkspaceDir,
|
||||
timeoutMs: 30_000,
|
||||
apiKey: null,
|
||||
},
|
||||
adapterKey: "test-adapter",
|
||||
client,
|
||||
workspaceLocalDir: localWorkspaceDir,
|
||||
});
|
||||
|
||||
// One `syncIn` call carrying exactly one operation for the whole workspace.
|
||||
expect(syncInCallCount).toBe(1);
|
||||
expect(captured).toHaveLength(1);
|
||||
const op = captured[0];
|
||||
|
||||
// Both host tars ride the one operation as two `file` mappings.
|
||||
const byBase = (base: string) =>
|
||||
op.files.find((mapping) => path.posix.basename(mapping.targetPath) === base);
|
||||
const gitMapping = byBase("git-workspace-upload.tar");
|
||||
const overlayMapping = byBase("workspace-upload.tar");
|
||||
expect(gitMapping).toBeDefined();
|
||||
expect(overlayMapping).toBeDefined();
|
||||
expect(op.files).toHaveLength(2);
|
||||
expect(op.files.every((mapping) => mapping.kind === "file")).toBe(true);
|
||||
|
||||
// Both tar targets live under `.paperclip-runtime`, so the git extract's wipe
|
||||
// (which preserves `.paperclip-runtime`) cannot delete the overlay tar before
|
||||
// the overlay extract runs.
|
||||
for (const mapping of op.files) {
|
||||
expect(mapping.targetPath).toContain("/.paperclip-runtime/");
|
||||
}
|
||||
|
||||
// Two ordered extract commands: git history first (wipe-except-preserved),
|
||||
// overlay second (merge, no wipe). No deleted paths in this clean worktree.
|
||||
const commands = op.postUploadCommands ?? [];
|
||||
expect(commands).toHaveLength(2);
|
||||
expect(commands[0].command).toContain("git-workspace-upload.tar");
|
||||
expect(commands[0].command).toContain(".paperclip-runtime");
|
||||
expect(commands[0].command).toContain("find ");
|
||||
expect(commands[1].command).toContain("workspace-upload.tar");
|
||||
expect(commands[1].command).not.toContain("git-workspace-upload.tar");
|
||||
expect(commands[1].command).not.toContain("find ");
|
||||
|
||||
// The pre-seeded runtime dir survived and the workspace overlay applied.
|
||||
await expect(
|
||||
readFile(path.join(remoteWorkspaceDir, ".paperclip-runtime", "keep.txt"), "utf8"),
|
||||
).resolves.toBe("keep\n");
|
||||
await expect(readFile(path.join(remoteWorkspaceDir, "tracked.txt"), "utf8")).resolves.toBe("tracked\n");
|
||||
});
|
||||
|
||||
it("the merged workspace confine guard covers both tar mappings (escape in either trips it)", () => {
|
||||
const runtimeRoot = "/home/daytona/paperclip-workspace/.paperclip-runtime/test-adapter";
|
||||
const tempRoot = "/tmp/paperclip-sandbox-sync-abc";
|
||||
const gitMapping = {
|
||||
sourcePath: `${tempRoot}/git-workspace.tar`,
|
||||
targetPath: `${runtimeRoot}/git-workspace-upload.tar`,
|
||||
kind: "file" as const,
|
||||
};
|
||||
const overlayMapping = {
|
||||
sourcePath: `${tempRoot}/workspace.tar`,
|
||||
targetPath: `${runtimeRoot}/workspace-upload.tar`,
|
||||
kind: "file" as const,
|
||||
};
|
||||
const roots = { sourceRoots: [tempRoot], targetRoots: [runtimeRoot] };
|
||||
|
||||
// A confined merged operation with both tar mappings passes the guard.
|
||||
expect(() =>
|
||||
assertSyncOperationsConfined(
|
||||
[{ operationId: "merged", files: [gitMapping, overlayMapping] }],
|
||||
roots,
|
||||
),
|
||||
).not.toThrow();
|
||||
|
||||
// A `..` target escape in the overlay mapping trips the guard, so the whole
|
||||
// merged operation is rejected before any transfer.
|
||||
expect(() =>
|
||||
assertSyncOperationsConfined(
|
||||
[{
|
||||
operationId: "merged",
|
||||
files: [
|
||||
gitMapping,
|
||||
{ ...overlayMapping, targetPath: `${runtimeRoot}/../../etc/workspace-upload.tar` },
|
||||
],
|
||||
}],
|
||||
roots,
|
||||
),
|
||||
).toThrow(/escapes its confinement root|not a confined absolute path/);
|
||||
|
||||
// An absolute-path source escape in the git mapping trips the guard too.
|
||||
expect(() =>
|
||||
assertSyncOperationsConfined(
|
||||
[{
|
||||
operationId: "merged",
|
||||
files: [{ ...gitMapping, sourcePath: "/etc/passwd" }, overlayMapping],
|
||||
}],
|
||||
roots,
|
||||
),
|
||||
).toThrow(/escapes its confinement root|not a confined absolute path/);
|
||||
});
|
||||
|
||||
// Regression lock: a representative `codex_local` start stages its inbound bytes
|
||||
// as TWO `syncIn` operations. The git-history and workspace-overlay tars share
|
||||
// ONE merged operation (one native `uploadFiles` round-trip that carries both
|
||||
// tars, with the two extract commands as ordered `postUploadCommands`); the
|
||||
// managed Codex `home` asset (auth.json merge) is the second operation. Every
|
||||
// inbound step is routed through `client.syncIn`, with no separate
|
||||
// custom-provision diversion. Assert the collapsed round-trip count so a future
|
||||
// change that re-inlines a `writeFile`+`run` sequence — or splits the merged
|
||||
// workspace operation back into two — fails loudly here instead of silently
|
||||
// regressing the start path.
|
||||
it("collapses a representative codex_local start to two syncIn round-trips: one merged workspace op plus the asset op", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-codex-roundtrip-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const sourceRepoDir = path.join(rootDir, "source-repo");
|
||||
@@ -1380,7 +1586,7 @@ describe("sandbox managed runtime", () => {
|
||||
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||
const homeDir = path.join(rootDir, "codex-home");
|
||||
|
||||
// Git-backed workspace → git history + workspace overlay are two staging steps.
|
||||
// Git-backed workspace → git history + workspace overlay share one merged op.
|
||||
await mkdir(sourceRepoDir, { recursive: true });
|
||||
await git(sourceRepoDir, ["init"]);
|
||||
await git(sourceRepoDir, ["checkout", "-b", "main"]);
|
||||
@@ -1453,36 +1659,168 @@ describe("sandbox managed runtime", () => {
|
||||
expect(directWrites).toEqual([]);
|
||||
expect(directRuns).toEqual([]);
|
||||
|
||||
// The collapsed count: exactly three inbound round-trips — git, workspace, home.
|
||||
expect(captured).toHaveLength(3);
|
||||
const gitOp = captured.find((op) =>
|
||||
op.files.some((mapping) => mapping.targetPath.endsWith("git-workspace-upload.tar")),
|
||||
);
|
||||
const workspaceOp = captured.find((op) =>
|
||||
op.files.some((mapping) => mapping.targetPath.endsWith("workspace-upload.tar")),
|
||||
);
|
||||
const homeOp = captured.find((op) =>
|
||||
op.files.some((mapping) => mapping.targetPath.endsWith("home-upload.tar")),
|
||||
);
|
||||
expect(gitOp).toBeDefined();
|
||||
// The collapsed count: exactly two inbound round-trips — the merged workspace
|
||||
// op (git history + overlay) and the home asset op.
|
||||
expect(captured).toHaveLength(2);
|
||||
const hasBase = (op: SandboxSyncOperation, base: string) =>
|
||||
op.files.some((mapping) => path.posix.basename(mapping.targetPath) === base);
|
||||
const workspaceOp = captured.find((op) => hasBase(op, "workspace-upload.tar"));
|
||||
const homeOp = captured.find((op) => hasBase(op, "home-upload.tar"));
|
||||
expect(workspaceOp).toBeDefined();
|
||||
expect(homeOp).toBeDefined();
|
||||
// The merged workspace op carries BOTH the git-history and overlay tars, with
|
||||
// both extract commands as ordered post-upload commands (git first, overlay
|
||||
// second). The two tars ride one native uploadFiles round-trip.
|
||||
expect(hasBase(workspaceOp!, "git-workspace-upload.tar")).toBe(true);
|
||||
expect(workspaceOp!.files).toHaveLength(2);
|
||||
expect((workspaceOp!.postUploadCommands ?? []).length).toBeGreaterThanOrEqual(2);
|
||||
|
||||
// Every operation is a single native uploadFiles (all `file` mappings) whose
|
||||
// Every operation is a native uploadFiles (all `file` mappings) whose
|
||||
// extract/merge rides as an ordered provider-executed post-upload command.
|
||||
for (const op of captured) {
|
||||
expect(op.files.length).toBeGreaterThanOrEqual(1);
|
||||
expect(op.files.every((mapping) => mapping.kind === "file")).toBe(true);
|
||||
expect(op.postUploadCommands ?? []).not.toHaveLength(0);
|
||||
}
|
||||
// Operation ids are distinct, so "3 operations" is 3 real round-trips.
|
||||
expect(new Set(captured.map((op) => op.operationId)).size).toBe(3);
|
||||
// Operation ids are distinct, so "2 operations" is 2 real round-trips.
|
||||
expect(new Set(captured.map((op) => op.operationId)).size).toBe(2);
|
||||
|
||||
// The credential asset actually materialized through the native seam.
|
||||
await expect(readFile(path.join(prepared.assetDirs.home, "auth.json"), "utf8"))
|
||||
.resolves.toBe("{\"OPENAI_API_KEY\":\"sk-test\"}\n");
|
||||
});
|
||||
|
||||
it("authors the advisory access intent rw on workspace, git, and asset inbound mappings", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-access-rw-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const sourceRepoDir = path.join(rootDir, "source-repo");
|
||||
const localWorkspaceDir = path.join(rootDir, "local-workspace");
|
||||
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||
const assetDir = path.join(rootDir, "asset-home");
|
||||
|
||||
// A git-backed workspace produces both a git-history tar and an overlay tar.
|
||||
await mkdir(sourceRepoDir, { recursive: true });
|
||||
await git(sourceRepoDir, ["init"]);
|
||||
await git(sourceRepoDir, ["checkout", "-b", "main"]);
|
||||
await git(sourceRepoDir, ["config", "user.name", "Paperclip Test"]);
|
||||
await git(sourceRepoDir, ["config", "user.email", "test@paperclip.dev"]);
|
||||
await writeFile(path.join(sourceRepoDir, "tracked.txt"), "tracked\n", "utf8");
|
||||
await git(sourceRepoDir, ["add", "tracked.txt"]);
|
||||
await git(sourceRepoDir, ["commit", "-m", "base"]);
|
||||
await git(sourceRepoDir, ["worktree", "add", "-b", "work", localWorkspaceDir, "HEAD"]);
|
||||
|
||||
await mkdir(assetDir, { recursive: true });
|
||||
await writeFile(path.join(assetDir, "config.toml"), "model = \"gpt\"\n", "utf8");
|
||||
|
||||
const client: SandboxManagedRuntimeClient = {
|
||||
makeDir: async (remotePath) => { await mkdir(remotePath, { recursive: true }); },
|
||||
writeFile: async (remotePath, bytes) => {
|
||||
await mkdir(path.dirname(remotePath), { recursive: true });
|
||||
await writeFile(remotePath, Buffer.from(bytes));
|
||||
},
|
||||
readFile: async (remotePath) => await readFile(remotePath),
|
||||
listFiles: async () => [],
|
||||
remove: async (remotePath) => { await rm(remotePath, { recursive: true, force: true }); },
|
||||
run: async (command) => { await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 }); },
|
||||
};
|
||||
const captured: SandboxSyncOperation[] = [];
|
||||
attachCapturingSyncIn(client, captured);
|
||||
|
||||
await prepareSandboxManagedRuntime({
|
||||
spec: {
|
||||
transport: "sandbox",
|
||||
provider: "test",
|
||||
sandboxId: "sandbox-1",
|
||||
remoteCwd: remoteWorkspaceDir,
|
||||
timeoutMs: 30_000,
|
||||
apiKey: null,
|
||||
},
|
||||
adapterKey: "test-adapter",
|
||||
client,
|
||||
workspaceLocalDir: localWorkspaceDir,
|
||||
assets: [{ key: "home", localDir: assetDir }],
|
||||
});
|
||||
|
||||
const findMapping = (base: string) =>
|
||||
captured
|
||||
.flatMap((op) => op.files)
|
||||
.find((mapping) => path.posix.basename(mapping.targetPath) === base);
|
||||
|
||||
// The workspace, git-history, and asset destinations receive read-write bytes,
|
||||
// so the author marks each mapping `access: "rw"`.
|
||||
expect(findMapping("workspace-upload.tar")?.access).toBe("rw");
|
||||
expect(findMapping("git-workspace-upload.tar")?.access).toBe("rw");
|
||||
expect(findMapping("home-upload.tar")?.access).toBe("rw");
|
||||
|
||||
// Each tar mapping uploads a staging archive under the runtime root, so its
|
||||
// `targetPath` is not the read-write destination. `writablePath` names the
|
||||
// directory that the post-upload extract command fills: the workspace
|
||||
// directory for the workspace and git tars, and the asset directory for the
|
||||
// asset tar.
|
||||
const remoteAssetDir = path.posix.join(remoteWorkspaceDir, ".paperclip-runtime", "test-adapter", "home");
|
||||
expect(findMapping("workspace-upload.tar")?.writablePath).toBe(remoteWorkspaceDir);
|
||||
expect(findMapping("git-workspace-upload.tar")?.writablePath).toBe(remoteWorkspaceDir);
|
||||
expect(findMapping("home-upload.tar")?.writablePath).toBe(remoteAssetDir);
|
||||
});
|
||||
|
||||
it("authors the advisory access intent ro on referenced-project inbound mappings", async () => {
|
||||
const flagKey = "PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC";
|
||||
const priorFlag = process.env[flagKey];
|
||||
process.env[flagKey] = "1";
|
||||
try {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-access-ro-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const localWorkspaceDir = path.join(rootDir, "local-workspace");
|
||||
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||
const referencedDir = path.join(rootDir, "referenced-project");
|
||||
await mkdir(localWorkspaceDir, { recursive: true });
|
||||
await mkdir(referencedDir, { recursive: true });
|
||||
await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor\n", "utf8");
|
||||
await writeFile(path.join(referencedDir, "notes.md"), "reference\n", "utf8");
|
||||
|
||||
const client: SandboxManagedRuntimeClient = {
|
||||
makeDir: async (remotePath) => { await mkdir(remotePath, { recursive: true }); },
|
||||
writeFile: async (remotePath, bytes) => {
|
||||
await mkdir(path.dirname(remotePath), { recursive: true });
|
||||
await writeFile(remotePath, Buffer.from(bytes));
|
||||
},
|
||||
readFile: async (remotePath) => await readFile(remotePath),
|
||||
listFiles: async () => [],
|
||||
remove: async (remotePath) => { await rm(remotePath, { recursive: true, force: true }); },
|
||||
run: async (command) => { await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 }); },
|
||||
};
|
||||
const captured: SandboxSyncOperation[] = [];
|
||||
attachCapturingSyncIn(client, captured);
|
||||
|
||||
await prepareSandboxManagedRuntime({
|
||||
spec: {
|
||||
transport: "sandbox",
|
||||
provider: "test",
|
||||
sandboxId: "sandbox-1",
|
||||
remoteCwd: remoteWorkspaceDir,
|
||||
timeoutMs: 30_000,
|
||||
apiKey: null,
|
||||
},
|
||||
adapterKey: "test-adapter",
|
||||
client,
|
||||
workspaceLocalDir: localWorkspaceDir,
|
||||
additionalSources: [{ localPath: referencedDir, projectId: "proj-first" }],
|
||||
});
|
||||
|
||||
const referencedMapping = captured
|
||||
.flatMap((op) => op.files)
|
||||
.find((mapping) => path.posix.basename(mapping.targetPath) === "project-proj-first");
|
||||
|
||||
// A referenced project is a read-only tree, so the author marks it `access: "ro"`.
|
||||
expect(referencedMapping).toBeDefined();
|
||||
expect(referencedMapping?.kind).toBe("directory");
|
||||
expect(referencedMapping?.access).toBe("ro");
|
||||
} finally {
|
||||
if (priorFlag === undefined) delete process.env[flagKey];
|
||||
else process.env[flagKey] = priorFlag;
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps the sandbox runtime core free of Codex-specific string literals", async () => {
|
||||
const coreSource = await readFile(new URL("./sandbox-managed-runtime.ts", import.meta.url), "utf8");
|
||||
// The seam must be generic: no adapter (Codex) knowledge may live in the core.
|
||||
@@ -1490,4 +1828,97 @@ describe("sandbox managed runtime", () => {
|
||||
expect(coreSource).not.toMatch(/auth\.json/i);
|
||||
expect(coreSource).not.toMatch(/merge-extract|merge-decision/i);
|
||||
});
|
||||
|
||||
// End-to-end: drive the WHOLE prepare path — `prepareCommandManagedRuntime`
|
||||
// building the client, syncing the anchor workspace, then staging the
|
||||
// referenced projects — through a runner that runs real shell commands on the
|
||||
// host filesystem (host FS stands in for the sandbox FS). The runner exposes no
|
||||
// native syncIn, so staging rides the base64/tar fallback, the same transport a
|
||||
// provider without native sync uses. In production the kill-switch
|
||||
// `PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC` gates whether run prep resolves any
|
||||
// referenced projects (OFF ⇒ none reach this layer). Enable it in-test only to
|
||||
// model the ON scenario, and prove multi-project isolation plus one-failure
|
||||
// isolation end-to-end.
|
||||
it("stages multiple referenced projects into isolated sandbox dirs end-to-end, skipping a failing source", async () => {
|
||||
const flagKey = "PAPERCLIP_MULTI_PROJECT_WORKSPACE_SYNC";
|
||||
const priorFlag = process.env[flagKey];
|
||||
process.env[flagKey] = "1";
|
||||
try {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-e2e-additional-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
|
||||
const localWorkspaceDir = path.join(rootDir, "local-workspace");
|
||||
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||
await mkdir(localWorkspaceDir, { recursive: true });
|
||||
await mkdir(remoteWorkspaceDir, { recursive: true });
|
||||
await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor content\n", "utf8");
|
||||
|
||||
// Two real referenced-project checkouts (one with a nested file) plus a
|
||||
// deliberately-missing source between them.
|
||||
const first = path.join(rootDir, "referenced-first");
|
||||
const second = path.join(rootDir, "referenced-second");
|
||||
await mkdir(path.join(first, "docs"), { recursive: true });
|
||||
await mkdir(second, { recursive: true });
|
||||
await writeFile(path.join(first, "docs", "guide.md"), "first guide\n", "utf8");
|
||||
await writeFile(path.join(second, "notes.md"), "second notes\n", "utf8");
|
||||
|
||||
const runner: CommandManagedRuntimeRunner = {
|
||||
execute: (input) =>
|
||||
new Promise<RunProcessResult>((resolve) => {
|
||||
const startedAt = new Date().toISOString();
|
||||
const command =
|
||||
input.command === "sh" ? "/bin/sh" : input.command === "bash" ? "/bin/bash" : input.command;
|
||||
const child = spawn(command, input.args ?? [], { cwd: input.cwd, env: { ...process.env, ...input.env } });
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
child.stdout.on("data", (chunk) => { stdout += chunk.toString("utf8"); });
|
||||
child.stderr.on("data", (chunk) => { stderr += chunk.toString("utf8"); });
|
||||
child.on("error", () => resolve({ exitCode: 127, signal: null, timedOut: false, stdout, stderr, pid: null, startedAt }));
|
||||
child.on("close", (code) => resolve({ exitCode: code ?? 0, signal: null, timedOut: false, stdout, stderr, pid: child.pid ?? null, startedAt }));
|
||||
if (input.stdin != null) child.stdin.write(input.stdin);
|
||||
child.stdin.end();
|
||||
}),
|
||||
};
|
||||
|
||||
const prepared = await prepareCommandManagedRuntime({
|
||||
runner,
|
||||
spec: { remoteCwd: remoteWorkspaceDir, timeoutMs: 30_000 },
|
||||
adapterKey: "test-adapter",
|
||||
workspaceLocalDir: localWorkspaceDir,
|
||||
additionalSources: [
|
||||
{ localPath: first, projectId: "proj-first" },
|
||||
{ localPath: path.join(rootDir, "referenced-missing"), projectId: "proj-missing" },
|
||||
{ localPath: second, projectId: "proj-second" },
|
||||
],
|
||||
});
|
||||
|
||||
const runtimeRootDir = path.posix.join(remoteWorkspaceDir, ".paperclip-runtime", "test-adapter");
|
||||
|
||||
// The anchor workspace synced normally and stays byte-identical.
|
||||
await expect(readFile(path.join(remoteWorkspaceDir, "README.md"), "utf8")).resolves.toBe("anchor content\n");
|
||||
|
||||
// Each healthy referenced project landed in its OWN isolated dir; the
|
||||
// missing one is skipped, not fatal.
|
||||
expect(Object.keys(prepared.additionalSourceDirs).sort()).toEqual(["proj-first", "proj-second"]);
|
||||
expect(prepared.additionalSourceDirs["proj-first"]).toBe(path.posix.join(runtimeRootDir, "project-proj-first"));
|
||||
expect(prepared.additionalSourceDirs["proj-second"]).toBe(path.posix.join(runtimeRootDir, "project-proj-second"));
|
||||
expect(prepared.additionalSourceDirs["proj-missing"]).toBeUndefined();
|
||||
|
||||
await expect(readFile(path.join(prepared.additionalSourceDirs["proj-first"], "docs", "guide.md"), "utf8"))
|
||||
.resolves.toBe("first guide\n");
|
||||
await expect(readFile(path.join(prepared.additionalSourceDirs["proj-second"], "notes.md"), "utf8"))
|
||||
.resolves.toBe("second notes\n");
|
||||
|
||||
// Neither project's tree leaked into the anchor workspace or into the other
|
||||
// project's dir.
|
||||
await expect(readFile(path.join(remoteWorkspaceDir, "notes.md"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(readFile(path.join(prepared.additionalSourceDirs["proj-first"], "notes.md"), "utf8")).rejects
|
||||
.toMatchObject({ code: "ENOENT" });
|
||||
await expect(readFile(path.join(runtimeRootDir, "project-proj-missing"), "utf8")).rejects
|
||||
.toMatchObject({ code: "ENOENT" });
|
||||
} finally {
|
||||
if (priorFlag === undefined) delete process.env[flagKey];
|
||||
else process.env[flagKey] = priorFlag;
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -121,6 +121,22 @@ export interface SandboxManagedRuntimeAsset {
|
||||
restore?: (ctx: SandboxManagedRuntimeAssetRestoreContext) => Promise<void>;
|
||||
}
|
||||
|
||||
/**
|
||||
* A referenced (additional) project to stage into the run sandbox as a plain,
|
||||
* read-only tree. `localPath` is the host checkout directory. Upstream code
|
||||
* already authorized and realized this directory (`project:read`); this layer
|
||||
* adds no authorization logic. `projectId` names the isolated remote
|
||||
* subdirectory (`project-<projectId>` under the runtime root) the tree lands in.
|
||||
*
|
||||
* Additional sources are plain trees only. They never carry the anchor
|
||||
* workspace's git-history, overlay, or `.paperclip-runtime` preservation
|
||||
* semantics — those stay anchor-only.
|
||||
*/
|
||||
export interface SandboxAdditionalSource {
|
||||
localPath: string;
|
||||
projectId: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-call byte-level progress hook. `transferredBytes`/`totalBytes` are decoded
|
||||
* file bytes (not the base64 wire size). `totalBytes` is null when the size is
|
||||
@@ -145,6 +161,24 @@ export interface SandboxSyncFileMapping {
|
||||
mode?: number;
|
||||
exclude?: string[];
|
||||
followSymlinks?: boolean;
|
||||
/**
|
||||
* Advisory read-write intent for the sandbox target. `"rw"` marks a target the
|
||||
* agent may change and keep; `"ro"` marks a read-only tree. An absent value
|
||||
* defaults to `"ro"` (read-only is the safe default for an advisory signal).
|
||||
* The field is advisory metadata for an optional sandbox feedback wrapper. It
|
||||
* does not change the transfer and adds no security.
|
||||
*/
|
||||
access?: "rw" | "ro";
|
||||
/**
|
||||
* The sandbox directory that becomes read-write when `access` is `"rw"` and a
|
||||
* post-upload command extracts `targetPath` into a different directory. A tar
|
||||
* mapping uploads an archive under the runtime root, so its `targetPath` is the
|
||||
* staging archive, not the directory the extract command fills. This field
|
||||
* names that final destination directory. When absent, the read-write
|
||||
* destination is the parent directory of `targetPath`. Advisory; ignored when
|
||||
* `access` is not `"rw"`.
|
||||
*/
|
||||
writablePath?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -207,7 +241,7 @@ export interface SandboxManagedRuntimeClient {
|
||||
): Promise<Buffer | Uint8Array | ArrayBuffer>;
|
||||
listFiles(remotePath: string): Promise<string[]>;
|
||||
remove(remotePath: string): Promise<void>;
|
||||
run(command: string, options: { timeoutMs: number; noProfile?: boolean }): Promise<void>;
|
||||
run(command: string, options: { timeoutMs: number }): Promise<void>;
|
||||
/**
|
||||
* Optional native inbound transfer. Present only when the sandbox provider
|
||||
* advertises both `environmentSyncIn` and `environmentSyncOut`; otherwise the
|
||||
@@ -259,6 +293,13 @@ export interface PreparedSandboxManagedRuntime {
|
||||
workspaceRemoteDir: string;
|
||||
runtimeRootDir: string;
|
||||
assetDirs: Record<string, string>;
|
||||
/**
|
||||
* Remote directory of each additional (referenced) project that staged
|
||||
* successfully, keyed by `projectId`. A project whose staging failed is
|
||||
* absent (per-project failure isolation). Empty when no additional sources
|
||||
* were requested.
|
||||
*/
|
||||
additionalSourceDirs: Record<string, string>;
|
||||
restoreWorkspace(onProgress?: RuntimeProgressSink): Promise<void>;
|
||||
}
|
||||
|
||||
@@ -637,6 +678,12 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
workspaceExclude?: string[];
|
||||
preserveAbsentOnRestore?: string[];
|
||||
assets?: SandboxManagedRuntimeAsset[];
|
||||
/**
|
||||
* Referenced (additional) projects to stage into the sandbox as plain,
|
||||
* read-only trees, each in its own isolated `project-<projectId>` directory.
|
||||
* Defaults to none, so a legacy/anchor-only call is behavior-identical.
|
||||
*/
|
||||
additionalSources?: SandboxAdditionalSource[];
|
||||
// Upload progress sink. Threaded for the byte-counting transport rewrite; the
|
||||
// child task wires it into writeFile/readFile.
|
||||
onProgress?: RuntimeProgressSink;
|
||||
@@ -686,6 +733,14 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
// Opaque, ordered, non-sensitive operation tokens — never a caller/asset id.
|
||||
const nextSyncOperationId = () => `sync-op-${++syncOperationSeq}`;
|
||||
|
||||
// Remote directory of each additional (referenced) project that stages
|
||||
// successfully, keyed by projectId. A project that fails to stage is absent.
|
||||
const additionalSourceDirs: Record<string, string> = {};
|
||||
// Additional projects stage as plain trees. Drop the heavy build/cache dirs a
|
||||
// reference tree does not need, and `.git` — additional sources never carry
|
||||
// git-history semantics (anchor-only).
|
||||
const additionalSourceExclude = mergeExcludes(SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES, [".git"]);
|
||||
|
||||
// Every delegated post-upload command (extract/wipe/remove-deleted/asset merge)
|
||||
// must run under the run-specific timeout (`spec.timeoutMs`), not the provider
|
||||
// sync client's default timeout — the two can differ, and before staging was
|
||||
@@ -709,74 +764,93 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
...(input.preserveAbsentOnRestore ?? []),
|
||||
]);
|
||||
|
||||
// Build one `SandboxSyncOperation` uploading a host tarball as a single file
|
||||
// mapping (rides native `uploadFiles`, or the base64-tar fallback) with the
|
||||
// extract/wipe/merge steps carried as ordered `postUploadCommands`, confine
|
||||
// it, and delegate to `syncIn`. `finish` emits the terminal progress line.
|
||||
const stageTarball = async (input2: {
|
||||
tarPath: string;
|
||||
remoteTar: string;
|
||||
postUploadCommands: SandboxPostUploadCommand[];
|
||||
// Stage one source directory into an isolated remote subdirectory through the
|
||||
// unified `syncIn` seam. Both the workspace/asset tar path and the additional
|
||||
// (referenced) project path use it: build one `SandboxSyncOperation` from the
|
||||
// caller's `files` mappings (a host tarball as a single `file` mapping, or a
|
||||
// whole `directory` mapping), carry any extract/wipe/merge steps as ordered
|
||||
// `postUploadCommands`, confine the operation's source and target to their
|
||||
// own roots (fail-closed), and delegate to `syncIn` (native transfer, or the
|
||||
// base64-tar fallback). `finish` emits the terminal progress line.
|
||||
const stageConfinedSyncIn = async (params: {
|
||||
files: SandboxSyncFileMapping[];
|
||||
postUploadCommands?: SandboxPostUploadCommand[];
|
||||
sourceRoots: string[];
|
||||
targetRoots: string[];
|
||||
progressLabel: string;
|
||||
statusPhase: RuntimeStatusPhase;
|
||||
progressBytes: number;
|
||||
}): Promise<void> => {
|
||||
const tarSize = (await fs.stat(input2.tarPath)).size;
|
||||
const operations: SandboxSyncOperation[] = [{
|
||||
operationId: nextSyncOperationId(),
|
||||
files: [{ sourcePath: input2.tarPath, targetPath: input2.remoteTar, kind: "file" }],
|
||||
postUploadCommands: withRunTimeout(input2.postUploadCommands),
|
||||
files: params.files,
|
||||
...(params.postUploadCommands
|
||||
? { postUploadCommands: withRunTimeout(params.postUploadCommands) }
|
||||
: {}),
|
||||
}];
|
||||
assertSyncOperationsConfined(operations, {
|
||||
sourceRoots: [tempDir],
|
||||
targetRoots: [runtimeRootDir],
|
||||
sourceRoots: params.sourceRoots,
|
||||
targetRoots: params.targetRoots,
|
||||
});
|
||||
const upload = makeTransferProgress(
|
||||
input.onProgress,
|
||||
"Syncing",
|
||||
"to",
|
||||
input2.progressLabel,
|
||||
{ sink: input.onRuntimeProgress, phase: input2.statusPhase },
|
||||
params.progressLabel,
|
||||
{ sink: input.onRuntimeProgress, phase: params.statusPhase },
|
||||
);
|
||||
await syncIn(operations);
|
||||
await upload.finish(tarSize, tarSize);
|
||||
await upload.finish(params.progressBytes, params.progressBytes);
|
||||
};
|
||||
|
||||
if (syncWorkspace && gitSnapshot) {
|
||||
await emitRuntimeStatus(input.onRuntimeProgress, "git_sync", "Syncing git history to sandbox");
|
||||
await withShallowGitWorkspaceClone({
|
||||
localDir: input.workspaceLocalDir,
|
||||
snapshot: gitSnapshot,
|
||||
}, async (cloneDir) => {
|
||||
// git-workspace preserves `.paperclip-runtime` on the target and the
|
||||
// workspace overlay merges on top rather than replacing — expressed as
|
||||
// the operation's ordered post-upload commands, not a plain replace.
|
||||
const gitTarPath = path.join(tempDir, "git-workspace.tar");
|
||||
await createTarballFromDirectory({
|
||||
localDir: cloneDir,
|
||||
archivePath: gitTarPath,
|
||||
exclude: [".paperclip-runtime"],
|
||||
});
|
||||
const remoteGitTar = path.posix.join(runtimeRootDir, "git-workspace-upload.tar");
|
||||
await stageTarball({
|
||||
tarPath: gitTarPath,
|
||||
remoteTar: remoteGitTar,
|
||||
postUploadCommands: [{
|
||||
command: buildWorkspaceTarExtractCommand({
|
||||
workspaceRemoteDir,
|
||||
remoteTar: remoteGitTar,
|
||||
wipeExceptNames: [".paperclip-runtime"],
|
||||
}),
|
||||
}],
|
||||
progressLabel: "git history",
|
||||
statusPhase: "git_sync",
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
if (syncWorkspace) {
|
||||
// A git-backed workspace and a plain workspace both stage through ONE
|
||||
// confined `syncIn` operation. A git-backed workspace carries TWO host tars —
|
||||
// the git-history clone and the working-tree overlay — as two `file` mappings
|
||||
// on the SAME operation, with their extract commands as ordered
|
||||
// `postUploadCommands`. One operation shares one mkdir, one confine guard, one
|
||||
// `uploadFiles`, and one rename exec, so the second `syncIn` round trip is
|
||||
// removed. Build the whole merged file set and command list BEFORE the confine
|
||||
// guard runs (inside `stageConfinedSyncIn`); never append a mapping after it.
|
||||
const workspaceFiles: SandboxSyncFileMapping[] = [];
|
||||
const workspacePostUploadCommands: SandboxPostUploadCommand[] = [];
|
||||
let workspaceUploadBytes = 0;
|
||||
|
||||
// 1. git-history tar (git-backed workspace only). Both tar targets live under
|
||||
// `runtimeRootDir` (`.paperclip-runtime/<adapterKey>`). The git extract
|
||||
// wipes the target tree EXCEPT `.paperclip-runtime`, so the overlay tar,
|
||||
// which sits under `.paperclip-runtime`, survives to run its own extract.
|
||||
if (gitSnapshot) {
|
||||
await emitRuntimeStatus(input.onRuntimeProgress, "git_sync", "Syncing git history to sandbox");
|
||||
const gitTarPath = path.join(tempDir, "git-workspace.tar");
|
||||
const remoteGitTar = path.posix.join(runtimeRootDir, "git-workspace-upload.tar");
|
||||
await withShallowGitWorkspaceClone({
|
||||
localDir: input.workspaceLocalDir,
|
||||
snapshot: gitSnapshot,
|
||||
}, async (cloneDir) => {
|
||||
await createTarballFromDirectory({
|
||||
localDir: cloneDir,
|
||||
archivePath: gitTarPath,
|
||||
exclude: [".paperclip-runtime"],
|
||||
});
|
||||
});
|
||||
workspaceFiles.push({ sourcePath: gitTarPath, targetPath: remoteGitTar, kind: "file", access: "rw", writablePath: workspaceRemoteDir });
|
||||
workspacePostUploadCommands.push({
|
||||
command: buildWorkspaceTarExtractCommand({
|
||||
workspaceRemoteDir,
|
||||
remoteTar: remoteGitTar,
|
||||
wipeExceptNames: [".paperclip-runtime"],
|
||||
}),
|
||||
});
|
||||
workspaceUploadBytes += (await fs.stat(gitTarPath)).size;
|
||||
}
|
||||
|
||||
// 2. workspace-overlay tar. A git-backed overlay merges on top of the just
|
||||
// extracted git tree (no wipe); a plain workspace wipes every child except
|
||||
// the preserved names first. The extract runs AFTER the git extract.
|
||||
await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing workspace to sandbox");
|
||||
const workspaceTarPath = path.join(tempDir, "workspace.tar");
|
||||
const workspaceArchiveDir = gitSnapshot ? path.join(tempDir, "workspace-overlay") : input.workspaceLocalDir;
|
||||
await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing workspace to sandbox");
|
||||
if (gitSnapshot) {
|
||||
await copySelectedWorkspaceEntries({
|
||||
sourceDir: input.workspaceLocalDir,
|
||||
@@ -791,15 +865,15 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
exclude: gitSnapshot ? undefined : workspaceArchiveExclude,
|
||||
});
|
||||
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-upload.tar");
|
||||
// git overlay merges on top of the just-extracted git tree (no wipe);
|
||||
// non-git workspace wipes every child except the preserved names first.
|
||||
const workspacePostUploadCommands: SandboxPostUploadCommand[] = [{
|
||||
workspaceFiles.push({ sourcePath: workspaceTarPath, targetPath: remoteWorkspaceTar, kind: "file", access: "rw", writablePath: workspaceRemoteDir });
|
||||
workspacePostUploadCommands.push({
|
||||
command: buildWorkspaceTarExtractCommand({
|
||||
workspaceRemoteDir,
|
||||
remoteTar: remoteWorkspaceTar,
|
||||
wipeExceptNames: gitSnapshot ? null : [...preservedNames],
|
||||
}),
|
||||
}];
|
||||
});
|
||||
// 3. Optional remove-deleted-paths command runs LAST, after both extracts.
|
||||
if (gitSnapshot && gitSnapshot.deletedPaths.length > 0) {
|
||||
workspacePostUploadCommands.push({
|
||||
command: buildRemoveDeletedPathsCommand({
|
||||
@@ -808,12 +882,19 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
}),
|
||||
});
|
||||
}
|
||||
await stageTarball({
|
||||
tarPath: workspaceTarPath,
|
||||
remoteTar: remoteWorkspaceTar,
|
||||
workspaceUploadBytes += (await fs.stat(workspaceTarPath)).size;
|
||||
|
||||
// One confined `syncIn` for the whole merged workspace file set. The confine
|
||||
// guard covers every mapping BEFORE any bytes upload (fail-closed): a source
|
||||
// or target escape in EITHER tar mapping stops the upload of both.
|
||||
await stageConfinedSyncIn({
|
||||
files: workspaceFiles,
|
||||
postUploadCommands: workspacePostUploadCommands,
|
||||
sourceRoots: [tempDir],
|
||||
targetRoots: [runtimeRootDir],
|
||||
progressLabel: "workspace",
|
||||
statusPhase: "config_sync",
|
||||
progressBytes: workspaceUploadBytes,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -835,7 +916,7 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
exclude: asset.exclude,
|
||||
});
|
||||
const files: SandboxSyncFileMapping[] = [
|
||||
{ sourcePath: assetTarPath, targetPath: remoteAssetTar, kind: "file" },
|
||||
{ sourcePath: assetTarPath, targetPath: remoteAssetTar, kind: "file", access: "rw", writablePath: remoteAssetDir },
|
||||
];
|
||||
// Stage provision helper files (e.g. the merge scripts) into the temp dir
|
||||
// and map them alongside the asset tar so they ride the same native upload.
|
||||
@@ -849,10 +930,14 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
: stageFile.contents;
|
||||
const stageHostPath = path.join(tempDir, `${asset.key}.stage.${safeName}`);
|
||||
await fs.writeFile(stageHostPath, stageBytes);
|
||||
// A stage helper file (for example a merge script) is a read-only input
|
||||
// that the provision command reads; the agent does not change it and does
|
||||
// not keep it. So it is `access: "ro"` and never joins the writable set.
|
||||
files.push({
|
||||
sourcePath: stageHostPath,
|
||||
targetPath: path.posix.join(runtimeRootDir, safeName),
|
||||
kind: "file",
|
||||
access: "ro",
|
||||
});
|
||||
}
|
||||
const postUploadCommand = asset.provision?.postUploadCommand?.({
|
||||
@@ -860,25 +945,64 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
assetDir: remoteAssetDir,
|
||||
runtimeRootDir,
|
||||
}) ?? buildDefaultExtractRuntimeAssetCommand({ remoteAssetDir, remoteAssetTar });
|
||||
const operations: SandboxSyncOperation[] = [{
|
||||
operationId: nextSyncOperationId(),
|
||||
const assetTarSize = (await fs.stat(assetTarPath)).size;
|
||||
await stageConfinedSyncIn({
|
||||
files,
|
||||
postUploadCommands: withRunTimeout([{ command: postUploadCommand }]),
|
||||
}];
|
||||
assertSyncOperationsConfined(operations, {
|
||||
postUploadCommands: [{ command: postUploadCommand }],
|
||||
sourceRoots: [tempDir],
|
||||
targetRoots: [runtimeRootDir],
|
||||
progressLabel: asset.key,
|
||||
statusPhase: "config_sync",
|
||||
progressBytes: assetTarSize,
|
||||
});
|
||||
const assetTarSize = (await fs.stat(assetTarPath)).size;
|
||||
const assetUpload = makeTransferProgress(
|
||||
input.onProgress,
|
||||
"Syncing",
|
||||
"to",
|
||||
asset.key,
|
||||
{ sink: input.onRuntimeProgress, phase: "config_sync" },
|
||||
);
|
||||
await syncIn(operations);
|
||||
await assetUpload.finish(assetTarSize, assetTarSize);
|
||||
}
|
||||
|
||||
// Stage each referenced (additional) project as a plain, read-only tree in
|
||||
// its OWN isolated remote directory (`project-<projectId>`). An additional
|
||||
// project rides one confined `syncIn` directory mapping — a native directory
|
||||
// transfer, or the base64-tar fallback — with source and target confined to
|
||||
// their own roots. No workspace, git-history, or `.paperclip-runtime`
|
||||
// semantics apply; those stay anchor-only. Per-project failure isolation: one
|
||||
// project's confinement or sync failure logs a warning and is skipped, and
|
||||
// the run plus the other projects continue. Only a project that stages
|
||||
// successfully appears in `additionalSourceDirs`.
|
||||
for (const source of input.additionalSources ?? []) {
|
||||
const { localPath, projectId } = source;
|
||||
const label = `project-${projectId}`;
|
||||
try {
|
||||
if (!path.posix.isAbsolute(localPath)) {
|
||||
throw new Error(`additional source localPath is not an absolute path: ${localPath}`);
|
||||
}
|
||||
if (
|
||||
projectId.length === 0 ||
|
||||
projectId.includes("/") ||
|
||||
projectId.includes("\\") ||
|
||||
projectId.includes("..")
|
||||
) {
|
||||
throw new Error(`additional source projectId is not a simple path segment: ${projectId}`);
|
||||
}
|
||||
const remoteProjectDir = path.posix.join(runtimeRootDir, label);
|
||||
await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing referenced project to sandbox");
|
||||
await stageConfinedSyncIn({
|
||||
files: [{
|
||||
sourcePath: localPath,
|
||||
targetPath: remoteProjectDir,
|
||||
kind: "directory",
|
||||
exclude: additionalSourceExclude,
|
||||
access: "ro",
|
||||
}],
|
||||
sourceRoots: [localPath],
|
||||
targetRoots: [remoteProjectDir],
|
||||
progressLabel: label,
|
||||
statusPhase: "config_sync",
|
||||
progressBytes: 0,
|
||||
});
|
||||
additionalSourceDirs[projectId] = remoteProjectDir;
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`[paperclip] Failed to stage referenced project ${projectId}; skipping it. ${String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
@@ -892,6 +1016,7 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
workspaceRemoteDir,
|
||||
runtimeRootDir,
|
||||
assetDirs,
|
||||
additionalSourceDirs,
|
||||
restoreWorkspace: async (onProgress?: RuntimeProgressSink) => {
|
||||
const restoreSink = onProgress ?? input.onProgress;
|
||||
if (!syncWorkspace) {
|
||||
|
||||
@@ -170,6 +170,49 @@ describe("ssh env-lab fixture", () => {
|
||||
await stopSshEnvLabFixture(statePath);
|
||||
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
||||
|
||||
it("builds a remote script that sources login profiles but no nvm", async () => {
|
||||
const target = await buildSshSpawnTarget({
|
||||
spec: {
|
||||
host: "ssh.example.test",
|
||||
port: 22,
|
||||
username: "ssh-user",
|
||||
remoteCwd: "/srv/paperclip/workspace",
|
||||
remoteWorkspacePath: "/srv/paperclip/workspace",
|
||||
privateKey: null,
|
||||
knownHosts: null,
|
||||
strictHostKeyChecking: true,
|
||||
},
|
||||
command: "node",
|
||||
args: ["--version"],
|
||||
env: { FOO: "bar" },
|
||||
});
|
||||
|
||||
// The remote script rides the last ssh argument. The SSH target is an
|
||||
// operator-configured host that can expose `node` only through a login
|
||||
// profile, so the wrapper sources the profiles. It no longer sources
|
||||
// `nvm.sh`; a profile that adds nvm still runs.
|
||||
const remoteScript = String(target.args.at(-1) ?? "");
|
||||
expect(remoteScript).not.toContain("nvm.sh");
|
||||
expect(remoteScript).not.toContain("NVM_DIR");
|
||||
// Source /etc/profile so a host that exposes the PATH through
|
||||
// /etc/profile.d scripts still resolves node and the agent CLI.
|
||||
expect(remoteScript).toContain("/etc/profile");
|
||||
expect(remoteScript).toContain(".profile");
|
||||
expect(remoteScript).toContain(".bash_profile");
|
||||
expect(remoteScript).toContain(".zprofile");
|
||||
// Fall back to .bashrc when no .bash_profile exists, so a host that adds
|
||||
// nvm in .bashrc still resolves node under a non-login SSH command.
|
||||
expect(remoteScript).toContain(".bashrc");
|
||||
// The last ssh argument wraps the script as `sh -c '...'`, so the inner
|
||||
// quotes are escaped. Assert the command still runs: cd, env, and the argv.
|
||||
expect(remoteScript).toContain("cd ");
|
||||
expect(remoteScript).toContain("/srv/paperclip/workspace");
|
||||
expect(remoteScript).toContain("exec env ");
|
||||
expect(remoteScript).toContain("node");
|
||||
expect(remoteScript).toContain("--version");
|
||||
await target.cleanup();
|
||||
});
|
||||
|
||||
it("rejects invalid environment variable keys when constructing SSH spawn targets", async () => {
|
||||
await expect(
|
||||
buildSshSpawnTarget({
|
||||
|
||||
@@ -1166,14 +1166,22 @@ export async function runSshCommand(
|
||||
}
|
||||
}
|
||||
|
||||
// Mirror buildSshSpawnTarget: source login profiles first, then run
|
||||
// `env KEY=VAL cmd` so user-supplied identity overrides win over anything
|
||||
// a profile re-exports. Without this, a remote profile that resets HOME
|
||||
// / NVM_DIR / etc. would silently undo the explicit env passed in here.
|
||||
// Mirror buildSshSpawnTarget: source the login profiles first, then run
|
||||
// `env KEY=VAL cmd` so user-supplied identity overrides win over anything a
|
||||
// profile re-exports. The SSH target is an operator-configured host, not a
|
||||
// Paperclip sandbox image, so it can expose `node` or an agent CLI only
|
||||
// through a login profile; a non-login SSH command would miss that PATH.
|
||||
// Source `/etc/profile` first so a host that exposes the PATH through
|
||||
// `/etc/profile.d` scripts still resolves node and the agent CLI.
|
||||
// The script no longer sources `nvm.sh`; a profile that adds nvm still runs.
|
||||
// .bash_profile typically sources .bashrc itself; only source .bashrc
|
||||
// directly when no .bash_profile exists, so a host that adds nvm in
|
||||
// .bashrc still resolves node without a double-run of the setup.
|
||||
const envArgs = envEntries.map(([key, value]) => `${key}=${shellQuote(value)}`);
|
||||
const remoteScript = [
|
||||
'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
|
||||
envArgs.length > 0
|
||||
? `exec env ${envArgs.join(" ")} sh -c ${shellQuote(remoteCommand)}`
|
||||
@@ -1223,12 +1231,22 @@ export async function buildSshSpawnTarget(input: {
|
||||
.filter((entry): entry is [string, string] => typeof entry[1] === "string")
|
||||
.map(([key, value]) => `${key}=${shellQuote(value)}`);
|
||||
const remoteCommandParts = [shellQuote(input.command), ...input.args.map((arg) => shellQuote(arg))].join(" ");
|
||||
// Source the login profiles first, then run `env KEY=VAL cmd` so
|
||||
// user-supplied identity overrides win over anything a profile re-exports.
|
||||
// The SSH target is an operator-configured host, not a Paperclip sandbox
|
||||
// image, so it can expose `node` or an agent CLI only through a login
|
||||
// profile; a non-login SSH command would miss that PATH. Source
|
||||
// `/etc/profile` first so a host that exposes the PATH through
|
||||
// `/etc/profile.d` scripts still resolves node and the agent CLI. The script
|
||||
// no longer sources `nvm.sh`; a profile that adds nvm still runs.
|
||||
// .bash_profile typically sources .bashrc itself; only source .bashrc
|
||||
// directly when no .bash_profile exists, so a host that adds nvm in
|
||||
// .bashrc still resolves node without a double-run of the setup.
|
||||
const remoteScript = [
|
||||
'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
|
||||
'export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"',
|
||||
'[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" >/dev/null 2>&1 || true',
|
||||
`cd ${shellQuote(input.spec.remoteCwd)}`,
|
||||
envArgs.length > 0
|
||||
? `exec env ${envArgs.join(" ")} ${remoteCommandParts}`
|
||||
|
||||
@@ -175,6 +175,14 @@ export interface AdapterExecutionContext {
|
||||
onRuntimeProgress?: RuntimeStatusSink;
|
||||
onSpawn?: (meta: { pid: number; processGroupId: number | null; startedAt: string }) => Promise<void>;
|
||||
authToken?: string;
|
||||
/**
|
||||
* The injected OpenTelemetry startup trace context (tracer + root
|
||||
* parent-context helper). The server passes the real, endpoint-gated
|
||||
* implementation; when absent, the ACPX engine uses a no-op, so the whole
|
||||
* span path stays inert. The type is an inline import so this module keeps no
|
||||
* top-level dependency on the timing helper.
|
||||
*/
|
||||
startupTraceContext?: import("./acpx-engine/startup-timing.js").StartupTraceContext;
|
||||
}
|
||||
|
||||
export interface AdapterModel {
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
-- Drop the cloud sync (cloud upstream) tables. The host-to-host transport
|
||||
-- has been removed in favor of the Import/Export flow, and nothing reads or
|
||||
-- writes these tables any more. The sender tables (connections, runs) came
|
||||
-- from 0089; the receiver-side tables never shipped, so these two are all
|
||||
-- that exist. The feature was experimental and flag-gated off by default,
|
||||
-- so its run history is intentionally discarded.
|
||||
DROP TABLE IF EXISTS "cloud_upstream_runs";
|
||||
--> statement-breakpoint
|
||||
DROP TABLE IF EXISTS "cloud_upstream_connections";
|
||||
File renamed without changes.
@@ -1363,8 +1363,15 @@
|
||||
{
|
||||
"idx": 196,
|
||||
"version": "7",
|
||||
"when": 1785170000002,
|
||||
"tag": "0196_issue_watchdog_mode",
|
||||
"when": 1785170001001,
|
||||
"tag": "0196_drop_cloud_upstream_tables",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 197,
|
||||
"version": "7",
|
||||
"when": 1785170001002,
|
||||
"tag": "0197_issue_watchdog_mode",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
|
||||
@@ -1,75 +0,0 @@
|
||||
import { boolean, index, integer, jsonb, pgTable, text, timestamp, uuid } from "drizzle-orm/pg-core";
|
||||
import { companies } from "./companies.js";
|
||||
|
||||
export const cloudUpstreamConnections = pgTable(
|
||||
"cloud_upstream_connections",
|
||||
{
|
||||
id: uuid("id").primaryKey().defaultRandom(),
|
||||
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
|
||||
remoteUrl: text("remote_url").notNull(),
|
||||
sourceInstanceId: text("source_instance_id").notNull(),
|
||||
sourceInstanceFingerprint: text("source_instance_fingerprint").notNull(),
|
||||
sourcePublicKey: text("source_public_key").notNull(),
|
||||
// Stored through the Cloud Upstream service as an encrypted credential envelope.
|
||||
privateKeyPem: text("private_key_pem").notNull(),
|
||||
tokenStatus: text("token_status").notNull(),
|
||||
scopes: text("scopes").array().notNull().default([]),
|
||||
authorizedGlobalUserId: text("authorized_global_user_id"),
|
||||
// Stored through the Cloud Upstream service as an encrypted credential envelope.
|
||||
accessToken: text("access_token"),
|
||||
tokenId: text("token_id"),
|
||||
tokenExpiresAt: timestamp("token_expires_at", { withTimezone: true }),
|
||||
|
||||
targetStackId: text("target_stack_id").notNull(),
|
||||
targetStackSlug: text("target_stack_slug"),
|
||||
targetStackDisplayName: text("target_stack_display_name"),
|
||||
targetCompanyId: text("target_company_id").notNull(),
|
||||
targetOrigin: text("target_origin").notNull(),
|
||||
targetPrimaryHost: text("target_primary_host").notNull(),
|
||||
targetProduct: text("target_product").notNull(),
|
||||
targetSchemaMajor: integer("target_schema_major").notNull(),
|
||||
targetMaxChunkBytes: integer("target_max_chunk_bytes").notNull(),
|
||||
|
||||
pendingState: text("pending_state"),
|
||||
pendingCodeVerifier: text("pending_code_verifier"),
|
||||
pendingRedirectUri: text("pending_redirect_uri"),
|
||||
pendingTokenUrl: text("pending_token_url"),
|
||||
|
||||
lastRunId: uuid("last_run_id"),
|
||||
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
},
|
||||
(table) => [
|
||||
index("cloud_upstream_connections_company_idx").on(table.companyId),
|
||||
],
|
||||
);
|
||||
|
||||
export const cloudUpstreamRuns = pgTable(
|
||||
"cloud_upstream_runs",
|
||||
{
|
||||
id: uuid("id").primaryKey().defaultRandom(),
|
||||
connectionId: uuid("connection_id").notNull().references(() => cloudUpstreamConnections.id, { onDelete: "cascade" }),
|
||||
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
|
||||
remoteRunId: text("remote_run_id"),
|
||||
status: text("status").notNull(),
|
||||
activeStep: text("active_step").notNull(),
|
||||
progressPercent: integer("progress_percent").notNull().default(0),
|
||||
dryRun: boolean("dry_run").notNull().default(false),
|
||||
retryOfRunId: uuid("retry_of_run_id"),
|
||||
summary: jsonb("summary").$type<import("@paperclipai/shared").CloudUpstreamSummaryCount[]>().notNull().default([]),
|
||||
warnings: jsonb("warnings").$type<import("@paperclipai/shared").CloudUpstreamWarning[]>().notNull().default([]),
|
||||
conflicts: jsonb("conflicts").$type<import("@paperclipai/shared").CloudUpstreamConflict[]>().notNull().default([]),
|
||||
events: jsonb("events").$type<import("@paperclipai/shared").CloudUpstreamRunEvent[]>().notNull().default([]),
|
||||
report: jsonb("report").$type<Record<string, unknown>>().notNull().default({}),
|
||||
idempotencyKey: text("idempotency_key").notNull(),
|
||||
manifestHash: text("manifest_hash").notNull(),
|
||||
targetUrl: text("target_url"),
|
||||
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
completedAt: timestamp("completed_at", { withTimezone: true }),
|
||||
},
|
||||
(table) => [
|
||||
index("cloud_upstream_runs_company_created_idx").on(table.companyId, table.createdAt),
|
||||
index("cloud_upstream_runs_connection_idx").on(table.connectionId),
|
||||
],
|
||||
);
|
||||
@@ -2,7 +2,6 @@ export { companies } from "./companies.js";
|
||||
export { companyLogos } from "./company_logos.js";
|
||||
export { authUsers, authSessions, authAccounts, authVerifications } from "./auth.js";
|
||||
export { instanceSettings } from "./instance_settings.js";
|
||||
export { cloudUpstreamConnections, cloudUpstreamRuns } from "./cloud_upstreams.js";
|
||||
export { instanceUserRoles } from "./instance_user_roles.js";
|
||||
export { userSidebarPreferences } from "./user_sidebar_preferences.js";
|
||||
export { agents } from "./agents.js";
|
||||
|
||||
@@ -92,8 +92,6 @@ export const LOCAL_TABLE_ROW_COUNTS = [
|
||||
{ table: "budget_incidents", localRows: 0 },
|
||||
{ table: "budget_policies", localRows: 0 },
|
||||
{ table: "cli_auth_challenges", localRows: 0 },
|
||||
{ table: "cloud_upstream_connections", localRows: 0 },
|
||||
{ table: "cloud_upstream_runs", localRows: 0 },
|
||||
{ table: "company_logos", localRows: 0 },
|
||||
{ table: "company_secret_provider_configs", localRows: 0 },
|
||||
{ table: "company_skill_comments", localRows: 0 },
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
import fs from "node:fs";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
__embeddedPostgresStartMaxAttemptsForTests as MAX_ATTEMPTS,
|
||||
__setEmbeddedPostgresCtorProviderForTests,
|
||||
__startEmbeddedPostgresWithRetryForTests as startWithRetry,
|
||||
} from "./test-embedded-postgres.js";
|
||||
|
||||
// A fake embedded-postgres constructor. It records every constructed instance so
|
||||
// the test can assert the retry uses a fresh port and a fresh data directory each
|
||||
// attempt. `start()` emits the same output the real cluster writes for a port
|
||||
// conflict, then rejects with an empty message (the real rejection shape). The
|
||||
// option type matches the real constructor so no type cast is needed.
|
||||
type FakeOptions = {
|
||||
databaseDir: string;
|
||||
user: string;
|
||||
password: string;
|
||||
port: number;
|
||||
persistent: boolean;
|
||||
initdbFlags?: string[];
|
||||
onLog?: (message: unknown) => void;
|
||||
onError?: (message: unknown) => void;
|
||||
};
|
||||
|
||||
const BIND_CONFLICT_LOG = 'could not bind IPv4 address "127.0.0.1": Address already in use';
|
||||
|
||||
function makeFakeCtor(failFirst: number) {
|
||||
const constructed: FakeOptions[] = [];
|
||||
let started = 0;
|
||||
|
||||
class FakeEmbeddedPostgres {
|
||||
private readonly options: FakeOptions;
|
||||
constructor(options: FakeOptions) {
|
||||
this.options = options;
|
||||
constructed.push(options);
|
||||
}
|
||||
async initialise(): Promise<void> {}
|
||||
async start(): Promise<void> {
|
||||
started += 1;
|
||||
if (started <= failFirst) {
|
||||
// Mirror the real failure: Postgres logs the reason, then `start()`
|
||||
// rejects with an Error whose message is empty.
|
||||
this.options.onLog?.(BIND_CONFLICT_LOG);
|
||||
throw new Error();
|
||||
}
|
||||
}
|
||||
async stop(): Promise<void> {}
|
||||
}
|
||||
|
||||
return { ctor: FakeEmbeddedPostgres, constructed };
|
||||
}
|
||||
|
||||
describe("startEmbeddedPostgresWithRetry", () => {
|
||||
afterEach(() => {
|
||||
__setEmbeddedPostgresCtorProviderForTests(null);
|
||||
});
|
||||
|
||||
it("recovers from a transient port conflict and returns on a later attempt", async () => {
|
||||
const { ctor, constructed } = makeFakeCtor(2);
|
||||
__setEmbeddedPostgresCtorProviderForTests(async () => ctor);
|
||||
|
||||
const started = await startWithRetry("paperclip-retry-recover-");
|
||||
|
||||
// The first two attempts fail, the third succeeds.
|
||||
expect(constructed).toHaveLength(3);
|
||||
|
||||
// Each attempt uses a fresh data directory. The two failed directories are
|
||||
// removed. The returned directory still exists.
|
||||
const dataDirs = constructed.map((options) => options.databaseDir);
|
||||
expect(new Set(dataDirs).size).toBe(3);
|
||||
expect(fs.existsSync(dataDirs[0])).toBe(false);
|
||||
expect(fs.existsSync(dataDirs[1])).toBe(false);
|
||||
expect(started.dataDir).toBe(dataDirs[2]);
|
||||
expect(fs.existsSync(started.dataDir)).toBe(true);
|
||||
|
||||
// Each attempt allocates a port.
|
||||
for (const options of constructed) {
|
||||
expect(Number.isInteger(options.port)).toBe(true);
|
||||
expect(options.port).toBeGreaterThan(0);
|
||||
}
|
||||
|
||||
// Clean up the returned attempt.
|
||||
await started.instance.stop();
|
||||
fs.rmSync(started.dataDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("throws with the real Postgres output after the attempt bound", async () => {
|
||||
const { ctor, constructed } = makeFakeCtor(Number.POSITIVE_INFINITY);
|
||||
__setEmbeddedPostgresCtorProviderForTests(async () => ctor);
|
||||
|
||||
await expect(startWithRetry("paperclip-retry-fail-")).rejects.toThrow(/after \d+ attempts/);
|
||||
|
||||
// The retry stops at the bound and does not loop forever.
|
||||
expect(constructed).toHaveLength(MAX_ATTEMPTS);
|
||||
|
||||
// Every failed attempt removes its data directory.
|
||||
for (const options of constructed) {
|
||||
expect(fs.existsSync(options.databaseDir)).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps the real failure reason instead of a generic fallback", async () => {
|
||||
const { ctor } = makeFakeCtor(Number.POSITIVE_INFINITY);
|
||||
__setEmbeddedPostgresCtorProviderForTests(async () => ctor);
|
||||
|
||||
const error = await startWithRetry("paperclip-retry-reason-").catch((caught: unknown) => caught);
|
||||
|
||||
expect(error).toBeInstanceOf(Error);
|
||||
// The thrown message carries the captured Postgres output, not only the
|
||||
// generic "embedded Postgres startup failed" text.
|
||||
expect((error as Error).message).toContain("Address already in use");
|
||||
});
|
||||
});
|
||||
@@ -3,6 +3,10 @@ import net from "node:net";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { applyPendingMigrations, ensurePostgresDatabase } from "./client.js";
|
||||
import {
|
||||
createEmbeddedPostgresLogBuffer,
|
||||
formatEmbeddedPostgresError,
|
||||
} from "./embedded-postgres-error.js";
|
||||
import { prepareEmbeddedPostgresNativeRuntime } from "./embedded-postgres-native.js";
|
||||
|
||||
type EmbeddedPostgresInstance = {
|
||||
@@ -47,12 +51,29 @@ function getReservedTestPorts(): Set<number> {
|
||||
return new Set(configuredPorts.filter((port) => Number.isInteger(port) && port > 0 && port <= 65535));
|
||||
}
|
||||
|
||||
async function getEmbeddedPostgresCtor(): Promise<EmbeddedPostgresCtor> {
|
||||
type EmbeddedPostgresCtorProvider = () => Promise<EmbeddedPostgresCtor>;
|
||||
|
||||
async function loadEmbeddedPostgresCtor(): Promise<EmbeddedPostgresCtor> {
|
||||
const mod = await import("embedded-postgres");
|
||||
await prepareEmbeddedPostgresNativeRuntime();
|
||||
return mod.default as EmbeddedPostgresCtor;
|
||||
}
|
||||
|
||||
let embeddedPostgresCtorProvider: EmbeddedPostgresCtorProvider = loadEmbeddedPostgresCtor;
|
||||
|
||||
// Test seam. Replace the embedded-postgres constructor provider so a test can
|
||||
// simulate a failed start without the native runtime. Pass `null` to restore
|
||||
// the default provider. This module is test support only, so the seam is safe.
|
||||
export function __setEmbeddedPostgresCtorProviderForTests(
|
||||
provider: EmbeddedPostgresCtorProvider | null,
|
||||
): void {
|
||||
embeddedPostgresCtorProvider = provider ?? loadEmbeddedPostgresCtor;
|
||||
}
|
||||
|
||||
async function getEmbeddedPostgresCtor(): Promise<EmbeddedPostgresCtor> {
|
||||
return await embeddedPostgresCtorProvider();
|
||||
}
|
||||
|
||||
async function getAvailablePort(): Promise<number> {
|
||||
const reservedPorts = getReservedTestPorts();
|
||||
for (let attempt = 0; attempt < 20; attempt += 1) {
|
||||
@@ -88,6 +109,11 @@ async function createEmbeddedPostgresTestInstance(tempDirPrefix: string) {
|
||||
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), tempDirPrefix));
|
||||
const port = await getAvailablePort();
|
||||
const EmbeddedPostgres = await getEmbeddedPostgresCtor();
|
||||
// Postgres writes the true reason for a failed start to its output, for
|
||||
// example `could not bind IPv4 address "127.0.0.1": Address already in use`.
|
||||
// The `start()` rejection carries an empty message, so we capture the output
|
||||
// in a bounded buffer and surface it in the thrown error.
|
||||
const logBuffer = createEmbeddedPostgresLogBuffer();
|
||||
const instance = new EmbeddedPostgres({
|
||||
databaseDir: dataDir,
|
||||
user: "paperclip",
|
||||
@@ -95,11 +121,11 @@ async function createEmbeddedPostgresTestInstance(tempDirPrefix: string) {
|
||||
port,
|
||||
persistent: true,
|
||||
initdbFlags: ["--encoding=UTF8", "--locale=C", "--lc-messages=C"],
|
||||
onLog: () => {},
|
||||
onError: () => {},
|
||||
onLog: (message) => logBuffer.append(message),
|
||||
onError: (message) => logBuffer.append(message),
|
||||
});
|
||||
|
||||
return { dataDir, port, instance };
|
||||
return { dataDir, port, instance, getRecentLogs: () => logBuffer.getRecentLogs() };
|
||||
}
|
||||
|
||||
function cleanupEmbeddedPostgresTestDirs(dataDir: string) {
|
||||
@@ -161,34 +187,72 @@ async function stopEmbeddedPostgresBounded(
|
||||
}
|
||||
}
|
||||
|
||||
function formatEmbeddedPostgresError(error: unknown): string {
|
||||
if (error instanceof Error && error.message.length > 0) return error.message;
|
||||
if (typeof error === "string" && error.length > 0) return error;
|
||||
return "embedded Postgres startup failed";
|
||||
// Upper bound on start attempts. `getAvailablePort` uses a check-then-use probe:
|
||||
// it binds port 0, reads the assigned port, closes the probe, then Postgres binds
|
||||
// that port. Under load another process can take the port in that window, so the
|
||||
// bind fails with "Address already in use" and `start()` rejects. Each retry uses
|
||||
// a fresh port and a fresh data directory, so a transient collision clears.
|
||||
const EMBEDDED_POSTGRES_START_MAX_ATTEMPTS = 5;
|
||||
|
||||
// Start one embedded Postgres cluster with a bounded retry. Each attempt gets a
|
||||
// fresh port and a fresh data directory. On a failed attempt we stop the cluster
|
||||
// and remove its data directory before the next attempt. After the last attempt
|
||||
// we throw with the real Postgres output so the failure is loud and diagnosable.
|
||||
async function startEmbeddedPostgresWithRetry(tempDirPrefix: string): Promise<{
|
||||
port: number;
|
||||
dataDir: string;
|
||||
instance: EmbeddedPostgresInstance;
|
||||
}> {
|
||||
let lastError = new Error("embedded Postgres startup failed");
|
||||
|
||||
for (let attempt = 1; attempt <= EMBEDDED_POSTGRES_START_MAX_ATTEMPTS; attempt += 1) {
|
||||
const created = await createEmbeddedPostgresTestInstance(tempDirPrefix);
|
||||
try {
|
||||
await created.instance.initialise();
|
||||
await created.instance.start();
|
||||
return { port: created.port, dataDir: created.dataDir, instance: created.instance };
|
||||
} catch (error) {
|
||||
lastError = formatEmbeddedPostgresError(error, {
|
||||
fallbackMessage: "embedded Postgres startup failed",
|
||||
recentLogs: created.getRecentLogs(),
|
||||
});
|
||||
// Stop the failed cluster and remove its data directory. The next attempt
|
||||
// allocates a fresh port and a fresh data directory.
|
||||
await stopEmbeddedPostgresBounded(created.instance, () =>
|
||||
cleanupEmbeddedPostgresTestDirs(created.dataDir),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error(
|
||||
`Failed to start embedded PostgreSQL test database after ${EMBEDDED_POSTGRES_START_MAX_ATTEMPTS} attempts: ${lastError.message}`,
|
||||
);
|
||||
}
|
||||
|
||||
// Test-only accessors. Production callers use `startEmbeddedPostgresTestDatabase`
|
||||
// or `getEmbeddedPostgresTestSupport`. A test drives the bounded retry directly
|
||||
// so it does not need a real Postgres connection.
|
||||
export const __startEmbeddedPostgresWithRetryForTests = startEmbeddedPostgresWithRetry;
|
||||
export const __embeddedPostgresStartMaxAttemptsForTests = EMBEDDED_POSTGRES_START_MAX_ATTEMPTS;
|
||||
|
||||
async function probeEmbeddedPostgresSupport(): Promise<EmbeddedPostgresTestSupport> {
|
||||
let dataDir: string | null = null;
|
||||
let instance: EmbeddedPostgresInstance | null = null;
|
||||
let started: { dataDir: string; instance: EmbeddedPostgresInstance } | null = null;
|
||||
|
||||
try {
|
||||
const created = await createEmbeddedPostgresTestInstance(
|
||||
"paperclip-embedded-postgres-probe-",
|
||||
);
|
||||
dataDir = created.dataDir;
|
||||
instance = created.instance;
|
||||
await instance.initialise();
|
||||
await instance.start();
|
||||
started = await startEmbeddedPostgresWithRetry("paperclip-embedded-postgres-probe-");
|
||||
return { supported: true };
|
||||
} catch (error) {
|
||||
return {
|
||||
supported: false,
|
||||
reason: formatEmbeddedPostgresError(error),
|
||||
reason: formatEmbeddedPostgresError(error, {
|
||||
fallbackMessage: "embedded Postgres startup failed",
|
||||
}).message,
|
||||
};
|
||||
} finally {
|
||||
await stopEmbeddedPostgresBounded(instance, () => {
|
||||
if (dataDir) cleanupEmbeddedPostgresTestDirs(dataDir);
|
||||
});
|
||||
if (started) {
|
||||
const { dataDir, instance } = started;
|
||||
await stopEmbeddedPostgresBounded(instance, () => cleanupEmbeddedPostgresTestDirs(dataDir));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -202,17 +266,11 @@ export async function getEmbeddedPostgresTestSupport(): Promise<EmbeddedPostgres
|
||||
export async function startEmbeddedPostgresTestDatabase(
|
||||
tempDirPrefix: string,
|
||||
): Promise<EmbeddedPostgresTestDatabase> {
|
||||
let dataDir: string | null = null;
|
||||
let instance: EmbeddedPostgresInstance | null = null;
|
||||
// The bounded retry hardens the cluster start against the port race. It throws
|
||||
// with the real Postgres output if every attempt fails.
|
||||
const { port, dataDir, instance } = await startEmbeddedPostgresWithRetry(tempDirPrefix);
|
||||
|
||||
try {
|
||||
const created = await createEmbeddedPostgresTestInstance(tempDirPrefix);
|
||||
dataDir = created.dataDir;
|
||||
instance = created.instance;
|
||||
const { port } = created;
|
||||
await instance.initialise();
|
||||
await instance.start();
|
||||
|
||||
const adminConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${port}/postgres`;
|
||||
await ensurePostgresDatabase(adminConnectionString, "paperclip");
|
||||
const connectionString = `postgres://paperclip:paperclip@127.0.0.1:${port}/paperclip`;
|
||||
@@ -221,17 +279,17 @@ export async function startEmbeddedPostgresTestDatabase(
|
||||
return {
|
||||
connectionString,
|
||||
cleanup: async () => {
|
||||
await stopEmbeddedPostgresBounded(instance, () => {
|
||||
if (dataDir) cleanupEmbeddedPostgresTestDirs(dataDir);
|
||||
});
|
||||
await stopEmbeddedPostgresBounded(instance, () => cleanupEmbeddedPostgresTestDirs(dataDir));
|
||||
},
|
||||
};
|
||||
} catch (error) {
|
||||
await stopEmbeddedPostgresBounded(instance, () => {
|
||||
if (dataDir) cleanupEmbeddedPostgresTestDirs(dataDir);
|
||||
});
|
||||
await stopEmbeddedPostgresBounded(instance, () => cleanupEmbeddedPostgresTestDirs(dataDir));
|
||||
throw new Error(
|
||||
`Failed to start embedded PostgreSQL test database: ${formatEmbeddedPostgresError(error)}`,
|
||||
`Failed to start embedded PostgreSQL test database: ${
|
||||
formatEmbeddedPostgresError(error, {
|
||||
fallbackMessage: "embedded Postgres startup failed",
|
||||
}).message
|
||||
}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
# Sandbox Runtime Requirements
|
||||
|
||||
This document states the sandbox environment as a contract. The sandbox owner
|
||||
must meet this contract. The Paperclip runtime does not build the environment at
|
||||
exec time. The environment is a requirement, not a build step.
|
||||
|
||||
This document states requirements. It does not state build steps.
|
||||
|
||||
## Required on PATH
|
||||
|
||||
- `node` must be installed and on the PATH.
|
||||
- Each agent CLI that the run uses must be installed and on the PATH. The set of
|
||||
agent CLIs includes `claude`, `codex`, `gemini`, and similar CLIs.
|
||||
- The owner installs only the CLIs that the run uses. The owner does not need to
|
||||
install a CLI that no run uses.
|
||||
|
||||
## Runtime dependencies
|
||||
|
||||
The sandbox execution and synchronization paths need more than `node` and the
|
||||
agent CLIs. The owner must also supply these:
|
||||
|
||||
- A POSIX shell as `sh`, normally `/bin/sh`. The runtime runs each command with
|
||||
`sh -c <script>`. The runtime uses `bash` only when the adapter sets the shell
|
||||
to `bash`.
|
||||
- `tar`. The synchronization path extracts and creates archives with `tar`. A
|
||||
sandbox without `tar` cannot receive or return workspace files.
|
||||
- A writable workspace directory. The runtime extracts the workspace archive
|
||||
into this directory.
|
||||
- A writable home directory. The agent CLIs write state and credentials under
|
||||
the home directory.
|
||||
- A writable cache directory and a writable temporary directory. The runtime and
|
||||
the agent CLIs write intermediate files to these locations.
|
||||
|
||||
## Detection contract
|
||||
|
||||
Paperclip probes each CLI before launch. Paperclip uses the same detection
|
||||
pattern that the runtime Dockerfiles use:
|
||||
|
||||
```bash
|
||||
command -v <cmd> || exit 1
|
||||
```
|
||||
|
||||
Paperclip probes each CLI with `command -v <cmd>`. Paperclip fails loudly when
|
||||
the CLI is absent and no install command is configured for the CLI.
|
||||
|
||||
## Optional CLI installation
|
||||
|
||||
An adapter can configure an install command for a CLI. When an install command
|
||||
is configured, the runtime obeys this flow:
|
||||
|
||||
1. The runtime probes the CLI with `command -v <cmd>`.
|
||||
2. If the CLI is already on the PATH, the runtime skips the install.
|
||||
3. If the CLI is absent, the runtime runs the configured install command one
|
||||
time.
|
||||
4. A failed install is not fatal. The runtime writes a log line and continues.
|
||||
The launch-time probe still reports a missing CLI and fails loudly.
|
||||
|
||||
An owner who relies on a configured install command must also supply the network
|
||||
access, the filesystem write access, and the package tooling that the install
|
||||
command needs. When no install command is configured, the runtime does not
|
||||
install the CLI. The owner must supply the CLI on the PATH.
|
||||
|
||||
## Firm rule
|
||||
|
||||
- The Paperclip runtime never modifies the login profile. The runtime never
|
||||
writes a profile file. The runtime never writes an rc file.
|
||||
- The Paperclip runtime never sources `nvm` on the exec path.
|
||||
- The sandbox owner supplies a ready PATH. The PATH must resolve `node` and each
|
||||
used agent CLI without any action from the runtime, except for a configured
|
||||
install command.
|
||||
@@ -38,6 +38,9 @@ describe("bridge exec", () => {
|
||||
expect(optionsArg).not.toHaveProperty("args");
|
||||
expect(optionsArg).not.toHaveProperty("stdin");
|
||||
expect(commandArg).toContain('. /etc/profile');
|
||||
// The wrapper sources no `nvm.sh`; the sandbox image supplies node on PATH.
|
||||
expect(commandArg).not.toContain("nvm.sh");
|
||||
expect(commandArg).not.toContain("NVM_DIR");
|
||||
expect(commandArg).toContain("cd ");
|
||||
expect(commandArg).toContain("/workspace/paperclip");
|
||||
expect(commandArg).toContain("PAPERCLIP_TEST_FLAG");
|
||||
|
||||
@@ -45,13 +45,14 @@ export function buildLoginShellScript(input: {
|
||||
.map(([key, value]) => `${key}=${shellQuote(value)}`);
|
||||
const commandParts = [shellQuote(input.command), ...input.args.map(shellQuote)].join(" ");
|
||||
const stdinRedirect = input.stdinFile ? ` < ${shellQuote(input.stdinFile)}` : "";
|
||||
// Source the common login profiles before exec so the command runs with the
|
||||
// interactive-shell PATH. The wrapper sources no `nvm.sh`; the sandbox image
|
||||
// supplies node on the PATH.
|
||||
const lines = [
|
||||
'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
|
||||
'export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"',
|
||||
'[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" >/dev/null 2>&1 || true',
|
||||
];
|
||||
if (input.cwd) {
|
||||
lines.push(`cd ${shellQuote(input.cwd)}`);
|
||||
|
||||
@@ -32,6 +32,7 @@ import plugin, {
|
||||
setDaytonaTimingClockForTest,
|
||||
setDaytonaHandleFreshnessClockForTest,
|
||||
__resetDaytonaSandboxHandleCacheForTest,
|
||||
__getDaytonaWritableDirsForTest,
|
||||
} from "./plugin.js";
|
||||
import manifest from "./manifest.js";
|
||||
|
||||
@@ -1101,11 +1102,12 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
const [command, cwdArg, envArg, timeoutArg] = sandbox.process.executeCommand.mock.calls[0] as [string, unknown, unknown, number];
|
||||
expect(command).toMatch(/\/etc\/profile/);
|
||||
expect(command).toMatch(/"\$HOME\/\.profile"/);
|
||||
expect(command).toMatch(/cd '\/workspace'/);
|
||||
expect(command).not.toMatch(/nvm\.sh/);
|
||||
expect(command).toMatch(/&& cd '\/workspace'/);
|
||||
expect(command).toMatch(/&& env GIT_TERMINAL_PROMPT='0' GCM_INTERACTIVE='Never' GIT_ASKPASS='echo' SSH_ASKPASS='echo' SSH_ASKPASS_REQUIRE='force' FOO='bar' 'printf' 'hello'$/);
|
||||
expect(command).not.toMatch(/(?:^|&& )exec /);
|
||||
// cwd/env are baked into the login-shell command itself; we pass undefined
|
||||
// to the SDK so it doesn't run the cd before profile sourcing.
|
||||
// cwd/env are baked into the command itself; we pass undefined to the SDK
|
||||
// so its own cwd argument does not run before the caller env is applied.
|
||||
expect(cwdArg).toBeUndefined();
|
||||
expect(envArg).toBeUndefined();
|
||||
expect(timeoutArg).toBe(1);
|
||||
@@ -1182,7 +1184,8 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
);
|
||||
const [command] = sandbox.process.executeCommand.mock.calls[0] as [string];
|
||||
expect(command).toMatch(/\/etc\/profile/);
|
||||
expect(command).toMatch(/cd '\/workspace'/);
|
||||
expect(command).not.toMatch(/nvm\.sh/);
|
||||
expect(command).toMatch(/&& cd '\/workspace'/);
|
||||
expect(command).toMatch(/env .* 'cat' < '\/tmp\/paperclip-stdin-/);
|
||||
expect(command).not.toMatch(/(?:^|&& )exec /);
|
||||
expect(sandbox.fs.deleteFile).toHaveBeenCalledWith(expect.stringMatching(/^\/tmp\/paperclip-stdin-/));
|
||||
@@ -1307,37 +1310,12 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
expect(result?.stderr).toMatch(/unreachable|credentials/i);
|
||||
});
|
||||
|
||||
// ─── No-profile fast path (A2) ─────────────────────────────────────────────
|
||||
// The opt-in `noProfile` flag sheds the ~600 ms login-shell profile/nvm
|
||||
// sourcing for command classes whose binary resolves on the default PATH
|
||||
// (file-sync `tar`/`base64`/`mkdir`/`mv`), while every other exec surface
|
||||
// (env prefix, cwd, quoting, stdin, durationMs) is preserved byte-for-byte.
|
||||
it("test_no_profile_fast_path_omits_profile_sourcing", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox();
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
|
||||
await plugin.definition.onEnvironmentExecute?.({
|
||||
driverKey: "daytona",
|
||||
companyId: "company-1",
|
||||
environmentId: "env-1",
|
||||
config: { timeoutMs: 300000, reuseLease: false },
|
||||
lease: { providerLeaseId: "sandbox-123", metadata: {} },
|
||||
command: "tar",
|
||||
args: ["-xf", "/workspace/upload.tar", "-C", "/workspace"],
|
||||
cwd: "/workspace",
|
||||
noProfile: true,
|
||||
timeoutMs: 1000,
|
||||
});
|
||||
|
||||
const [command] = sandbox.process.executeCommand.mock.calls[0] as [string];
|
||||
expect(command).not.toMatch(/\/etc\/profile/);
|
||||
expect(command).not.toMatch(/nvm\.sh/);
|
||||
expect(command).not.toMatch(/NVM_DIR/);
|
||||
expect(command).not.toMatch(/\.bash_profile/);
|
||||
});
|
||||
|
||||
it("test_no_profile_fast_path_preserves_env_cwd_and_duration", async () => {
|
||||
// ─── Exec command shape ────────────────────────────────────────────────────
|
||||
// The wrapper sources the login profiles so `node` resolves on the reference
|
||||
// image, then runs the command. It no longer sources `nvm.sh`, while every
|
||||
// other exec surface (env prefix, cwd, quoting, stdin, durationMs) stays
|
||||
// intact.
|
||||
it("test_exec_command_preserves_env_cwd_and_duration", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox();
|
||||
sandbox.process.executeCommand.mockResolvedValue({
|
||||
@@ -1357,29 +1335,30 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
args: ["-d"],
|
||||
cwd: "/workspace",
|
||||
env: { FOO: "bar" },
|
||||
noProfile: true,
|
||||
timeoutMs: 1000,
|
||||
});
|
||||
|
||||
const [command] = sandbox.process.executeCommand.mock.calls[0] as [string];
|
||||
// The full exec surface is preserved on the fast path — only profile sourcing
|
||||
// is dropped. The command must still start with the `cd` (no profile lines
|
||||
// ahead of it) and carry the env prefix and noninteractive git defaults.
|
||||
expect(command).toMatch(/^cd '\/workspace' && env /);
|
||||
// The command sources the login profiles first, then runs the `cd` and the
|
||||
// env prefix with the noninteractive git defaults.
|
||||
expect(command).toMatch(/^if \[ -f \/etc\/profile \]/);
|
||||
expect(command).toMatch(/&& cd '\/workspace' && env /);
|
||||
expect(command).toMatch(/GIT_TERMINAL_PROMPT='0'/);
|
||||
expect(command).toMatch(/FOO='bar' 'base64' '-d'$/);
|
||||
expect(command).not.toMatch(/\/etc\/profile/);
|
||||
// durationMs attribution is unchanged on the fast path.
|
||||
expect(command).toMatch(/\/etc\/profile/);
|
||||
expect(command).not.toMatch(/nvm\.sh/);
|
||||
// durationMs attribution stays intact.
|
||||
expect(typeof (result!.metadata as Record<string, unknown>)?.durationMs).toBe("number");
|
||||
});
|
||||
|
||||
it("test_default_path_still_sources_profile", async () => {
|
||||
it("test_exec_command_sources_profile_without_nvm", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox();
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
|
||||
// No `noProfile` flag: the fail-safe default must still source the login
|
||||
// profile so node-launching execs resolve their nvm/profile PATH.
|
||||
// A node-launching exec resolves `node` through the login profiles, which
|
||||
// Daytona's non-login `executeCommand` shell does not source on its own. The
|
||||
// wrapper sources the profiles but no longer sources `nvm.sh`.
|
||||
await plugin.definition.onEnvironmentExecute?.({
|
||||
driverKey: "daytona",
|
||||
companyId: "company-1",
|
||||
@@ -1394,7 +1373,9 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
|
||||
const [command] = sandbox.process.executeCommand.mock.calls[0] as [string];
|
||||
expect(command).toMatch(/\/etc\/profile/);
|
||||
expect(command).toMatch(/nvm\.sh/);
|
||||
expect(command).toMatch(/"\$HOME\/\.profile"/);
|
||||
expect(command).not.toMatch(/nvm\.sh/);
|
||||
expect(command).not.toMatch(/NVM_DIR/);
|
||||
});
|
||||
|
||||
// ─── Per-lease started-sandbox handle cache ────────────────────────────────
|
||||
@@ -2156,6 +2137,35 @@ describe("Daytona sandbox provider plugin", () => {
|
||||
expect(mockGet).toHaveBeenCalledTimes(2);
|
||||
expect(second.process.executeCommand).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("realizes the workspace from the acquire-seeded handle without a client.get", async () => {
|
||||
process.env.DAYTONA_API_KEY = "host-key";
|
||||
const sandbox = createMockSandbox({ id: "sandbox-seed" });
|
||||
mockCreate.mockResolvedValue(sandbox);
|
||||
|
||||
const base = { driverKey: "daytona", companyId: "company-1", environmentId: "env-1" };
|
||||
const config = { image: "node:20", timeoutMs: 300000, reuseLease: false };
|
||||
|
||||
const lease = await plugin.definition.onEnvironmentAcquireLease?.({
|
||||
...base,
|
||||
runId: "run-1",
|
||||
config,
|
||||
});
|
||||
expect(lease?.providerLeaseId).toBe("sandbox-seed");
|
||||
|
||||
const realize = await plugin.definition.onEnvironmentRealizeWorkspace?.({
|
||||
...base,
|
||||
lease: { providerLeaseId: lease!.providerLeaseId, metadata: lease!.metadata },
|
||||
workspace: { remotePath: "/home/daytona/paperclip-workspace" },
|
||||
config,
|
||||
});
|
||||
|
||||
// Acquire seeded the handle under the exact scope realize reads, so realize
|
||||
// reuses it and never pays a real REST re-fetch.
|
||||
expect(mockGet).not.toHaveBeenCalled();
|
||||
expect(sandbox.fs.createFolder).toHaveBeenCalledWith("/home/daytona/paperclip-workspace", "755");
|
||||
expect(realize?.cwd).toBe("/home/daytona/paperclip-workspace");
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -2191,6 +2201,124 @@ describe("daytona native file-sync hooks", () => {
|
||||
expect(plugin.definition.onEnvironmentSyncOut).toBeTypeOf("function");
|
||||
});
|
||||
|
||||
it("records the writablePath destination of a staging-tar rw mapping, not the staging parent", async () => {
|
||||
const hostDir = await makeHostDir();
|
||||
const source = path.join(hostDir, "workspace.tar");
|
||||
await fs.writeFile(source, "bytes");
|
||||
|
||||
const sandbox = createMockSandbox();
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
|
||||
const params = {
|
||||
driverKey: "daytona",
|
||||
companyId: "company-1",
|
||||
environmentId: "env-1",
|
||||
config: { timeoutMs: 300000, reuseLease: false },
|
||||
lease: syncLease(),
|
||||
operations: [
|
||||
{
|
||||
operationId: "sync-op-rw",
|
||||
files: [
|
||||
{
|
||||
// The mapping uploads a staging tar under the runtime root, and a
|
||||
// post-upload command extracts it into the workspace directory. So
|
||||
// `writablePath` names the real read-write destination.
|
||||
sourcePath: source,
|
||||
targetPath: `${REMOTE_DIR}/.paperclip-runtime/workspace-upload.tar`,
|
||||
kind: "file" as const,
|
||||
access: "rw" as const,
|
||||
writablePath: REMOTE_DIR,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
await plugin.definition.onEnvironmentSyncIn?.(params);
|
||||
|
||||
// The set holds the extract destination, not the staging archive parent.
|
||||
const recorded = __getDaytonaWritableDirsForTest(params);
|
||||
expect(recorded).toContain(REMOTE_DIR);
|
||||
expect(recorded).not.toContain(`${REMOTE_DIR}/.paperclip-runtime`);
|
||||
});
|
||||
|
||||
it("falls back to the parent directory of an rw mapping with no writablePath", async () => {
|
||||
const hostDir = await makeHostDir();
|
||||
const source = path.join(hostDir, "in-place.txt");
|
||||
await fs.writeFile(source, "bytes");
|
||||
|
||||
const sandbox = createMockSandbox();
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
|
||||
const params = {
|
||||
driverKey: "daytona",
|
||||
companyId: "company-1",
|
||||
environmentId: "env-1",
|
||||
config: { timeoutMs: 300000, reuseLease: false },
|
||||
lease: syncLease(),
|
||||
operations: [
|
||||
{
|
||||
operationId: "sync-op-rw-inplace",
|
||||
files: [
|
||||
{
|
||||
// No post-upload extract, so the mapping writes `targetPath` in
|
||||
// place and the parent directory is the read-write destination.
|
||||
sourcePath: source,
|
||||
targetPath: `${REMOTE_DIR}/data/in-place.txt`,
|
||||
kind: "file" as const,
|
||||
access: "rw" as const,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
await plugin.definition.onEnvironmentSyncIn?.(params);
|
||||
|
||||
expect(__getDaytonaWritableDirsForTest(params)).toContain(`${REMOTE_DIR}/data`);
|
||||
});
|
||||
|
||||
it("skips ro and access-absent sync targets in the advisory writable set", async () => {
|
||||
const hostDir = await makeHostDir();
|
||||
const roSource = path.join(hostDir, "referenced");
|
||||
const defaultSource = path.join(hostDir, "default.tar");
|
||||
await fs.mkdir(roSource, { recursive: true });
|
||||
await fs.writeFile(path.join(roSource, "notes.md"), "reference");
|
||||
await fs.writeFile(defaultSource, "bytes");
|
||||
|
||||
const sandbox = createMockSandbox();
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
|
||||
const params = {
|
||||
driverKey: "daytona",
|
||||
companyId: "company-1",
|
||||
environmentId: "env-1",
|
||||
config: { timeoutMs: 300000, reuseLease: false },
|
||||
lease: syncLease(),
|
||||
operations: [
|
||||
{
|
||||
operationId: "sync-op-ro",
|
||||
files: [
|
||||
{
|
||||
sourcePath: roSource,
|
||||
targetPath: `${REMOTE_DIR}/.paperclip-runtime/project-proj-first`,
|
||||
kind: "directory" as const,
|
||||
access: "ro" as const,
|
||||
},
|
||||
{
|
||||
// An absent `access` defaults to read-only, so it is not recorded.
|
||||
sourcePath: defaultSource,
|
||||
targetPath: `${REMOTE_DIR}/.paperclip-runtime/default-upload.tar`,
|
||||
kind: "file" as const,
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
await plugin.definition.onEnvironmentSyncIn?.(params);
|
||||
|
||||
// Neither the ro directory nor the access-absent file directory is recorded.
|
||||
expect(__getDaytonaWritableDirsForTest(params)).toEqual([]);
|
||||
});
|
||||
|
||||
it("syncIn coalesces file mappings into one uploadFiles batch to reserved temp destinations, then one batched mv, applying secret mode via setFilePermissions before the rename", async () => {
|
||||
const hostDir = await makeHostDir();
|
||||
const secretSource = path.join(hostDir, "auth.json");
|
||||
@@ -2947,6 +3075,159 @@ describe("daytona native file-sync hooks", () => {
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
// -------------------------------------------------------------------------
|
||||
// Merged git-workspace operation. A git-backed workspace stage-sync rides ONE
|
||||
// operation whose `files` carry the git-history tar and the workspace-overlay
|
||||
// tar, with the two extract commands as ordered `postUploadCommands`. The
|
||||
// operation shares one mkdir, one confine guard, one `uploadFiles`, and one
|
||||
// rename exec.
|
||||
// -------------------------------------------------------------------------
|
||||
|
||||
it("stages a merged git-workspace operation as one uploadFiles batch and one rename exec, both extracts in order", async () => {
|
||||
const hostDir = await makeHostDir();
|
||||
const gitTar = path.join(hostDir, "git-workspace.tar");
|
||||
const overlayTar = path.join(hostDir, "workspace.tar");
|
||||
await fs.writeFile(gitTar, "git-bytes");
|
||||
await fs.writeFile(overlayTar, "overlay-bytes");
|
||||
const runtimeDir = `${REMOTE_DIR}/.paperclip-runtime/adapter`;
|
||||
|
||||
const sandbox = createMockSandbox();
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
|
||||
const gitExtract = "git-history-extract";
|
||||
const overlayExtract = "workspace-overlay-extract";
|
||||
const result = await plugin.definition.onEnvironmentSyncIn?.({
|
||||
driverKey: "daytona",
|
||||
companyId: "company-1",
|
||||
environmentId: "env-1",
|
||||
config: { timeoutMs: 300000, reuseLease: false },
|
||||
lease: syncLease(),
|
||||
operations: [
|
||||
{
|
||||
operationId: "merged-workspace",
|
||||
files: [
|
||||
{ sourcePath: gitTar, targetPath: `${runtimeDir}/git-workspace-upload.tar`, kind: "file" },
|
||||
{ sourcePath: overlayTar, targetPath: `${runtimeDir}/workspace-upload.tar`, kind: "file" },
|
||||
],
|
||||
postUploadCommands: [{ command: gitExtract }, { command: overlayExtract }],
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
// One bulk upload carries BOTH tars; one rename exec promotes both temps.
|
||||
expect(sandbox.fs.uploadFiles).toHaveBeenCalledTimes(1);
|
||||
const [uploads] = sandbox.fs.uploadFiles.mock.calls[0] as [Array<{ source: string; destination: string }>];
|
||||
expect(uploads).toHaveLength(2);
|
||||
const mvCalls = sandbox.process.executeCommand.mock.calls.filter(([cmd]: [string]) =>
|
||||
String(cmd).includes("mv -f"),
|
||||
);
|
||||
expect(mvCalls).toHaveLength(1);
|
||||
expect(String(mvCalls[0][0]).match(/mv -f /g)).toHaveLength(2);
|
||||
|
||||
// Both extract commands ran, in array order, AFTER the upload (git first).
|
||||
const orderOf = (cmd: string) => {
|
||||
const idx = sandbox.process.executeCommand.mock.calls.findIndex(([c]: [string]) => c === cmd);
|
||||
return sandbox.process.executeCommand.mock.invocationCallOrder[idx];
|
||||
};
|
||||
expect(orderOf(gitExtract)).toBeLessThan(orderOf(overlayExtract));
|
||||
expect(sandbox.fs.uploadFiles.mock.invocationCallOrder[0]).toBeLessThan(orderOf(gitExtract));
|
||||
|
||||
expect(result).toEqual({
|
||||
operations: [{
|
||||
operationId: "merged-workspace",
|
||||
filesTransferred: 2,
|
||||
bytesTransferred: "git-bytes".length + "overlay-bytes".length,
|
||||
}],
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects a merged operation when either tar mapping target escapes the remote dir, before uploading", async () => {
|
||||
const hostDir = await makeHostDir();
|
||||
const gitTar = path.join(hostDir, "git-workspace.tar");
|
||||
const overlayTar = path.join(hostDir, "workspace.tar");
|
||||
await fs.writeFile(gitTar, "git-bytes");
|
||||
await fs.writeFile(overlayTar, "overlay-bytes");
|
||||
const runtimeDir = `${REMOTE_DIR}/.paperclip-runtime/adapter`;
|
||||
|
||||
const sandbox = createMockSandbox();
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
|
||||
await expect(
|
||||
plugin.definition.onEnvironmentSyncIn?.({
|
||||
driverKey: "daytona",
|
||||
companyId: "company-1",
|
||||
environmentId: "env-1",
|
||||
config: { timeoutMs: 300000, reuseLease: false },
|
||||
lease: syncLease(),
|
||||
operations: [
|
||||
{
|
||||
operationId: "merged-escape",
|
||||
files: [
|
||||
{ sourcePath: gitTar, targetPath: `${runtimeDir}/git-workspace-upload.tar`, kind: "file" },
|
||||
// The overlay mapping target escapes the workspace remote dir.
|
||||
{ sourcePath: overlayTar, targetPath: `${REMOTE_DIR}/../../etc/workspace-upload.tar`, kind: "file" },
|
||||
],
|
||||
postUploadCommands: [{ command: "git-history-extract" }, { command: "workspace-overlay-extract" }],
|
||||
},
|
||||
],
|
||||
}),
|
||||
).rejects.toThrow(/escapes the workspace remote dir|not a confined absolute path/);
|
||||
|
||||
// Neither tar uploaded: the confine check on the escaping mapping trips first.
|
||||
expect(sandbox.fs.uploadFiles).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("stops the overlay and remove-deleted commands when the git extract fails (merged operation fail-fast)", async () => {
|
||||
const hostDir = await makeHostDir();
|
||||
const gitTar = path.join(hostDir, "git-workspace.tar");
|
||||
const overlayTar = path.join(hostDir, "workspace.tar");
|
||||
await fs.writeFile(gitTar, "git-bytes");
|
||||
await fs.writeFile(overlayTar, "overlay-bytes");
|
||||
const runtimeDir = `${REMOTE_DIR}/.paperclip-runtime/adapter`;
|
||||
|
||||
const sandbox = createMockSandbox();
|
||||
// The first (git-history) extract exits non-zero; every transfer/guard script
|
||||
// stays green so the fail-fast loop is the only thing that can trip this test.
|
||||
sandbox.process.executeCommand.mockImplementation(async (command: string) => {
|
||||
if (command === "git-history-extract") {
|
||||
return { exitCode: 5, result: "boom", artifacts: { stdout: "boom" } };
|
||||
}
|
||||
return { exitCode: 0, result: "", artifacts: { stdout: "" } };
|
||||
});
|
||||
mockGet.mockResolvedValue(sandbox);
|
||||
|
||||
await expect(
|
||||
plugin.definition.onEnvironmentSyncIn?.({
|
||||
driverKey: "daytona",
|
||||
companyId: "company-1",
|
||||
environmentId: "env-1",
|
||||
config: { timeoutMs: 300000, reuseLease: false },
|
||||
lease: syncLease(),
|
||||
operations: [
|
||||
{
|
||||
operationId: "merged-failfast",
|
||||
files: [
|
||||
{ sourcePath: gitTar, targetPath: `${runtimeDir}/git-workspace-upload.tar`, kind: "file" },
|
||||
{ sourcePath: overlayTar, targetPath: `${runtimeDir}/workspace-upload.tar`, kind: "file" },
|
||||
],
|
||||
postUploadCommands: [
|
||||
{ command: "git-history-extract" },
|
||||
{ command: "workspace-overlay-extract" },
|
||||
{ command: "remove-deleted-paths" },
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
).rejects.toThrow(/post-upload command failed \(exit 5\)/);
|
||||
|
||||
// Fail-fast: the overlay extract and the remove-deleted command never ran.
|
||||
const ran = (cmd: string) =>
|
||||
sandbox.process.executeCommand.mock.calls.some(([c]: [string]) => c === cmd);
|
||||
expect(ran("git-history-extract")).toBe(true);
|
||||
expect(ran("workspace-overlay-extract")).toBe(false);
|
||||
expect(ran("remove-deleted-paths")).toBe(false);
|
||||
});
|
||||
|
||||
it("rejects a post-upload command cwd that escapes the remote dir lexically, before any exec (C2)", async () => {
|
||||
const hostDir = await makeHostDir();
|
||||
const source = path.join(hostDir, "config.txt");
|
||||
@@ -3037,6 +3318,11 @@ describe("daytona native file-sync hooks", () => {
|
||||
|
||||
// Same operation, now with an (empty) postUploadCommands array — must be
|
||||
// byte-identical: an absent/empty command list adds zero execs.
|
||||
// Reset the process-scoped handle cache so the second operation fetches its
|
||||
// own `withEmpty` handle. Both operations reuse the same providerLeaseId, so
|
||||
// without this reset the cache serves the first `baseline` handle again and
|
||||
// `withEmpty` records zero execs.
|
||||
__resetDaytonaSandboxHandleCacheForTest();
|
||||
const withEmpty = createMockSandbox();
|
||||
mockGet.mockResolvedValue(withEmpty);
|
||||
await plugin.definition.onEnvironmentSyncIn?.({
|
||||
|
||||
@@ -36,6 +36,7 @@ import type {
|
||||
PluginEnvironmentSyncResult,
|
||||
PluginEnvironmentValidateConfigParams,
|
||||
PluginEnvironmentValidationResult,
|
||||
PluginSyncOperation,
|
||||
} from "@paperclipai/plugin-sdk";
|
||||
import { performSyncIn, performSyncOut } from "./file-sync.js";
|
||||
|
||||
@@ -655,18 +656,20 @@ function isGitNetworkCommand(command: string, args: string[]): boolean {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Mirror the E2B sandbox executor: source common login profiles (and nvm)
|
||||
// before running the command so Daytona one-shot calls see the same PATH an
|
||||
// interactive shell would. Without this, adapter probes can fail to resolve
|
||||
// CLIs that are installed via profile-driven PATH mutations inside the
|
||||
// sandbox image.
|
||||
// Build the one-shot exec command. Daytona's `executeCommand` runs the script
|
||||
// in a non-login shell, so it does not source `/etc/profile` on its own. The
|
||||
// Daytona reference image puts `node`, `claude`, and the other CLIs on the PATH
|
||||
// through `/etc/profile.d/00-restore-env.sh`, which only `/etc/profile` sources.
|
||||
// So the wrapper sources the login profiles itself; a non-login shell is then
|
||||
// enough to resolve the CLIs. The wrapper no longer sources `nvm.sh`; the
|
||||
// sandbox image supplies `node` on the PATH. See the sandbox runtime
|
||||
// requirements document.
|
||||
function buildLoginShellScript(input: {
|
||||
command: string;
|
||||
args: string[];
|
||||
cwd?: string;
|
||||
env?: Record<string, string>;
|
||||
stdinPath?: string;
|
||||
noProfile?: boolean;
|
||||
}): string {
|
||||
const callerEnv = input.env ?? {};
|
||||
for (const key of Object.keys(callerEnv)) {
|
||||
@@ -685,25 +688,17 @@ function buildLoginShellScript(input: {
|
||||
: commandParts;
|
||||
// Each `executeCommand` call runs in its own shell, so we don't `exec`-
|
||||
// replace it; running the command as the last `&&`-chained line is enough to
|
||||
// surface the right exit code. Env is interpolated after profile sourcing so
|
||||
// the caller's env wins over any defaults the profile exports.
|
||||
// surface the right exit code.
|
||||
const finalLine = envArgs.length > 0
|
||||
? `env ${envArgs.join(" ")} ${redirectedCommand}`
|
||||
: redirectedCommand;
|
||||
const profileSourcingLines = input.noProfile === true
|
||||
? []
|
||||
: [
|
||||
'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi',
|
||||
// .bash_profile typically sources .bashrc itself; only source .bashrc
|
||||
// directly when no .bash_profile exists to avoid double-running setup.
|
||||
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
|
||||
'export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"',
|
||||
'[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" >/dev/null 2>&1 || true',
|
||||
];
|
||||
const lines = [
|
||||
...profileSourcingLines,
|
||||
'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi',
|
||||
// .bash_profile typically sources .bashrc itself; only source .bashrc
|
||||
// directly when no .bash_profile exists to avoid double-running setup.
|
||||
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
|
||||
];
|
||||
if (input.cwd) {
|
||||
lines.push(`cd ${shellQuote(input.cwd)}`);
|
||||
@@ -1076,6 +1071,21 @@ const sandboxHandleCache = (() => {
|
||||
}
|
||||
}
|
||||
|
||||
// Seed the cache with a handle the caller already holds (e.g. the fresh handle
|
||||
// from `createSandbox` on a cold acquire), so the next `get` under the same
|
||||
// scope reuses it instead of paying a real `client.get`. The seed must land
|
||||
// under the exact composite key the reader uses, or the reader misses and the
|
||||
// saved round trip is lost. Assert the handle belongs to the lease so a caller
|
||||
// that builds a wrong scope fails loudly here instead of caching a foreign
|
||||
// handle.
|
||||
function seed(scope: SandboxScope, sandbox: Sandbox): void {
|
||||
assertHandleMatchesLease(sandbox, scope.providerLeaseId);
|
||||
entries.set(sandboxHandleCacheKey(scope), {
|
||||
sandbox: Promise.resolve(sandbox),
|
||||
verifiedAtMs: handleFreshnessNow(),
|
||||
});
|
||||
}
|
||||
|
||||
function clear(scope: SandboxScope): void {
|
||||
entries.delete(sandboxHandleCacheKey(scope));
|
||||
}
|
||||
@@ -1084,7 +1094,56 @@ const sandboxHandleCache = (() => {
|
||||
entries.clear();
|
||||
}
|
||||
|
||||
return { get, clear, reset, markFresh };
|
||||
return { get, seed, clear, reset, markFresh };
|
||||
})();
|
||||
|
||||
// Advisory writable-set store. It holds, per lease scope, the sandbox
|
||||
// directories that a sync operation declared read-write (`access: "rw"`). An
|
||||
// optional sandbox feedback wrapper reads this set later to bind those
|
||||
// directories read-write, so an agent gets real-time feedback when a write to a
|
||||
// non-persistent path fails. The store is advisory and best-effort in-memory
|
||||
// state: it adds no security (the ephemeral sandbox stays the only boundary),
|
||||
// and a cold store (for example after a worker restart) degrades to the
|
||||
// workspace baseline, never to a crash. The store is keyed the same way as
|
||||
// `sandboxHandleCache`, by `sandboxHandleCacheKey(scope)`.
|
||||
const sandboxHandleWritableDirs = (() => {
|
||||
const dirsByKey = new Map<string, Set<string>>();
|
||||
|
||||
// Record the read-write destination directory of every `access: "rw"`
|
||||
// mapping. Skip read-only mappings (`access` absent or `"ro"`). Read-only is
|
||||
// the safe default for an advisory signal.
|
||||
//
|
||||
// A workspace, git-history, or asset mapping uploads a tar archive, so its
|
||||
// `targetPath` is the staging archive under the runtime root, not the directory
|
||||
// that the post-upload extract command fills. For those mappings the author
|
||||
// sets `writablePath` to the final destination directory, so this records the
|
||||
// real read-write destination, not the staging parent. When `writablePath` is
|
||||
// absent the mapping writes `targetPath` in place, so the parent directory of
|
||||
// `targetPath` is the destination.
|
||||
function recordWritableTargets(scope: SandboxScope, operations: PluginSyncOperation[]): void {
|
||||
const key = sandboxHandleCacheKey(scope);
|
||||
for (const operation of operations) {
|
||||
for (const mapping of operation.files) {
|
||||
if (mapping.access !== "rw") continue;
|
||||
let dirs = dirsByKey.get(key);
|
||||
if (!dirs) {
|
||||
dirs = new Set<string>();
|
||||
dirsByKey.set(key, dirs);
|
||||
}
|
||||
dirs.add(mapping.writablePath ?? path.posix.dirname(mapping.targetPath));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function get(scope: SandboxScope): ReadonlySet<string> {
|
||||
return dirsByKey.get(sandboxHandleCacheKey(scope)) ?? new Set<string>();
|
||||
}
|
||||
|
||||
function reset(): void {
|
||||
dirsByKey.clear();
|
||||
}
|
||||
|
||||
return { recordWritableTargets, get, reset };
|
||||
})();
|
||||
|
||||
/**
|
||||
@@ -1097,6 +1156,29 @@ export function __resetDaytonaSandboxHandleCacheForTest(): void {
|
||||
sandboxHandleTeardownGates.reset();
|
||||
sandboxHandleActivityGates.reset();
|
||||
sandboxHandleLeaseAdmissionStates.reset();
|
||||
sandboxHandleWritableDirs.reset();
|
||||
}
|
||||
|
||||
/**
|
||||
* Test seam: read the advisory writable directories recorded for a sync scope.
|
||||
* The caller passes the same `onEnvironmentSyncIn` inputs, so this rebuilds the
|
||||
* exact scope key the hook used. Not used in production.
|
||||
*/
|
||||
export function __getDaytonaWritableDirsForTest(input: {
|
||||
driverKey: string;
|
||||
companyId: string;
|
||||
environmentId: string;
|
||||
lease: { providerLeaseId?: string | null };
|
||||
config: Record<string, unknown>;
|
||||
}): string[] {
|
||||
const scope: SandboxScope = {
|
||||
driverKey: input.driverKey,
|
||||
companyId: input.companyId,
|
||||
environmentId: input.environmentId,
|
||||
providerLeaseId: input.lease.providerLeaseId ?? "",
|
||||
config: parseDriverConfig(input.config),
|
||||
};
|
||||
return [...sandboxHandleWritableDirs.get(scope)];
|
||||
}
|
||||
|
||||
async function getSandbox(scope: SandboxScope, options: SandboxLookupOptions = {}): Promise<Sandbox> {
|
||||
@@ -1158,16 +1240,15 @@ async function executeOneShot(
|
||||
cwd: params.cwd,
|
||||
env: params.env,
|
||||
stdinPath: stdinPath ?? undefined,
|
||||
noProfile: params.noProfile === true,
|
||||
});
|
||||
|
||||
// Pass cwd undefined: `buildLoginShellScript` already injects `cd` after
|
||||
// profile sourcing when params.cwd is set, and the Daytona executor's own
|
||||
// cwd argument runs before our login-shell init, which is the wrong order
|
||||
// (env from .bashrc would override caller env).
|
||||
// Time only the `executeCommand` REST round-trip (Open Q1) — the ~600ms
|
||||
// login-shell wrapper — so the caller can attribute a step's exec time to
|
||||
// the provider boundary via the free-form `metadata.durationMs`.
|
||||
// Pass cwd undefined: `buildLoginShellScript` already injects the `cd` after
|
||||
// it sources the login profiles, when params.cwd is set. The Daytona
|
||||
// executor's own cwd argument runs before that profile sourcing, which is
|
||||
// the wrong order (a profile could reset the caller env).
|
||||
// Time only the `executeCommand` REST round-trip so the caller can
|
||||
// attribute a step's exec time to the provider boundary through the
|
||||
// free-form `metadata.durationMs`.
|
||||
execStart = timingNow();
|
||||
const result = await sandbox.process.executeCommand(command, undefined, undefined, timeoutSeconds);
|
||||
const durationMs = timingNow() - execStart;
|
||||
@@ -1338,6 +1419,19 @@ const plugin = definePlugin({
|
||||
providerLeaseId: sandbox.id,
|
||||
config,
|
||||
});
|
||||
// Seed the handle cache with the fresh handle under the exact scope that
|
||||
// `onEnvironmentRealizeWorkspace` reads (providerLeaseId === sandbox.id).
|
||||
// Realize then reuses this handle instead of paying a real `client.get`.
|
||||
sandboxHandleCache.seed(
|
||||
{
|
||||
driverKey: params.driverKey,
|
||||
companyId: params.companyId,
|
||||
environmentId: params.environmentId,
|
||||
providerLeaseId: sandbox.id,
|
||||
config,
|
||||
},
|
||||
sandbox,
|
||||
);
|
||||
return {
|
||||
providerLeaseId: sandbox.id,
|
||||
metadata: leaseMetadata({
|
||||
@@ -1853,6 +1947,9 @@ const plugin = definePlugin({
|
||||
providerLeaseId: params.lease.providerLeaseId,
|
||||
config,
|
||||
};
|
||||
// Collect the advisory read-write destinations for this scope. This records
|
||||
// intent only; it does not change the transfer below.
|
||||
sandboxHandleWritableDirs.recordWritableTargets(scope, params.operations);
|
||||
return await withSandboxActivityGate(scope, async () => {
|
||||
const sandbox = await getSandbox(scope, { bypassTeardownGate: true });
|
||||
await ensureSandboxStarted(sandbox, timeoutSeconds);
|
||||
|
||||
@@ -364,6 +364,9 @@ describe("E2B sandbox provider plugin", () => {
|
||||
expect(fgCall).toBeDefined();
|
||||
if (!fgCall) throw new Error("fgCall not found");
|
||||
expect(fgCall[0]).toMatch(/\.profile/);
|
||||
// The wrapper sources no `nvm.sh`; the sandbox image supplies node on PATH.
|
||||
expect(fgCall[0]).not.toMatch(/nvm\.sh/);
|
||||
expect(fgCall[0]).not.toMatch(/NVM_DIR/);
|
||||
expect(fgCall[0]).toMatch(/exec env FOO='bar' 'printf' 'hello'$/);
|
||||
expect(fgCall[1]).toEqual(expect.objectContaining({ cwd: "/workspace", timeoutMs: 1000 }));
|
||||
expect(fgCall[1]).not.toHaveProperty("envs");
|
||||
|
||||
@@ -152,13 +152,13 @@ function isValidShellEnvKey(value: string) {
|
||||
return /^[A-Za-z_][A-Za-z0-9_]*$/.test(value);
|
||||
}
|
||||
|
||||
// Mirror SSH's buildSshSpawnTarget: source the user's login profiles (and nvm)
|
||||
// before exec so commands run with the same PATH the user sees in an
|
||||
// interactive shell. e2b's `sandbox.commands.run` otherwise spawns a
|
||||
// non-login, non-interactive shell whose PATH does not include npm-globals,
|
||||
// nvm shims, or anything else the template installs via .profile/.bashrc —
|
||||
// which makes the hello probe fail with `exec: <cli>: not found` even when
|
||||
// the binary is on disk.
|
||||
// Source the user's login profiles before exec so commands run with the same
|
||||
// PATH the user sees in an interactive shell. e2b's `sandbox.commands.run`
|
||||
// otherwise spawns a non-login, non-interactive shell whose PATH does not
|
||||
// include npm-globals or anything else the template installs via
|
||||
// .profile/.bashrc — which makes the hello probe fail with
|
||||
// `exec: <cli>: not found` even when the binary is on disk. The wrapper no
|
||||
// longer sources `nvm.sh`; the sandbox image supplies `node` on the PATH.
|
||||
function buildLoginShellScript(input: {
|
||||
command: string;
|
||||
args: string[];
|
||||
@@ -186,8 +186,6 @@ function buildLoginShellScript(input: {
|
||||
// .bash_profile -> .bashrc.
|
||||
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
|
||||
'export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"',
|
||||
'[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" >/dev/null 2>&1 || true',
|
||||
execLine,
|
||||
].join(" && ");
|
||||
}
|
||||
|
||||
@@ -576,8 +576,12 @@ describe("exe.dev sandbox provider plugin", () => {
|
||||
|
||||
expect(spawnMock).toHaveBeenCalledTimes(1);
|
||||
expect(spawnMock.mock.calls[0]?.[0]).toBe("ssh");
|
||||
expect(String(spawnMock.mock.calls[0]?.[1]?.at(-1) ?? "")).toContain("/workspace");
|
||||
expect(String(spawnMock.mock.calls[0]?.[1]?.at(-1) ?? "")).toContain("FOO='");
|
||||
const remoteScript = String(spawnMock.mock.calls[0]?.[1]?.at(-1) ?? "");
|
||||
expect(remoteScript).toContain("/workspace");
|
||||
expect(remoteScript).toContain("FOO='");
|
||||
// The wrapper sources no `nvm.sh`; the sandbox image supplies node on PATH.
|
||||
expect(remoteScript).not.toContain("nvm.sh");
|
||||
expect(remoteScript).not.toContain("NVM_DIR");
|
||||
const child = spawnMock.mock.results[0]?.value as MockChildProcess;
|
||||
expect(child.stdin.written).toBe("input-body");
|
||||
expect(child.stdin.ended).toBe(true);
|
||||
|
||||
@@ -537,13 +537,14 @@ function buildLoginShellScript(input: {
|
||||
const finalLine = envArgs.length > 0
|
||||
? `exec env ${envArgs.join(" ")} ${commandParts}`
|
||||
: `exec ${commandParts}`;
|
||||
// Source the common login profiles before exec so the command runs with the
|
||||
// interactive-shell PATH. The wrapper sources no `nvm.sh`; the sandbox image
|
||||
// supplies node on the PATH.
|
||||
const lines = [
|
||||
'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
|
||||
'export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"',
|
||||
'[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" >/dev/null 2>&1 || true',
|
||||
];
|
||||
if (input.cwd) {
|
||||
lines.push(`cd ${shellQuote(input.cwd)}`);
|
||||
|
||||
@@ -541,7 +541,7 @@ describe("Modal sandbox provider plugin", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("executes commands with a login-shell wrapper that injects env after profile sourcing", async () => {
|
||||
it("executes commands with a non-login-shell wrapper that injects env after profile sourcing", async () => {
|
||||
const sandbox = createFakeSandbox({
|
||||
execImpl: async (argv: string[]) =>
|
||||
makeFakeProcess({
|
||||
@@ -568,9 +568,12 @@ describe("Modal sandbox provider plugin", () => {
|
||||
expect(sandbox.execCalls).toHaveLength(1);
|
||||
const call = sandbox.execCalls[0]!;
|
||||
expect(call.argv[0]).toBe("sh");
|
||||
expect(call.argv[1]).toBe("-lc");
|
||||
expect(call.argv[1]).toBe("-c");
|
||||
const script = call.argv[2]!;
|
||||
expect(script).toMatch(/\/etc\/profile/);
|
||||
// The wrapper sources no `nvm.sh`; the sandbox image supplies node on PATH.
|
||||
expect(script).not.toMatch(/nvm\.sh/);
|
||||
expect(script).not.toMatch(/NVM_DIR/);
|
||||
expect(script).toMatch(/cd '\/srv\/work'/);
|
||||
expect(script).toMatch(/&& exec env FOO='bar' 'printf' 'hello'$/);
|
||||
expect(call.params).toMatchObject({
|
||||
|
||||
@@ -213,11 +213,13 @@ function isValidShellEnvKey(value: string): boolean {
|
||||
|
||||
// Modal's `sandbox.exec` takes an argv array and bypasses the shell entirely,
|
||||
// so adapter probes that rely on PATH mutations from /etc/profile or ~/.bashrc
|
||||
// do not work without an explicit login shell. Mirroring the Daytona / E2B
|
||||
// providers, wrap the user command in a `sh -lc` script that sources common
|
||||
// login profiles plus nvm before invoking it. Env is set after profile sourcing
|
||||
// so caller env wins; stdin is staged to a temp file and shell-redirected so
|
||||
// fast-failing commands do not race a streaming stdin writer.
|
||||
// do not work without an explicit profile source. Wrap the user command in a
|
||||
// `sh -c` script that sources the common login profiles before it runs. The
|
||||
// script sources no `nvm.sh`; the sandbox image supplies node on the PATH. The
|
||||
// script sources the profiles itself, so a non-login shell (`sh -c`) is enough.
|
||||
// Env is set after profile sourcing so caller env wins; stdin is staged to a
|
||||
// temp file and shell-redirected so fast-failing commands do not race a
|
||||
// streaming stdin writer.
|
||||
function buildLoginShellScript(input: {
|
||||
command: string;
|
||||
args: string[];
|
||||
@@ -244,8 +246,6 @@ function buildLoginShellScript(input: {
|
||||
'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
|
||||
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
|
||||
'export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"',
|
||||
'[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" >/dev/null 2>&1 || true',
|
||||
];
|
||||
if (input.cwd) {
|
||||
lines.push(`cd ${shellQuote(input.cwd)}`);
|
||||
@@ -624,7 +624,7 @@ const plugin = definePlugin({
|
||||
env: params.env,
|
||||
stdinPath: stdinPath ?? undefined,
|
||||
});
|
||||
const proc = await sandbox.exec(["sh", "-lc", script], {
|
||||
const proc = await sandbox.exec(["sh", "-c", script], {
|
||||
timeoutMs: callerTimeoutMs,
|
||||
stdout: "pipe",
|
||||
stderr: "pipe",
|
||||
|
||||
@@ -651,8 +651,6 @@ export interface PluginEnvironmentExecuteParams extends PluginEnvironmentDriverB
|
||||
env?: Record<string, string>;
|
||||
stdin?: string;
|
||||
timeoutMs?: number;
|
||||
/** Skip login-shell profile sourcing when the command already resolves on the sandbox default PATH. */
|
||||
noProfile?: boolean;
|
||||
}
|
||||
|
||||
export interface PluginEnvironmentExecuteResult {
|
||||
@@ -693,6 +691,30 @@ export interface PluginSyncFileMapping {
|
||||
* as links; `true` dereferences them to their target bytes. Mirrors tar's `-h`.
|
||||
*/
|
||||
followSymlinks?: boolean;
|
||||
/**
|
||||
* Advisory read-write intent for the sandbox target. `"rw"` means the author
|
||||
* expects the agent to change the bytes at the target and keep the change.
|
||||
* `"ro"` means the target is a read-only tree. An absent value defaults to
|
||||
* `"ro"` (read-only is the safe default for an advisory signal).
|
||||
*
|
||||
* This field is advisory metadata for an optional sandbox feedback wrapper. It
|
||||
* does not change the transfer and adds no security. A provider may read it to
|
||||
* bind the read-write targets read-write under the wrapper, but the ephemeral
|
||||
* sandbox stays the only security boundary.
|
||||
*/
|
||||
access?: "rw" | "ro";
|
||||
/**
|
||||
* The sandbox directory that becomes read-write when `access` is `"rw"` and a
|
||||
* post-upload command extracts `targetPath` into a different directory. A
|
||||
* workspace, git-history, or asset mapping uploads a tar archive, so its
|
||||
* `targetPath` is the staging archive under the runtime root, not the directory
|
||||
* that the extract command fills. This field names that final destination
|
||||
* directory, so a consumer records the real read-write destination, not the
|
||||
* staging parent. When absent, the read-write destination is the parent
|
||||
* directory of `targetPath`. This field is advisory and ignored when `access`
|
||||
* is not `"rw"`.
|
||||
*/
|
||||
writablePath?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -143,14 +143,6 @@ export const INSTANCE_FEATURE_CATALOG: Record<InstanceFeatureKey, FeatureCatalog
|
||||
cloudDefault: false,
|
||||
selfHostedDefault: false,
|
||||
},
|
||||
enableCloudSync: {
|
||||
title: "Cloud Sync",
|
||||
description:
|
||||
"Show local Paperclip Cloud upstream connection, preview, push, retry, and activation review surfaces.",
|
||||
tier: "managed",
|
||||
cloudDefault: false,
|
||||
selfHostedDefault: false,
|
||||
},
|
||||
enableExternalObjects: {
|
||||
title: "External Objects",
|
||||
description:
|
||||
|
||||
@@ -593,6 +593,8 @@ export type {
|
||||
CompanySkillForkReassignment,
|
||||
CompanySkillForkResult,
|
||||
CompanySkillForkPrecheckResult,
|
||||
CompanySkillRenameRequest,
|
||||
CompanySkillRenameResult,
|
||||
CompanySkillUpdateRequest,
|
||||
CompanySkillUpdateStatus,
|
||||
CompanySkillAuditSeverity,
|
||||
@@ -1013,6 +1015,9 @@ export type {
|
||||
CompanyPortabilityFileEntry,
|
||||
CompanyPortabilityCompanyManifestEntry,
|
||||
CompanyPortabilitySidebarOrder,
|
||||
CompanyPortabilityLabelManifestEntry,
|
||||
CompanyPortabilityBlobManifestEntry,
|
||||
CompanyPortabilityEmbeddedAssetManifestEntry,
|
||||
CompanyPortabilityAgentManifestEntry,
|
||||
CompanyPortabilitySkillManifestEntry,
|
||||
CompanyPortabilityProjectManifestEntry,
|
||||
@@ -1020,6 +1025,10 @@ export type {
|
||||
CompanyPortabilityIssueRoutineTriggerManifestEntry,
|
||||
CompanyPortabilityIssueRoutineManifestEntry,
|
||||
CompanyPortabilityIssueCommentManifestEntry,
|
||||
CompanyPortabilityIssueDocumentManifestEntry,
|
||||
CompanyPortabilityIssueWorkProductManifestEntry,
|
||||
CompanyPortabilityIssueMonitorManifestEntry,
|
||||
CompanyPortabilityIssueAttachmentManifestEntry,
|
||||
CompanyPortabilityIssueManifestEntry,
|
||||
CompanyPortabilityManifest,
|
||||
CompanyPortabilityExportResult,
|
||||
@@ -1389,22 +1398,6 @@ export {
|
||||
SMOKE_RUN_TRIGGERS,
|
||||
} from "./types/smoke-lab.js";
|
||||
|
||||
export type {
|
||||
CloudUpstreamConnectStartResponse,
|
||||
CloudUpstreamActivationDecision,
|
||||
CloudUpstreamActivationEntityType,
|
||||
CloudUpstreamConnection,
|
||||
CloudUpstreamConflict,
|
||||
CloudUpstreamPreview,
|
||||
CloudUpstreamRun,
|
||||
CloudUpstreamRunEvent,
|
||||
CloudUpstreamsState,
|
||||
CloudUpstreamStep,
|
||||
CloudUpstreamSummaryCount,
|
||||
CloudUpstreamTarget,
|
||||
CloudUpstreamWarning,
|
||||
} from "./types/cloud-upstream.js";
|
||||
|
||||
export type { ServerGitInfo, ServerGitLocalChanges, ServerInfoSnapshot } from "./types/server-info.js";
|
||||
|
||||
export {
|
||||
@@ -1967,6 +1960,8 @@ export {
|
||||
companySkillCommentCreateSchema,
|
||||
companySkillCommentUpdateSchema,
|
||||
companySkillForkSchema,
|
||||
companySkillRenameSchema,
|
||||
companySkillRenameResultSchema,
|
||||
companySkillUpdateSchema,
|
||||
companySkillUpdateStatusSchema,
|
||||
companySkillAuditFindingSchema,
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
buildExportFidelityWarnings,
|
||||
normalizeExportFidelityCounts,
|
||||
type ExportFidelityCounts,
|
||||
} from "./portability-fidelity.js";
|
||||
|
||||
const zeroCounts: ExportFidelityCounts = {
|
||||
labelDefinitions: 0,
|
||||
issueLabelReferences: 0,
|
||||
issueBlockerRelations: 0,
|
||||
issueDocuments: 0,
|
||||
issueWorkProducts: 0,
|
||||
issueAttachments: 0,
|
||||
approvals: 0,
|
||||
costEvents: 0,
|
||||
activityLogEntries: 0,
|
||||
issueMonitors: 0,
|
||||
};
|
||||
|
||||
describe("buildExportFidelityWarnings", () => {
|
||||
it("returns no warnings when the export carries everything", () => {
|
||||
expect(buildExportFidelityWarnings(zeroCounts)).toEqual([]);
|
||||
});
|
||||
|
||||
it("emits no warnings for data the bundle now carries", () => {
|
||||
expect(buildExportFidelityWarnings({
|
||||
...zeroCounts,
|
||||
labelDefinitions: 2,
|
||||
issueLabelReferences: 3,
|
||||
issueBlockerRelations: 2,
|
||||
issueDocuments: 1,
|
||||
issueWorkProducts: 3,
|
||||
issueAttachments: 4,
|
||||
issueMonitors: 8,
|
||||
})).toEqual([]);
|
||||
});
|
||||
|
||||
it("emits one warning per unsupported data category with counts", () => {
|
||||
const warnings = buildExportFidelityWarnings({
|
||||
...zeroCounts,
|
||||
approvals: 5,
|
||||
costEvents: 6,
|
||||
activityLogEntries: 7,
|
||||
});
|
||||
expect(warnings.map((warning) => warning.code)).toEqual([
|
||||
"approvals_not_exported",
|
||||
"cost_history_not_exported",
|
||||
"activity_history_not_exported",
|
||||
]);
|
||||
expect(warnings.every((warning) => warning.severity === "warning")).toBe(true);
|
||||
expect(warnings[0]?.message).toBe("5 approvals are not included in the export bundle.");
|
||||
expect(warnings[1]?.message).toBe("6 cost events are not included in the export bundle.");
|
||||
});
|
||||
});
|
||||
|
||||
describe("normalizeExportFidelityCounts", () => {
|
||||
it("round-trips a valid counts object", () => {
|
||||
const counts = { ...zeroCounts, issueAttachments: 12 };
|
||||
expect(normalizeExportFidelityCounts(counts)).toEqual(counts);
|
||||
});
|
||||
|
||||
it("rejects non-objects, arrays, and missing keys", () => {
|
||||
expect(normalizeExportFidelityCounts(null)).toBeNull();
|
||||
expect(normalizeExportFidelityCounts([])).toBeNull();
|
||||
expect(normalizeExportFidelityCounts("counts")).toBeNull();
|
||||
const { issueMonitors: _dropped, ...partial } = zeroCounts;
|
||||
expect(normalizeExportFidelityCounts(partial)).toBeNull();
|
||||
});
|
||||
|
||||
it("rejects negative and non-finite values", () => {
|
||||
expect(normalizeExportFidelityCounts({ ...zeroCounts, approvals: -1 })).toBeNull();
|
||||
expect(normalizeExportFidelityCounts({ ...zeroCounts, approvals: Number.NaN })).toBeNull();
|
||||
expect(normalizeExportFidelityCounts({ ...zeroCounts, approvals: Number.POSITIVE_INFINITY })).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,64 @@
|
||||
export type PortabilityFidelitySeverity = "info" | "warning" | "blocker";
|
||||
|
||||
export interface PortabilityFidelityWarning {
|
||||
code: string;
|
||||
severity: PortabilityFidelitySeverity;
|
||||
message: string;
|
||||
}
|
||||
|
||||
export const EXPORT_FIDELITY_REPORT_SCHEMA = "paperclip-export-fidelity-v1";
|
||||
|
||||
export const EXPORT_FIDELITY_COUNT_KEYS = [
|
||||
"labelDefinitions",
|
||||
"issueLabelReferences",
|
||||
"issueBlockerRelations",
|
||||
"issueDocuments",
|
||||
"issueWorkProducts",
|
||||
"issueAttachments",
|
||||
"approvals",
|
||||
"costEvents",
|
||||
"activityLogEntries",
|
||||
"issueMonitors",
|
||||
] as const;
|
||||
|
||||
export type ExportFidelityCounts = Record<(typeof EXPORT_FIDELITY_COUNT_KEYS)[number], number>;
|
||||
|
||||
export interface ExportFidelityReport {
|
||||
schema: typeof EXPORT_FIDELITY_REPORT_SCHEMA;
|
||||
companyId: string;
|
||||
counts: ExportFidelityCounts;
|
||||
warnings: PortabilityFidelityWarning[];
|
||||
generatedAt: string;
|
||||
}
|
||||
|
||||
const UNSUPPORTED_DATA_WARNINGS: ReadonlyArray<[code: string, countKey: keyof ExportFidelityCounts, singular: string, plural: string]> = [
|
||||
["approvals_not_exported", "approvals", "approval", "approvals"],
|
||||
["cost_history_not_exported", "costEvents", "cost event", "cost events"],
|
||||
["activity_history_not_exported", "activityLogEntries", "activity log entry", "activity log entries"],
|
||||
];
|
||||
|
||||
export function buildExportFidelityWarnings(counts: ExportFidelityCounts): PortabilityFidelityWarning[] {
|
||||
const warnings: PortabilityFidelityWarning[] = [];
|
||||
for (const [code, countKey, singular, plural] of UNSUPPORTED_DATA_WARNINGS) {
|
||||
const rowCount = counts[countKey];
|
||||
if (rowCount <= 0) continue;
|
||||
warnings.push({
|
||||
code,
|
||||
severity: "warning",
|
||||
message: `${rowCount} ${rowCount === 1 ? `${singular} is` : `${plural} are`} not included in the export bundle.`,
|
||||
});
|
||||
}
|
||||
return warnings;
|
||||
}
|
||||
|
||||
export function normalizeExportFidelityCounts(value: unknown): ExportFidelityCounts | null {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) return null;
|
||||
const record = value as Record<string, unknown>;
|
||||
const counts = {} as Record<(typeof EXPORT_FIDELITY_COUNT_KEYS)[number], number>;
|
||||
for (const key of EXPORT_FIDELITY_COUNT_KEYS) {
|
||||
const raw = record[key];
|
||||
if (typeof raw !== "number" || !Number.isFinite(raw) || raw < 0) return null;
|
||||
counts[key] = raw;
|
||||
}
|
||||
return counts;
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
export type NormalizedSha256 = `sha256:${string}`;
|
||||
|
||||
export function normalizedContentHash(value: unknown): NormalizedSha256 {
|
||||
return `sha256:${createHash("sha256").update(canonicalJson(value)).digest("hex")}`;
|
||||
}
|
||||
|
||||
export function canonicalJson(value: unknown): string {
|
||||
return JSON.stringify(sortJson(value));
|
||||
}
|
||||
|
||||
export function sha256HexOfBytes(data: Uint8Array): string {
|
||||
return createHash("sha256").update(data).digest("hex");
|
||||
}
|
||||
|
||||
function sortJson(value: unknown): unknown {
|
||||
if (Array.isArray(value)) return value.map(sortJson);
|
||||
if (typeof value !== "object" || value === null) return value;
|
||||
return Object.fromEntries(
|
||||
Object.entries(value as Record<string, unknown>)
|
||||
.sort(([left], [right]) => left.localeCompare(right))
|
||||
.map(([key, entry]) => [key, sortJson(entry)]),
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,303 @@
|
||||
import { deflateRawSync } from "node:zlib";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
bytesToPortableFileEntry,
|
||||
isBlobStorePath,
|
||||
readZipArchive,
|
||||
} from "./portability-zip.js";
|
||||
|
||||
// A minimal, faithful zip writer so the node reader can be round-tripped
|
||||
// against both STORE (method 0) and DEFLATE (method 8) entries. The layout
|
||||
// matches the browser writer in ui/src/lib/zip.ts: local file headers, then a
|
||||
// central directory, then the end-of-central-directory record.
|
||||
const crcTable = (() => {
|
||||
const table = new Uint32Array(256);
|
||||
for (let i = 0; i < 256; i += 1) {
|
||||
let crc = i;
|
||||
for (let bit = 0; bit < 8; bit += 1) {
|
||||
crc = (crc & 1) === 1 ? (crc >>> 1) ^ 0xedb88320 : crc >>> 1;
|
||||
}
|
||||
table[i] = crc >>> 0;
|
||||
}
|
||||
return table;
|
||||
})();
|
||||
|
||||
function crc32(bytes: Uint8Array) {
|
||||
let crc = 0xffffffff;
|
||||
for (const byte of bytes) crc = (crc >>> 8) ^ crcTable[(crc ^ byte) & 0xff]!;
|
||||
return (crc ^ 0xffffffff) >>> 0;
|
||||
}
|
||||
|
||||
interface ZipInput {
|
||||
path: string;
|
||||
bytes: Uint8Array;
|
||||
method?: 0 | 8;
|
||||
}
|
||||
|
||||
function buildZip(entries: ZipInput[], rootPath: string): Uint8Array {
|
||||
const encoder = new TextEncoder();
|
||||
const localChunks: Buffer[] = [];
|
||||
const centralChunks: Buffer[] = [];
|
||||
let localOffset = 0;
|
||||
|
||||
for (const entry of entries) {
|
||||
const method = entry.method ?? 0;
|
||||
const fileName = encoder.encode(`${rootPath}/${entry.path}`);
|
||||
const checksum = crc32(entry.bytes);
|
||||
const body = method === 8 ? deflateRawSync(Buffer.from(entry.bytes)) : Buffer.from(entry.bytes);
|
||||
|
||||
const localHeader = Buffer.alloc(30 + fileName.length);
|
||||
localHeader.writeUInt32LE(0x04034b50, 0);
|
||||
localHeader.writeUInt16LE(20, 4);
|
||||
localHeader.writeUInt16LE(0x0800, 6);
|
||||
localHeader.writeUInt16LE(method, 8);
|
||||
localHeader.writeUInt32LE(checksum, 14);
|
||||
localHeader.writeUInt32LE(body.length, 18);
|
||||
localHeader.writeUInt32LE(entry.bytes.length, 22);
|
||||
localHeader.writeUInt16LE(fileName.length, 26);
|
||||
Buffer.from(fileName).copy(localHeader, 30);
|
||||
|
||||
const centralHeader = Buffer.alloc(46 + fileName.length);
|
||||
centralHeader.writeUInt32LE(0x02014b50, 0);
|
||||
centralHeader.writeUInt16LE(20, 4);
|
||||
centralHeader.writeUInt16LE(20, 6);
|
||||
centralHeader.writeUInt16LE(0x0800, 8);
|
||||
centralHeader.writeUInt16LE(method, 10);
|
||||
centralHeader.writeUInt32LE(checksum, 16);
|
||||
centralHeader.writeUInt32LE(body.length, 20);
|
||||
centralHeader.writeUInt32LE(entry.bytes.length, 24);
|
||||
centralHeader.writeUInt16LE(fileName.length, 28);
|
||||
centralHeader.writeUInt32LE(localOffset, 42);
|
||||
Buffer.from(fileName).copy(centralHeader, 46);
|
||||
|
||||
localChunks.push(localHeader, body);
|
||||
centralChunks.push(centralHeader);
|
||||
localOffset += localHeader.length + body.length;
|
||||
}
|
||||
|
||||
const centralDirectory = Buffer.concat(centralChunks);
|
||||
const eocd = Buffer.alloc(22);
|
||||
eocd.writeUInt32LE(0x06054b50, 0);
|
||||
eocd.writeUInt16LE(entries.length, 8);
|
||||
eocd.writeUInt16LE(entries.length, 10);
|
||||
eocd.writeUInt32LE(centralDirectory.length, 12);
|
||||
eocd.writeUInt32LE(localOffset, 16);
|
||||
|
||||
return new Uint8Array(Buffer.concat([...localChunks, centralDirectory, eocd]));
|
||||
}
|
||||
|
||||
describe("isBlobStorePath", () => {
|
||||
it("matches blobs/ entries at the archive root and under a package root", () => {
|
||||
expect(isBlobStorePath("blobs/4f2d1c9a")).toBe(true);
|
||||
expect(isBlobStorePath("paperclip-demo/blobs/4f2d1c9a")).toBe(true);
|
||||
expect(isBlobStorePath("tasks/pap-1/TASK.md")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("bytesToPortableFileEntry", () => {
|
||||
it("keeps blobs/ entries as base64 octet streams regardless of extension", () => {
|
||||
const bytes = new Uint8Array([0x00, 0x01, 0x80, 0xfe, 0xff]);
|
||||
expect(bytesToPortableFileEntry("blobs/4f2d1c9a", bytes)).toEqual({
|
||||
encoding: "base64",
|
||||
data: Buffer.from(bytes).toString("base64"),
|
||||
contentType: "application/octet-stream",
|
||||
});
|
||||
});
|
||||
|
||||
it("decodes valid UTF-8 entries to text and falls back to base64 for invalid bytes", () => {
|
||||
const text = new TextEncoder().encode("# Notes\n\ncafé ✅\n");
|
||||
expect(bytesToPortableFileEntry("tasks/pap-1/TASK.md", text)).toBe("# Notes\n\ncafé ✅\n");
|
||||
const invalid = new Uint8Array([0x68, 0x69, 0xff, 0xfe, 0xc0]);
|
||||
expect(bytesToPortableFileEntry("tasks/pap-1/raw", invalid)).toEqual({
|
||||
encoding: "base64",
|
||||
data: Buffer.from(invalid).toString("base64"),
|
||||
contentType: "application/octet-stream",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("readZipArchive", () => {
|
||||
it("round-trips STORE, DEFLATE, and base64 blob entries byte-exactly and strips the shared root", async () => {
|
||||
const blobBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff, 0x13, 0x37]);
|
||||
// A text body large and repetitive enough that DEFLATE actually shrinks it,
|
||||
// so the DEFLATE decode path is exercised, not just written.
|
||||
const deflated = `# Weekly report\n${"paperclip ".repeat(512)}\n`;
|
||||
|
||||
const archive = buildZip(
|
||||
[
|
||||
{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n"), method: 0 },
|
||||
{ path: "reports/weekly.md", bytes: new TextEncoder().encode(deflated), method: 8 },
|
||||
{ path: "blobs/4f2d1c9a", bytes: blobBytes, method: 0 },
|
||||
],
|
||||
"paperclip-demo",
|
||||
);
|
||||
|
||||
await expect(readZipArchive(archive)).resolves.toEqual({
|
||||
rootPath: "paperclip-demo",
|
||||
files: {
|
||||
"COMPANY.md": "---\nname: Demo\n---\n",
|
||||
"reports/weekly.md": deflated,
|
||||
"blobs/4f2d1c9a": {
|
||||
encoding: "base64",
|
||||
data: Buffer.from(blobBytes).toString("base64"),
|
||||
contentType: "application/octet-stream",
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("throws on a truncated archive so a partial upload fails closed", async () => {
|
||||
const archive = buildZip(
|
||||
[{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") }],
|
||||
"paperclip-demo",
|
||||
);
|
||||
// Chop the tail so a declared entry body runs past the end of the buffer.
|
||||
const truncated = archive.slice(0, 40);
|
||||
await expect(readZipArchive(truncated)).rejects.toThrow(/truncated|Invalid zip/i);
|
||||
});
|
||||
|
||||
it("rejects data-descriptor entries the writer never emits", async () => {
|
||||
const archive = buildZip(
|
||||
[{ path: "COMPANY.md", bytes: new TextEncoder().encode("hi") }],
|
||||
"paperclip-demo",
|
||||
);
|
||||
// Flip bit 0x0008 in the local header's general-purpose flag (offset 6).
|
||||
archive[6] = archive[6]! | 0x08;
|
||||
await expect(readZipArchive(archive)).rejects.toThrow(/data descriptors/i);
|
||||
});
|
||||
|
||||
// Locate the first central-directory file header so a test can lop off the
|
||||
// whole directory + EOCD, leaving only intact local entries.
|
||||
function centralDirectoryOffset(archive: Uint8Array): number {
|
||||
for (let i = 0; i + 4 <= archive.length; i += 1) {
|
||||
if (archive[i] === 0x50 && archive[i + 1] === 0x4b && archive[i + 2] === 0x01 && archive[i + 3] === 0x02) {
|
||||
return i;
|
||||
}
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
it("rejects an archive truncated before the central directory", async () => {
|
||||
const archive = buildZip(
|
||||
[
|
||||
{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") },
|
||||
{ path: "agents/ceo/AGENTS.md", bytes: new TextEncoder().encode("---\nname: CEO\n---\n") },
|
||||
],
|
||||
"paperclip-demo",
|
||||
);
|
||||
// Keep every local entry intact but drop the directory + EOCD, mimicking an
|
||||
// upload cut at a record boundary. The reader must not import the fragment.
|
||||
const withoutDirectory = archive.slice(0, centralDirectoryOffset(archive));
|
||||
await expect(readZipArchive(withoutDirectory)).rejects.toThrow(/truncated before the central directory/i);
|
||||
});
|
||||
|
||||
it("rejects an archive whose end-of-central-directory record is missing", async () => {
|
||||
const archive = buildZip(
|
||||
[{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") }],
|
||||
"paperclip-demo",
|
||||
);
|
||||
// The central directory survives but the trailing 22-byte EOCD is gone.
|
||||
await expect(readZipArchive(archive.slice(0, archive.length - 22))).rejects.toThrow(
|
||||
/end-of-central-directory/i,
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects an archive whose central directory count does not match the entries read", async () => {
|
||||
const archive = buildZip(
|
||||
[{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") }],
|
||||
"paperclip-demo",
|
||||
);
|
||||
// Overstate the total-entries field (EOCD offset +10 → last 12 bytes in) so a
|
||||
// silently-dropped central-directory record is caught.
|
||||
archive[archive.length - 12] = 5;
|
||||
archive[archive.length - 11] = 0;
|
||||
await expect(readZipArchive(archive)).rejects.toThrow(/central directory declares 5 entries/i);
|
||||
});
|
||||
|
||||
it("rejects a truncated archive re-terminated with a forged EOCD that matches the surviving entries", async () => {
|
||||
// The exact silent-partial-import this reader guards against: an archive is
|
||||
// cut after a complete leading local entry (losing the real central directory
|
||||
// and EOCD), then re-terminated with a hand-forged 22-byte EOCD whose entry
|
||||
// count matches the surviving local entry. An entry-count-only check would
|
||||
// wave it through; validating the central directory it points at must not.
|
||||
const archive = buildZip(
|
||||
[{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") }],
|
||||
"paperclip-demo",
|
||||
);
|
||||
// Keep only the intact local entry (everything before the first central record).
|
||||
const localOnly = archive.slice(0, centralDirectoryOffset(archive));
|
||||
|
||||
const forgedEocd = Buffer.alloc(22);
|
||||
forgedEocd.writeUInt32LE(0x06054b50, 0);
|
||||
forgedEocd.writeUInt16LE(1, 8); // entries on this disk
|
||||
forgedEocd.writeUInt16LE(1, 10); // total entries — matches the one surviving local entry
|
||||
forgedEocd.writeUInt32LE(0, 12); // central directory size (forged)
|
||||
forgedEocd.writeUInt32LE(0, 16); // central directory offset (forged)
|
||||
const forged = new Uint8Array(Buffer.concat([Buffer.from(localOnly), forgedEocd]));
|
||||
|
||||
await expect(readZipArchive(forged)).rejects.toThrow(/central directory location is inconsistent/i);
|
||||
});
|
||||
|
||||
it("rejects a forged EOCD whose central directory offset points at non-directory bytes", async () => {
|
||||
const archive = buildZip(
|
||||
[{ path: "COMPANY.md", bytes: new TextEncoder().encode("---\nname: Demo\n---\n") }],
|
||||
"paperclip-demo",
|
||||
);
|
||||
const firstEntryOnly = archive.slice(0, centralDirectoryOffset(archive));
|
||||
// A forged EOCD whose offset+size abut the record (passing the location
|
||||
// check) but point into the local entry, which carries no directory signature.
|
||||
const forgedEocd = Buffer.alloc(22);
|
||||
forgedEocd.writeUInt32LE(0x06054b50, 0);
|
||||
forgedEocd.writeUInt16LE(1, 8);
|
||||
forgedEocd.writeUInt16LE(1, 10);
|
||||
forgedEocd.writeUInt32LE(46, 12); // size
|
||||
forgedEocd.writeUInt32LE(firstEntryOnly.length - 46, 16); // start = eocdOffset - size
|
||||
const forged = new Uint8Array(Buffer.concat([Buffer.from(firstEntryOnly), forgedEocd]));
|
||||
|
||||
await expect(readZipArchive(forged)).rejects.toThrow(/malformed central directory record/i);
|
||||
});
|
||||
|
||||
it("rejects two entries that normalize to the same path instead of silently overwriting", async () => {
|
||||
const archive = buildZip(
|
||||
[
|
||||
{ path: "docs/x.md", bytes: new TextEncoder().encode("first") },
|
||||
{ path: "docs//x.md", bytes: new TextEncoder().encode("second") },
|
||||
],
|
||||
"paperclip-demo",
|
||||
);
|
||||
await expect(readZipArchive(archive)).rejects.toThrow(/duplicate entry path "docs\/x\.md"/i);
|
||||
});
|
||||
|
||||
it("bounds a highly compressible DEFLATE entry at the per-entry decompressed limit", async () => {
|
||||
// Compresses tiny but expands to 8 KiB; a 1 KiB cap must reject it before it
|
||||
// materializes. Real packages sit far under the 256 MB production default.
|
||||
const bomb = new TextEncoder().encode("a".repeat(8 * 1024));
|
||||
const archive = buildZip([{ path: "bomb.txt", bytes: bomb, method: 8 }], "paperclip-demo");
|
||||
await expect(
|
||||
readZipArchive(archive, { maxEntryDecompressedBytes: 1024, maxTotalDecompressedBytes: 1 << 30 }),
|
||||
).rejects.toThrow(/per-entry limit/i);
|
||||
});
|
||||
|
||||
it("bounds a stored entry at the per-entry decompressed limit", async () => {
|
||||
const stored = new TextEncoder().encode("b".repeat(4 * 1024));
|
||||
const archive = buildZip([{ path: "big.bin", bytes: stored, method: 0 }], "paperclip-demo");
|
||||
await expect(
|
||||
readZipArchive(archive, { maxEntryDecompressedBytes: 1024, maxTotalDecompressedBytes: 1 << 30 }),
|
||||
).rejects.toThrow(/per-entry limit/i);
|
||||
});
|
||||
|
||||
it("bounds the aggregate decompressed size across many entries", async () => {
|
||||
const chunk = new TextEncoder().encode("c".repeat(600));
|
||||
const archive = buildZip(
|
||||
[
|
||||
{ path: "a.txt", bytes: chunk, method: 0 },
|
||||
{ path: "b.txt", bytes: chunk, method: 0 },
|
||||
],
|
||||
"paperclip-demo",
|
||||
);
|
||||
// Each entry is under the per-entry cap, but together they cross the total.
|
||||
await expect(
|
||||
readZipArchive(archive, { maxEntryDecompressedBytes: 4096, maxTotalDecompressedBytes: 1000 }),
|
||||
).rejects.toThrow(/exceed the 1000-byte limit/i);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,308 @@
|
||||
import { inflateRawSync } from "node:zlib";
|
||||
import path from "node:path";
|
||||
import type { CompanyPortabilityFileEntry } from "./types/company-portability.js";
|
||||
|
||||
// Node-side reader for the company portability zip package. It produces the
|
||||
// exact `{ rootPath, files }` shape the inline import source carries, so the
|
||||
// server can accept a raw uploaded zip and feed the importer the same bundle
|
||||
// the browser used to expand and post as inline JSON. The browser has its own
|
||||
// reader in `ui/src/lib/zip.ts` (DecompressionStream); this one uses node zlib.
|
||||
// Both must stay byte-compatible with the writer in `ui/src/lib/zip.ts`.
|
||||
|
||||
const textDecoder = new TextDecoder();
|
||||
// ignoreBOM keeps a leading BOM in the decoded text so text entries
|
||||
// re-encode to their original bytes; fatal surfaces invalid UTF-8.
|
||||
const strictTextDecoder = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true });
|
||||
|
||||
// Decompression-bomb guards. The compressed upload is already capped upstream
|
||||
// (multer/express.raw at PORTABLE_ZIP_UPLOAD_LIMIT_BYTES), but DEFLATE lets a
|
||||
// small compressed archive expand to gigabytes. Bound the expansion so a
|
||||
// malicious highly-compressible package cannot exhaust server memory: a
|
||||
// per-entry ceiling (passed to zlib as maxOutputLength, so it fails before
|
||||
// over-allocating) plus an aggregate ceiling across all entries. Both sit far
|
||||
// above any real company package (inline JSON was historically capped at 64MB)
|
||||
// yet far below what a bomb would need.
|
||||
export const MAX_ZIP_ENTRY_DECOMPRESSED_BYTES = 256 * 1024 * 1024;
|
||||
export const MAX_ZIP_TOTAL_DECOMPRESSED_BYTES = 512 * 1024 * 1024;
|
||||
|
||||
export const binaryContentTypeByExtension: Record<string, string> = {
|
||||
".gif": "image/gif",
|
||||
".jpeg": "image/jpeg",
|
||||
".jpg": "image/jpeg",
|
||||
".png": "image/png",
|
||||
".svg": "image/svg+xml",
|
||||
".webp": "image/webp",
|
||||
};
|
||||
|
||||
function normalizeArchivePath(pathValue: string) {
|
||||
return pathValue
|
||||
.replace(/\\/g, "/")
|
||||
.split("/")
|
||||
.filter(Boolean)
|
||||
.join("/");
|
||||
}
|
||||
|
||||
function readUint16(source: Uint8Array, offset: number) {
|
||||
return source[offset]! | (source[offset + 1]! << 8);
|
||||
}
|
||||
|
||||
function readUint32(source: Uint8Array, offset: number) {
|
||||
return (
|
||||
source[offset]! |
|
||||
(source[offset + 1]! << 8) |
|
||||
(source[offset + 2]! << 16) |
|
||||
(source[offset + 3]! << 24)
|
||||
) >>> 0;
|
||||
}
|
||||
|
||||
const LOCAL_FILE_SIGNATURE = 0x04034b50;
|
||||
const CENTRAL_DIRECTORY_SIGNATURE = 0x02014b50;
|
||||
const EOCD_SIGNATURE = 0x06054b50;
|
||||
|
||||
// Locate the end-of-central-directory record by scanning back from the tail.
|
||||
// The record is 22 bytes plus an optional trailing comment (max 0xffff), so the
|
||||
// signature lives within the last 22 + 0xffff bytes; returns -1 when absent
|
||||
// (a truncated or non-zip upload), which the reader treats as fail-closed.
|
||||
function findEndOfCentralDirectoryOffset(bytes: Uint8Array): number {
|
||||
const minOffset = Math.max(0, bytes.length - (22 + 0xffff));
|
||||
for (let offset = bytes.length - 22; offset >= minOffset; offset -= 1) {
|
||||
if (readUint32(bytes, offset) === EOCD_SIGNATURE) return offset;
|
||||
}
|
||||
return -1;
|
||||
}
|
||||
|
||||
// Fully validate the central directory the EOCD advertises against the local
|
||||
// entries actually read from the archive body. Trusting only the EOCD's entry
|
||||
// count is not enough: a truncated archive with a forged 22-byte EOCD whose
|
||||
// count matches the surviving local entries would otherwise be accepted, and the
|
||||
// importer would silently process a partial company package (dropping agents,
|
||||
// issues, and so on). This walks the real central directory and requires that:
|
||||
// • it is fully present and sits immediately before the EOCD (no gap, no
|
||||
// pointer past the buffer — a truncated tail fails here);
|
||||
// • every record carries the central-directory signature and its lengths sum
|
||||
// to exactly the declared directory size; and
|
||||
// • every record references a real local file header, and the record count
|
||||
// equals both the EOCD's declared count and the local entries parsed.
|
||||
function validateCentralDirectory(bytes: Uint8Array, eocdOffset: number, localHeaderCount: number) {
|
||||
const declaredEntryCount = readUint16(bytes, eocdOffset + 10);
|
||||
const centralDirectorySize = readUint32(bytes, eocdOffset + 12);
|
||||
const centralDirectoryStart = readUint32(bytes, eocdOffset + 16);
|
||||
if (centralDirectoryStart > eocdOffset || centralDirectoryStart + centralDirectorySize !== eocdOffset) {
|
||||
throw new Error(
|
||||
"Invalid zip archive: central directory location is inconsistent (truncated or forged).",
|
||||
);
|
||||
}
|
||||
|
||||
const directoryEnd = centralDirectoryStart + centralDirectorySize;
|
||||
let cursor = centralDirectoryStart;
|
||||
let recordCount = 0;
|
||||
while (cursor < directoryEnd) {
|
||||
if (cursor + 46 > directoryEnd || readUint32(bytes, cursor) !== CENTRAL_DIRECTORY_SIGNATURE) {
|
||||
throw new Error("Invalid zip archive: malformed central directory record.");
|
||||
}
|
||||
const fileNameLength = readUint16(bytes, cursor + 28);
|
||||
const extraFieldLength = readUint16(bytes, cursor + 30);
|
||||
const commentLength = readUint16(bytes, cursor + 32);
|
||||
const localHeaderOffset = readUint32(bytes, cursor + 42);
|
||||
if (localHeaderOffset + 4 > bytes.length || readUint32(bytes, localHeaderOffset) !== LOCAL_FILE_SIGNATURE) {
|
||||
throw new Error("Invalid zip archive: central directory references a missing local entry.");
|
||||
}
|
||||
cursor += 46 + fileNameLength + extraFieldLength + commentLength;
|
||||
recordCount += 1;
|
||||
}
|
||||
|
||||
if (cursor !== directoryEnd) {
|
||||
throw new Error("Invalid zip archive: central directory size does not match its records.");
|
||||
}
|
||||
if (recordCount !== declaredEntryCount || recordCount !== localHeaderCount) {
|
||||
throw new Error(
|
||||
`Invalid zip archive: central directory declares ${declaredEntryCount} entries but ${localHeaderCount} local entries were read (truncated or corrupt).`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function sharedArchiveRoot(paths: string[]) {
|
||||
if (paths.length === 0) return null;
|
||||
const firstSegments = paths
|
||||
.map((entry) => normalizeArchivePath(entry).split("/").filter(Boolean))
|
||||
.filter((parts) => parts.length > 0);
|
||||
if (firstSegments.length === 0) return null;
|
||||
const candidate = firstSegments[0]![0]!;
|
||||
return firstSegments.every((parts) => parts.length > 1 && parts[0] === candidate)
|
||||
? candidate
|
||||
: null;
|
||||
}
|
||||
|
||||
export function isBlobStorePath(pathValue: string) {
|
||||
return /(^|\/)blobs\/[^/]+$/.test(normalizeArchivePath(pathValue));
|
||||
}
|
||||
|
||||
function decodeStrictUtf8(bytes: Uint8Array): string | null {
|
||||
let text: string;
|
||||
try {
|
||||
text = strictTextDecoder.decode(bytes);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
return Buffer.from(text, "utf8").equals(Buffer.from(bytes)) ? text : null;
|
||||
}
|
||||
|
||||
export function bytesToPortableFileEntry(pathValue: string, bytes: Uint8Array): CompanyPortabilityFileEntry {
|
||||
// Content-addressed blob entries are opaque bytes regardless of extension.
|
||||
if (isBlobStorePath(pathValue)) {
|
||||
return { encoding: "base64", data: Buffer.from(bytes).toString("base64"), contentType: "application/octet-stream" };
|
||||
}
|
||||
const contentType = binaryContentTypeByExtension[path.extname(pathValue).toLowerCase()];
|
||||
if (contentType) {
|
||||
return { encoding: "base64", data: Buffer.from(bytes).toString("base64"), contentType };
|
||||
}
|
||||
const text = decodeStrictUtf8(bytes);
|
||||
if (text !== null) return text;
|
||||
// Bytes that are not valid UTF-8 must not be decoded lossily; fall back
|
||||
// to base64 so they round-trip exactly.
|
||||
return { encoding: "base64", data: Buffer.from(bytes).toString("base64"), contentType: "application/octet-stream" };
|
||||
}
|
||||
|
||||
// Size caps applied while expanding an archive. Callers use the module defaults;
|
||||
// the limits are parameterizable so the guard can be exercised in tests without
|
||||
// allocating hundreds of megabytes.
|
||||
export interface ReadZipArchiveLimits {
|
||||
maxEntryDecompressedBytes: number;
|
||||
maxTotalDecompressedBytes: number;
|
||||
}
|
||||
|
||||
const DEFAULT_ZIP_LIMITS: ReadZipArchiveLimits = {
|
||||
maxEntryDecompressedBytes: MAX_ZIP_ENTRY_DECOMPRESSED_BYTES,
|
||||
maxTotalDecompressedBytes: MAX_ZIP_TOTAL_DECOMPRESSED_BYTES,
|
||||
};
|
||||
|
||||
function inflateZipEntry(compressionMethod: number, bytes: Uint8Array, maxEntryDecompressedBytes: number) {
|
||||
if (compressionMethod === 0) {
|
||||
if (bytes.length > maxEntryDecompressedBytes) {
|
||||
throw new Error(
|
||||
`Unsupported zip archive: a stored entry exceeds the ${maxEntryDecompressedBytes}-byte per-entry limit.`,
|
||||
);
|
||||
}
|
||||
return bytes;
|
||||
}
|
||||
if (compressionMethod !== 8) {
|
||||
throw new Error("Unsupported zip archive: only STORE and DEFLATE entries are supported.");
|
||||
}
|
||||
try {
|
||||
// maxOutputLength makes zlib throw (ERR_BUFFER_TOO_LARGE) before it allocates
|
||||
// past the per-entry ceiling, so a bomb entry never materializes in memory.
|
||||
return new Uint8Array(inflateRawSync(bytes, { maxOutputLength: maxEntryDecompressedBytes }));
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException | undefined)?.code === "ERR_BUFFER_TOO_LARGE") {
|
||||
throw new Error(
|
||||
`Unsupported zip archive: a compressed entry expands beyond the ${maxEntryDecompressedBytes}-byte per-entry limit.`,
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
export async function readZipArchive(
|
||||
source: ArrayBuffer | Uint8Array,
|
||||
limits: ReadZipArchiveLimits = DEFAULT_ZIP_LIMITS,
|
||||
): Promise<{
|
||||
rootPath: string | null;
|
||||
files: Record<string, CompanyPortabilityFileEntry>;
|
||||
}> {
|
||||
const { maxEntryDecompressedBytes, maxTotalDecompressedBytes } = limits;
|
||||
const bytes = source instanceof Uint8Array ? source : new Uint8Array(source);
|
||||
const entries: Array<{ path: string; body: CompanyPortabilityFileEntry }> = [];
|
||||
let offset = 0;
|
||||
// Count every local file header (including directory entries) so the tally can
|
||||
// be reconciled against the central directory below; guard total expansion so
|
||||
// a bomb split across many entries is still bounded.
|
||||
let localHeaderCount = 0;
|
||||
let totalDecompressedBytes = 0;
|
||||
let reachedCentralDirectory = false;
|
||||
|
||||
while (offset + 4 <= bytes.length) {
|
||||
const signature = readUint32(bytes, offset);
|
||||
if (signature === CENTRAL_DIRECTORY_SIGNATURE || signature === EOCD_SIGNATURE) {
|
||||
reachedCentralDirectory = true;
|
||||
break;
|
||||
}
|
||||
if (signature !== LOCAL_FILE_SIGNATURE) {
|
||||
throw new Error("Invalid zip archive: unsupported local file header.");
|
||||
}
|
||||
|
||||
if (offset + 30 > bytes.length) {
|
||||
throw new Error("Invalid zip archive: truncated local file header.");
|
||||
}
|
||||
|
||||
const generalPurposeFlag = readUint16(bytes, offset + 6);
|
||||
const compressionMethod = readUint16(bytes, offset + 8);
|
||||
const compressedSize = readUint32(bytes, offset + 18);
|
||||
const fileNameLength = readUint16(bytes, offset + 26);
|
||||
const extraFieldLength = readUint16(bytes, offset + 28);
|
||||
|
||||
if ((generalPurposeFlag & 0x0008) !== 0) {
|
||||
throw new Error("Unsupported zip archive: data descriptors are not supported.");
|
||||
}
|
||||
|
||||
const nameOffset = offset + 30;
|
||||
const bodyOffset = nameOffset + fileNameLength + extraFieldLength;
|
||||
const bodyEnd = bodyOffset + compressedSize;
|
||||
if (bodyEnd > bytes.length) {
|
||||
throw new Error("Invalid zip archive: truncated file contents.");
|
||||
}
|
||||
|
||||
localHeaderCount += 1;
|
||||
const rawArchivePath = textDecoder.decode(bytes.slice(nameOffset, nameOffset + fileNameLength));
|
||||
const archivePath = normalizeArchivePath(rawArchivePath);
|
||||
const isDirectoryEntry = /\/$/.test(rawArchivePath.replace(/\\/g, "/"));
|
||||
if (archivePath && !isDirectoryEntry) {
|
||||
const entryBytes = inflateZipEntry(compressionMethod, bytes.slice(bodyOffset, bodyEnd), maxEntryDecompressedBytes);
|
||||
totalDecompressedBytes += entryBytes.length;
|
||||
if (totalDecompressedBytes > maxTotalDecompressedBytes) {
|
||||
throw new Error(
|
||||
`Unsupported zip archive: decompressed contents exceed the ${maxTotalDecompressedBytes}-byte limit.`,
|
||||
);
|
||||
}
|
||||
entries.push({
|
||||
path: archivePath,
|
||||
body: bytesToPortableFileEntry(archivePath, entryBytes),
|
||||
});
|
||||
}
|
||||
|
||||
offset = bodyEnd;
|
||||
}
|
||||
|
||||
// A complete archive always ends with a central directory after its local
|
||||
// entries. If the scan ran off the end of the buffer without reaching one, the
|
||||
// upload was truncated at a record boundary — fail closed rather than import a
|
||||
// leading fragment. Then fully validate the central directory the EOCD points
|
||||
// at so a truncated tail with a forged EOCD (whose count happens to match the
|
||||
// surviving entries) cannot smuggle in a partial import.
|
||||
if (!reachedCentralDirectory) {
|
||||
throw new Error("Invalid zip archive: truncated before the central directory.");
|
||||
}
|
||||
const eocdOffset = findEndOfCentralDirectoryOffset(bytes);
|
||||
if (eocdOffset === -1) {
|
||||
throw new Error("Invalid zip archive: missing end-of-central-directory record.");
|
||||
}
|
||||
validateCentralDirectory(bytes, eocdOffset, localHeaderCount);
|
||||
|
||||
const rootPath = sharedArchiveRoot(entries.map((entry) => entry.path));
|
||||
const files: Record<string, CompanyPortabilityFileEntry> = {};
|
||||
for (const entry of entries) {
|
||||
const normalizedPath =
|
||||
rootPath && entry.path.startsWith(`${rootPath}/`)
|
||||
? entry.path.slice(rootPath.length + 1)
|
||||
: entry.path;
|
||||
if (!normalizedPath) continue;
|
||||
// Two entries that normalize to the same path (e.g. `a/b` and `a//b`) make
|
||||
// the package ambiguous; reject it rather than silently letting the later
|
||||
// entry's contents win over the earlier one.
|
||||
if (Object.prototype.hasOwnProperty.call(files, normalizedPath)) {
|
||||
throw new Error(`Invalid zip archive: duplicate entry path "${normalizedPath}".`);
|
||||
}
|
||||
files[normalizedPath] = entry.body;
|
||||
}
|
||||
|
||||
return { rootPath, files };
|
||||
}
|
||||
@@ -55,6 +55,68 @@ If a dimension is privacy-protected before emission, emit only the protected
|
||||
value and its matching public marker as defined by the typed helper or generated
|
||||
contract. Do not emit private source material in telemetry dimensions.
|
||||
|
||||
## Sandbox Startup Trace Spans
|
||||
|
||||
Paperclip opens OpenTelemetry spans on the sandbox start path. These spans are a
|
||||
separate telemetry surface from the first-party events above. The generated
|
||||
telemetry contract does not cover them, so this section is their canonical
|
||||
contract.
|
||||
|
||||
The spans are opt-in. Paperclip exports them only when an OTLP endpoint is
|
||||
configured. With no endpoint the whole span path is a no-op. Paperclip opens the
|
||||
spans only for a run that targets a remote sandbox. A local run and an SSH run
|
||||
stay out of these spans.
|
||||
|
||||
Span attributes use a closed allowlist. A command, an argument, an environment
|
||||
value, a file path, an identifier, or program output never rides a span. It rides
|
||||
neither as an attribute nor as an event. Each numeric attribute is finite.
|
||||
Paperclip omits an attribute when its value is absent.
|
||||
|
||||
### Spans
|
||||
|
||||
| Span | Scope | Parent |
|
||||
| --- | --- | --- |
|
||||
| `sandbox.startup` | The one root span for a sandbox bring-up. | none (root) |
|
||||
| `workspace.resolve` | Workspace resolution step. | `sandbox.startup` |
|
||||
| `codex-home.seed` | Managed-home seed step. | `sandbox.startup` |
|
||||
| `skills.reconcile` | Skills reconcile step. | `sandbox.startup` |
|
||||
| `stage.sync` | Workspace stage-sync step. | `sandbox.startup` |
|
||||
| `bridge.paperclip` | Paperclip bridge start step. | `sandbox.startup` |
|
||||
| `bridge.process-session` | Process-session bridge start step. | `sandbox.startup` |
|
||||
| `acp.handshake` | ACP session handshake step. | `sandbox.startup` |
|
||||
| `provider.execute` | One host-to-sandbox provider exec call. | none |
|
||||
|
||||
The root span sets the error status when the bring-up fails. Each step span sets
|
||||
the error status when its step fails.
|
||||
|
||||
### Startup span attributes
|
||||
|
||||
The bring-up step spans use this closed attribute allowlist.
|
||||
|
||||
| Attribute | Type | Optional | Meaning |
|
||||
| --- | --- | --- | --- |
|
||||
| `step` | string | no | The bring-up step name. |
|
||||
| `provider` | string | yes | The normalized provider family. |
|
||||
| `roundTrips` | number | yes | Host-to-sandbox round trips for the step. |
|
||||
| `providerExecMs` | number | yes | Provider-reported exec time for the step, in milliseconds. |
|
||||
| `providerGetMs` | number | yes | Provider-reported fetch time for the step, in milliseconds. |
|
||||
|
||||
### Provider exec span attributes
|
||||
|
||||
The `provider.execute` span uses this closed attribute allowlist. Paperclip omits
|
||||
a duration attribute when the provider does not report the value.
|
||||
|
||||
| Attribute | Type | Optional | Meaning |
|
||||
| --- | --- | --- | --- |
|
||||
| `provider` | string | no | The normalized provider family. |
|
||||
| `exit` | string | no | `ok` when the exit code is 0, else `error`. |
|
||||
| `provider.exec.duration_ms` | number | yes | Provider-reported exec wall time, in milliseconds. |
|
||||
| `provider.get.duration_ms` | number | yes | Provider-reported fetch wall time, in milliseconds. |
|
||||
|
||||
To add a startup span attribute or a provider span attribute, extend the
|
||||
allowlist in the code first. Keep the attribute low-cardinality and free of user
|
||||
content.
|
||||
|
||||
## Dimension Values
|
||||
|
||||
Telemetry dimension values must be primitives. Use only the value types allowed
|
||||
|
||||
@@ -1,110 +0,0 @@
|
||||
export type CloudUpstreamStep = "connect" | "scan" | "preview" | "push" | "verify" | "activate";
|
||||
|
||||
export type CloudUpstreamRunStatus = "previewed" | "running" | "succeeded" | "failed" | "cancelled";
|
||||
|
||||
export type CloudUpstreamActivationEntityType = "agents" | "routines" | "monitors";
|
||||
|
||||
export interface CloudUpstreamActivationDecision {
|
||||
entityType: CloudUpstreamActivationEntityType;
|
||||
count: number;
|
||||
status: "paused" | "activated";
|
||||
activatedAt: string | null;
|
||||
}
|
||||
|
||||
export interface CloudUpstreamTarget {
|
||||
stackId: string;
|
||||
stackSlug: string | null;
|
||||
stackDisplayName: string | null;
|
||||
companyId: string;
|
||||
primaryHost: string;
|
||||
origin: string;
|
||||
product: string;
|
||||
schemaMajor: number;
|
||||
maxChunkBytes: number;
|
||||
}
|
||||
|
||||
export interface CloudUpstreamConnection {
|
||||
id: string;
|
||||
companyId: string;
|
||||
remoteUrl: string;
|
||||
target: CloudUpstreamTarget;
|
||||
tokenStatus: "pending" | "connected" | "expired" | "revoked";
|
||||
scopes: string[];
|
||||
authorizedGlobalUserId: string | null;
|
||||
expiresAt: string | null;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
lastRunId: string | null;
|
||||
}
|
||||
|
||||
export interface CloudUpstreamSummaryCount {
|
||||
key: string;
|
||||
label: string;
|
||||
count: number;
|
||||
}
|
||||
|
||||
export interface CloudUpstreamWarning {
|
||||
code: string;
|
||||
severity: "warning" | "blocker";
|
||||
title: string;
|
||||
detail: string;
|
||||
}
|
||||
|
||||
export interface CloudUpstreamConflict {
|
||||
id: string;
|
||||
entityType: string;
|
||||
sourceLabel: string;
|
||||
targetLabel: string;
|
||||
plannedAction: "create" | "update" | "skip" | "blocked";
|
||||
reason: string;
|
||||
}
|
||||
|
||||
export interface CloudUpstreamPreview {
|
||||
connectionId: string;
|
||||
sourceCompanyId: string;
|
||||
target: CloudUpstreamTarget;
|
||||
schemaCompatible: boolean;
|
||||
summary: CloudUpstreamSummaryCount[];
|
||||
warnings: CloudUpstreamWarning[];
|
||||
conflicts: CloudUpstreamConflict[];
|
||||
generatedAt: string;
|
||||
}
|
||||
|
||||
export interface CloudUpstreamRunEvent {
|
||||
id: string;
|
||||
at: string;
|
||||
phase: CloudUpstreamStep;
|
||||
type: "created" | "updated" | "skipped" | "conflict" | "retrying" | "failed" | "completed";
|
||||
message: string;
|
||||
}
|
||||
|
||||
export interface CloudUpstreamRun {
|
||||
id: string;
|
||||
connectionId: string;
|
||||
companyId: string;
|
||||
status: CloudUpstreamRunStatus;
|
||||
activeStep: CloudUpstreamStep;
|
||||
progressPercent: number;
|
||||
dryRun: boolean;
|
||||
summary: CloudUpstreamSummaryCount[];
|
||||
warnings: CloudUpstreamWarning[];
|
||||
conflicts: CloudUpstreamConflict[];
|
||||
events: CloudUpstreamRunEvent[];
|
||||
targetUrl: string | null;
|
||||
report: Record<string, unknown>;
|
||||
retryOfRunId: string | null;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
completedAt: string | null;
|
||||
}
|
||||
|
||||
export interface CloudUpstreamsState {
|
||||
connections: CloudUpstreamConnection[];
|
||||
runs: CloudUpstreamRun[];
|
||||
}
|
||||
|
||||
export interface CloudUpstreamConnectStartResponse {
|
||||
pendingConnectionId: string;
|
||||
authorizationUrl: string;
|
||||
connection: CloudUpstreamConnection;
|
||||
}
|
||||
@@ -49,6 +49,39 @@ export interface CompanyPortabilitySidebarOrder {
|
||||
projects: string[];
|
||||
}
|
||||
|
||||
export interface CompanyPortabilityLabelManifestEntry {
|
||||
name: string;
|
||||
color: string;
|
||||
}
|
||||
|
||||
export interface CompanyPortabilityBlobManifestEntry {
|
||||
sha256: string;
|
||||
byteSize: number;
|
||||
contentType: string;
|
||||
}
|
||||
|
||||
export interface CompanyPortabilityEmbeddedAssetManifestEntry {
|
||||
/**
|
||||
* The asset id on the SOURCE board. Source ids are only meaningful as the
|
||||
* rewrite key for the /api/assets/<assetId>/content references embedded in
|
||||
* the bundle's markdown; the importer mints fresh asset ids and rewrites
|
||||
* every reference to them.
|
||||
*/
|
||||
assetId: string;
|
||||
/** Content address of the image bytes in the bundle's blobs/ store. */
|
||||
sha256: string;
|
||||
contentType: string;
|
||||
originalFilename: string | null;
|
||||
/**
|
||||
* Export categories whose files reference this asset ("agents",
|
||||
* "projects", "skills", "tasks", "routines"), or "always" when an
|
||||
* always-exported root file references it. Lets export selection follow
|
||||
* the toggles of the referencing files. Absent entries are treated as
|
||||
* always included.
|
||||
*/
|
||||
ownedBy?: string[];
|
||||
}
|
||||
|
||||
export interface CompanyPortabilityProjectManifestEntry {
|
||||
slug: string;
|
||||
name: string;
|
||||
@@ -107,6 +140,41 @@ export interface CompanyPortabilityIssueCommentManifestEntry {
|
||||
createdAt: string | null;
|
||||
}
|
||||
|
||||
export interface CompanyPortabilityIssueDocumentManifestEntry {
|
||||
key: string;
|
||||
title: string | null;
|
||||
format: string;
|
||||
path: string;
|
||||
}
|
||||
|
||||
export interface CompanyPortabilityIssueWorkProductManifestEntry {
|
||||
type: string;
|
||||
provider: string;
|
||||
externalId: string | null;
|
||||
title: string;
|
||||
url: string | null;
|
||||
status: string;
|
||||
reviewState: string;
|
||||
isPrimary: boolean;
|
||||
healthStatus: string;
|
||||
summary: string | null;
|
||||
metadata: Record<string, unknown> | null;
|
||||
}
|
||||
|
||||
export interface CompanyPortabilityIssueMonitorManifestEntry {
|
||||
notes: string | null;
|
||||
scheduledBy: string | null;
|
||||
hadSchedule: boolean;
|
||||
}
|
||||
|
||||
export interface CompanyPortabilityIssueAttachmentManifestEntry {
|
||||
sha256: string;
|
||||
contentType: string;
|
||||
originalFilename: string | null;
|
||||
byteSize: number;
|
||||
commentIndex: number | null;
|
||||
}
|
||||
|
||||
export interface CompanyPortabilityIssueManifestEntry {
|
||||
slug: string;
|
||||
identifier: string | null;
|
||||
@@ -122,10 +190,16 @@ export interface CompanyPortabilityIssueManifestEntry {
|
||||
status: string | null;
|
||||
priority: string | null;
|
||||
labelIds: string[];
|
||||
labelNames?: string[];
|
||||
billingCode: string | null;
|
||||
executionWorkspaceSettings: Record<string, unknown> | null;
|
||||
assigneeAdapterOverrides: Record<string, unknown> | null;
|
||||
comments: CompanyPortabilityIssueCommentManifestEntry[];
|
||||
blockedBy?: string[];
|
||||
documents?: CompanyPortabilityIssueDocumentManifestEntry[];
|
||||
workProducts?: CompanyPortabilityIssueWorkProductManifestEntry[];
|
||||
monitor?: CompanyPortabilityIssueMonitorManifestEntry | null;
|
||||
attachments?: CompanyPortabilityIssueAttachmentManifestEntry[];
|
||||
metadata: Record<string, unknown> | null;
|
||||
}
|
||||
|
||||
@@ -181,6 +255,9 @@ export interface CompanyPortabilityManifest {
|
||||
includes: CompanyPortabilityInclude;
|
||||
company: CompanyPortabilityCompanyManifestEntry | null;
|
||||
sidebar: CompanyPortabilitySidebarOrder | null;
|
||||
labels?: CompanyPortabilityLabelManifestEntry[];
|
||||
blobs?: CompanyPortabilityBlobManifestEntry[];
|
||||
embeddedAssets?: CompanyPortabilityEmbeddedAssetManifestEntry[];
|
||||
agents: CompanyPortabilityAgentManifestEntry[];
|
||||
skills: CompanyPortabilitySkillManifestEntry[];
|
||||
projects: CompanyPortabilityProjectManifestEntry[];
|
||||
@@ -222,6 +299,13 @@ export type CompanyPortabilitySource =
|
||||
type: "inline";
|
||||
rootPath?: string | null;
|
||||
files: Record<string, CompanyPortabilityFileEntry>;
|
||||
/**
|
||||
* Client-declared file count (`Object.keys(files).length`). The import
|
||||
* path fails closed when the received file set is smaller, so a
|
||||
* truncated inline body cannot import a fragment. Optional for callers
|
||||
* that predate the check.
|
||||
*/
|
||||
expectedFileCount?: number;
|
||||
}
|
||||
| {
|
||||
type: "github";
|
||||
@@ -302,6 +386,7 @@ export interface CompanyPortabilityAdapterOverride {
|
||||
export interface CompanyPortabilityImportRequest extends CompanyPortabilityPreviewRequest {
|
||||
adapterOverrides?: Record<string, CompanyPortabilityAdapterOverride>;
|
||||
secretValues?: Record<string, string>;
|
||||
pauseAutomations?: boolean;
|
||||
}
|
||||
|
||||
export interface CompanyPortabilityImportResult {
|
||||
@@ -324,6 +409,13 @@ export interface CompanyPortabilityImportResult {
|
||||
name: string;
|
||||
reason: string | null;
|
||||
}[];
|
||||
routines: {
|
||||
slug: string;
|
||||
id: string | null;
|
||||
action: "created";
|
||||
title: string;
|
||||
status: string;
|
||||
}[];
|
||||
envInputs: CompanyPortabilityEnvInput[];
|
||||
warnings: string[];
|
||||
}
|
||||
|
||||
@@ -247,6 +247,19 @@ export interface CompanySkillForkPrecheckResult {
|
||||
existingForks: CompanySkillForkSummary[];
|
||||
}
|
||||
|
||||
export interface CompanySkillRenameRequest {
|
||||
name: string;
|
||||
slug?: string | null;
|
||||
}
|
||||
|
||||
export interface CompanySkillRenameResult {
|
||||
skill: CompanySkill;
|
||||
previousName: string;
|
||||
previousSlug: string;
|
||||
previousKey: string;
|
||||
reassignments: CompanySkillForkReassignment[];
|
||||
}
|
||||
|
||||
export interface CompanySkillUpdateRequest {
|
||||
description?: string | null;
|
||||
iconUrl?: string | null;
|
||||
|
||||
@@ -141,6 +141,8 @@ export type {
|
||||
CompanySkillForkReassignment,
|
||||
CompanySkillForkResult,
|
||||
CompanySkillForkPrecheckResult,
|
||||
CompanySkillRenameRequest,
|
||||
CompanySkillRenameResult,
|
||||
CompanySkillUpdateRequest,
|
||||
CompanySkillUpdateStatus,
|
||||
CompanySkillAuditSeverity,
|
||||
@@ -815,6 +817,9 @@ export type {
|
||||
CompanyPortabilityFileEntry,
|
||||
CompanyPortabilityCompanyManifestEntry,
|
||||
CompanyPortabilitySidebarOrder,
|
||||
CompanyPortabilityLabelManifestEntry,
|
||||
CompanyPortabilityBlobManifestEntry,
|
||||
CompanyPortabilityEmbeddedAssetManifestEntry,
|
||||
CompanyPortabilityAgentManifestEntry,
|
||||
CompanyPortabilitySkillManifestEntry,
|
||||
CompanyPortabilityProjectManifestEntry,
|
||||
@@ -822,6 +827,10 @@ export type {
|
||||
CompanyPortabilityIssueRoutineTriggerManifestEntry,
|
||||
CompanyPortabilityIssueRoutineManifestEntry,
|
||||
CompanyPortabilityIssueCommentManifestEntry,
|
||||
CompanyPortabilityIssueDocumentManifestEntry,
|
||||
CompanyPortabilityIssueWorkProductManifestEntry,
|
||||
CompanyPortabilityIssueMonitorManifestEntry,
|
||||
CompanyPortabilityIssueAttachmentManifestEntry,
|
||||
CompanyPortabilityIssueManifestEntry,
|
||||
CompanyPortabilityManifest,
|
||||
CompanyPortabilityExportResult,
|
||||
|
||||
@@ -57,7 +57,6 @@ export interface InstanceExperimentalSettings {
|
||||
enableWatchdogEverything: boolean;
|
||||
enableIssuePlanDecompositions: boolean;
|
||||
enableExperimentalFileViewer: boolean;
|
||||
enableCloudSync: boolean;
|
||||
enableExternalObjects: boolean;
|
||||
enableSmokeLab: boolean;
|
||||
enableBuiltInAgents: boolean;
|
||||
|
||||
@@ -330,6 +330,19 @@ export interface WorkspaceRealizationRequest {
|
||||
branchName: string | null;
|
||||
worktreePath: string | null;
|
||||
};
|
||||
/**
|
||||
* Read-only referenced (mentioned) project sources for this run, one per authorized additional
|
||||
* project. Additive and backward-compatible: it defaults to an empty array for legacy payloads
|
||||
* and for the anchor-only path. Additional sources are plain trees; they never get git-worktree
|
||||
* realization (that stays the anchor-only path).
|
||||
*/
|
||||
additionalSources?: Array<{
|
||||
localPath: string;
|
||||
projectId: string | null;
|
||||
projectWorkspaceId: string | null;
|
||||
repoUrl: string | null;
|
||||
repoRef: string | null;
|
||||
}>;
|
||||
runtimeOverlay: {
|
||||
provisionCommand: string | null;
|
||||
teardownCommand: string | null;
|
||||
@@ -360,6 +373,20 @@ export interface WorkspaceRealizationRecord {
|
||||
branchName: string | null;
|
||||
worktreePath: string | null;
|
||||
};
|
||||
/**
|
||||
* Realized read-only referenced (mentioned) project workspaces for this run, one per authorized
|
||||
* additional source in the request. This field carries each resolved path to the execution
|
||||
* target so the target can expose the referenced trees to the agent. It is additive and
|
||||
* backward-compatible: it defaults to an empty array for legacy records and for the anchor-only
|
||||
* path.
|
||||
*/
|
||||
additional?: Array<{
|
||||
path: string;
|
||||
projectId: string | null;
|
||||
projectWorkspaceId: string | null;
|
||||
repoUrl: string | null;
|
||||
repoRef: string | null;
|
||||
}>;
|
||||
remote: {
|
||||
path: string | null;
|
||||
host?: string | null;
|
||||
|
||||
@@ -57,6 +57,17 @@ export const portabilitySidebarOrderSchema = z.object({
|
||||
projects: z.array(z.string().min(1)).default([]),
|
||||
});
|
||||
|
||||
export const portabilityLabelManifestEntrySchema = z.object({
|
||||
name: z.string().min(1),
|
||||
color: z.string().min(1),
|
||||
});
|
||||
|
||||
export const portabilityBlobManifestEntrySchema = z.object({
|
||||
sha256: z.string().min(1),
|
||||
byteSize: z.number().int().nonnegative(),
|
||||
contentType: z.string().min(1),
|
||||
});
|
||||
|
||||
export const portabilityAgentManifestEntrySchema = z.object({
|
||||
slug: z.string().min(1),
|
||||
name: z.string().min(1),
|
||||
@@ -151,6 +162,41 @@ export const portabilityIssueCommentManifestEntrySchema = z.object({
|
||||
createdAt: z.string().datetime().nullable(),
|
||||
});
|
||||
|
||||
export const portabilityIssueDocumentManifestEntrySchema = z.object({
|
||||
key: z.string().min(1),
|
||||
title: z.string().nullable(),
|
||||
format: z.string().min(1),
|
||||
path: z.string().min(1),
|
||||
});
|
||||
|
||||
export const portabilityIssueWorkProductManifestEntrySchema = z.object({
|
||||
type: z.string().min(1),
|
||||
provider: z.string().min(1),
|
||||
externalId: z.string().nullable(),
|
||||
title: z.string().min(1),
|
||||
url: z.string().nullable(),
|
||||
status: z.string().min(1),
|
||||
reviewState: z.string().min(1),
|
||||
isPrimary: z.boolean().default(false),
|
||||
healthStatus: z.string().min(1),
|
||||
summary: z.string().nullable(),
|
||||
metadata: z.record(z.string(), z.unknown()).nullable(),
|
||||
});
|
||||
|
||||
export const portabilityIssueMonitorManifestEntrySchema = z.object({
|
||||
notes: z.string().nullable(),
|
||||
scheduledBy: z.string().nullable(),
|
||||
hadSchedule: z.boolean().default(false),
|
||||
});
|
||||
|
||||
export const portabilityIssueAttachmentManifestEntrySchema = z.object({
|
||||
sha256: z.string().min(1),
|
||||
contentType: z.string().min(1),
|
||||
originalFilename: z.string().nullable(),
|
||||
byteSize: z.number().int().nonnegative(),
|
||||
commentIndex: z.number().int().nonnegative().nullable().default(null),
|
||||
});
|
||||
|
||||
export const portabilityIssueManifestEntrySchema = z.object({
|
||||
slug: z.string().min(1),
|
||||
identifier: z.string().min(1).nullable(),
|
||||
@@ -166,10 +212,16 @@ export const portabilityIssueManifestEntrySchema = z.object({
|
||||
status: z.string().nullable(),
|
||||
priority: z.string().nullable(),
|
||||
labelIds: z.array(z.string().min(1)).default([]),
|
||||
labelNames: z.array(z.string().min(1)).default([]),
|
||||
billingCode: z.string().nullable(),
|
||||
executionWorkspaceSettings: z.record(z.string(), z.unknown()).nullable(),
|
||||
assigneeAdapterOverrides: z.record(z.string(), z.unknown()).nullable(),
|
||||
comments: z.array(portabilityIssueCommentManifestEntrySchema).default([]),
|
||||
blockedBy: z.array(z.string().min(1)).default([]),
|
||||
documents: z.array(portabilityIssueDocumentManifestEntrySchema).default([]),
|
||||
workProducts: z.array(portabilityIssueWorkProductManifestEntrySchema).default([]),
|
||||
monitor: portabilityIssueMonitorManifestEntrySchema.nullable().default(null),
|
||||
attachments: z.array(portabilityIssueAttachmentManifestEntrySchema).default([]),
|
||||
metadata: z.record(z.string(), z.unknown()).nullable(),
|
||||
});
|
||||
|
||||
@@ -191,6 +243,8 @@ export const portabilityManifestSchema = z.object({
|
||||
}),
|
||||
company: portabilityCompanyManifestEntrySchema.nullable(),
|
||||
sidebar: portabilitySidebarOrderSchema.nullable(),
|
||||
labels: z.array(portabilityLabelManifestEntrySchema).default([]),
|
||||
blobs: z.array(portabilityBlobManifestEntrySchema).default([]),
|
||||
agents: z.array(portabilityAgentManifestEntrySchema),
|
||||
skills: z.array(portabilitySkillManifestEntrySchema).default([]),
|
||||
projects: z.array(portabilityProjectManifestEntrySchema).default([]),
|
||||
@@ -203,6 +257,12 @@ export const portabilitySourceSchema = z.discriminatedUnion("type", [
|
||||
type: z.literal("inline"),
|
||||
rootPath: z.string().min(1).optional().nullable(),
|
||||
files: z.record(z.string(), portabilityFileEntrySchema),
|
||||
// Self-describing completeness count. The client sets this to
|
||||
// `Object.keys(files).length`; the import path rejects the payload when the
|
||||
// received file set is smaller, so a truncated or re-framed body fails
|
||||
// closed instead of importing a fragment. Optional for backwards
|
||||
// compatibility with callers that predate the check.
|
||||
expectedFileCount: z.number().int().nonnegative().optional(),
|
||||
}),
|
||||
z.object({
|
||||
type: z.literal("github"),
|
||||
@@ -262,6 +322,7 @@ export const portabilityAdapterOverrideSchema = z.object({
|
||||
export const companyPortabilityImportSchema = companyPortabilityPreviewSchema.extend({
|
||||
adapterOverrides: z.record(z.string().min(1), portabilityAdapterOverrideSchema).optional(),
|
||||
secretValues: z.record(z.string().min(1), z.string()).optional(),
|
||||
pauseAutomations: z.boolean().optional(),
|
||||
});
|
||||
|
||||
export type CompanyPortabilityImport = z.infer<typeof companyPortabilityImportSchema>;
|
||||
@@ -200,6 +200,19 @@ export const companySkillForkPrecheckResultSchema = z.object({
|
||||
existingForks: z.array(companySkillForkSummarySchema),
|
||||
});
|
||||
|
||||
export const companySkillRenameSchema = z.object({
|
||||
name: z.string().min(1).regex(/^[^\r\n]+$/, "Name must be a single line"),
|
||||
slug: z.string().min(1).nullable().optional(),
|
||||
});
|
||||
|
||||
export const companySkillRenameResultSchema = z.object({
|
||||
skill: companySkillSchema,
|
||||
previousName: z.string(),
|
||||
previousSlug: z.string(),
|
||||
previousKey: z.string(),
|
||||
reassignments: z.array(companySkillForkReassignmentSchema),
|
||||
});
|
||||
|
||||
export const companySkillUpdateSchema = z.object({
|
||||
description: z.string().nullable().optional(),
|
||||
iconUrl: z.string().nullable().optional(),
|
||||
@@ -569,6 +582,7 @@ export type CompanySkillVersionCreate = z.infer<typeof companySkillVersionCreate
|
||||
export type CompanySkillCommentCreate = z.infer<typeof companySkillCommentCreateSchema>;
|
||||
export type CompanySkillCommentUpdate = z.infer<typeof companySkillCommentUpdateSchema>;
|
||||
export type CompanySkillFork = z.infer<typeof companySkillForkSchema>;
|
||||
export type CompanySkillRename = z.infer<typeof companySkillRenameSchema>;
|
||||
export type CompanySkillUpdate = z.infer<typeof companySkillUpdateSchema>;
|
||||
export type CatalogSkillListQuery = z.infer<typeof catalogSkillListQuerySchema>;
|
||||
export type CompanySkillInstallCatalog = z.infer<typeof companySkillInstallCatalogSchema>;
|
||||
|
||||
@@ -156,6 +156,8 @@ export {
|
||||
companySkillForkReassignmentSchema,
|
||||
companySkillForkResultSchema,
|
||||
companySkillForkPrecheckResultSchema,
|
||||
companySkillRenameSchema,
|
||||
companySkillRenameResultSchema,
|
||||
companySkillUpdateSchema,
|
||||
companySkillUpdateStatusSchema,
|
||||
companySkillAuditFindingSchema,
|
||||
@@ -207,6 +209,7 @@ export {
|
||||
type CompanySkillCommentCreate,
|
||||
type CompanySkillCommentUpdate,
|
||||
type CompanySkillFork,
|
||||
type CompanySkillRename,
|
||||
type CatalogSkillListQuery,
|
||||
type CompanySkillInstallCatalog,
|
||||
type CompanySkillInstallUpdate,
|
||||
|
||||
@@ -51,7 +51,6 @@ export const instanceExperimentalSettingsSchema = z.object({
|
||||
enableWatchdogEverything: z.boolean().default(false),
|
||||
enableIssuePlanDecompositions: z.boolean().default(false),
|
||||
enableExperimentalFileViewer: z.boolean().default(false),
|
||||
enableCloudSync: z.boolean().default(false),
|
||||
enableExternalObjects: z.boolean().default(false),
|
||||
enableSmokeLab: z.boolean().default(false),
|
||||
enableBuiltInAgents: z.boolean().default(false),
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
name: simplified-english
|
||||
description: Write user-facing comments, plans, and documents in ASD-STE100 Simplified Technical English — short, unambiguous sentences with approved words and one meaning each — so readers understand them the first time.
|
||||
key: paperclipai/optional/content/simplified-english
|
||||
recommendedForRoles:
|
||||
- engineer
|
||||
- product
|
||||
- writer
|
||||
- devrel
|
||||
tags:
|
||||
- writing
|
||||
- communication
|
||||
- clarity
|
||||
- style
|
||||
---
|
||||
|
||||
# Simplified English
|
||||
|
||||
For user-facing comments, plans, and documents, write using only ASD-STE100 Simplified Technical English.
|
||||
|
||||
## Core rules
|
||||
|
||||
- Use short sentences (procedures ≤ 20 words, descriptions ≤ 25 words).
|
||||
- Give one instruction per sentence.
|
||||
- Use approved words with one meaning each; avoid synonyms and jargon.
|
||||
- Use the active voice and the present tense.
|
||||
- Use articles ("the", "a") and do not drop words to save space.
|
||||
- Write positive instructions; avoid negative or vague qualifiers.
|
||||
- Keep paragraphs to one topic.
|
||||
|
||||
## Approved words
|
||||
|
||||
The approved words are the ASD-STE100 controlled vocabulary — the Dictionary in the current ASD-STE100 specification, plus the technical names and technical verbs that your subject needs. When a word is not approved, use the simplest common word that has one meaning. Prefer these house choices:
|
||||
|
||||
- "start" / "stop" (not "initiate", "commence", "terminate", "kill")
|
||||
- "make" (not "implement", "leverage", "utilize")
|
||||
- "before" / "after" (not "prior to", "subsequent to")
|
||||
- "about" (not "regarding", "in relation to")
|
||||
- "help" (not "facilitate")
|
||||
- "use" (not "utilize", "employ")
|
||||
@@ -2,7 +2,7 @@
|
||||
"schemaVersion": 1,
|
||||
"packageName": "@paperclipai/skills-catalog",
|
||||
"packageVersion": "0.3.1",
|
||||
"generatedAt": "2026-07-24T18:19:49.696Z",
|
||||
"generatedAt": "2026-07-28T22:35:25.645Z",
|
||||
"skills": [
|
||||
{
|
||||
"id": "paperclipai:bundled:docs:doc-maintenance",
|
||||
@@ -485,6 +485,42 @@
|
||||
],
|
||||
"contentHash": "sha256:efe8ea89b552df95222609867c9c75f1b40e16f457d34e7e4a124f54daa33efa"
|
||||
},
|
||||
{
|
||||
"id": "paperclipai:optional:content:simplified-english",
|
||||
"key": "paperclipai/optional/content/simplified-english",
|
||||
"kind": "optional",
|
||||
"category": "content",
|
||||
"slug": "simplified-english",
|
||||
"name": "simplified-english",
|
||||
"description": "Write user-facing comments, plans, and documents in ASD-STE100 Simplified Technical English — short, unambiguous sentences with approved words and one meaning each — so readers understand them the first time.",
|
||||
"path": "catalog/optional/content/simplified-english",
|
||||
"entrypoint": "SKILL.md",
|
||||
"trustLevel": "markdown_only",
|
||||
"compatibility": "compatible",
|
||||
"defaultInstall": false,
|
||||
"recommendedForRoles": [
|
||||
"engineer",
|
||||
"product",
|
||||
"writer",
|
||||
"devrel"
|
||||
],
|
||||
"requires": [],
|
||||
"tags": [
|
||||
"writing",
|
||||
"communication",
|
||||
"clarity",
|
||||
"style"
|
||||
],
|
||||
"files": [
|
||||
{
|
||||
"path": "SKILL.md",
|
||||
"kind": "skill",
|
||||
"sizeBytes": 1578,
|
||||
"sha256": "a0f7385599873910e2fabf400346016a07dacbc7ece27c0e3616ae8fd94ebfbb"
|
||||
}
|
||||
],
|
||||
"contentHash": "sha256:642321f40d51e0125df05c6fbace49aeb07ec30392058756909c259c9c5eac6a"
|
||||
},
|
||||
{
|
||||
"id": "paperclipai:optional:finance:ramp",
|
||||
"key": "paperclipai/optional/finance/ramp",
|
||||
|
||||
@@ -20,6 +20,7 @@ const EXPECTED_BUNDLED_KEYS = [
|
||||
const EXPECTED_OPTIONAL_KEYS = [
|
||||
"paperclipai/optional/browser/agent-browser",
|
||||
"paperclipai/optional/content/release-announcement",
|
||||
"paperclipai/optional/content/simplified-english",
|
||||
"paperclipai/optional/finance/ramp",
|
||||
"paperclipai/optional/product/design-critique",
|
||||
"paperclipai/optional/research/last30days",
|
||||
|
||||
@@ -19,24 +19,30 @@ if [ "$(id -u)" -ne 0 ]; then
|
||||
fi
|
||||
|
||||
# Adjust the node user's UID/GID if they differ from the runtime request
|
||||
# and fix volume ownership only when a remap is needed
|
||||
changed=0
|
||||
|
||||
if [ "$(id -u node)" -ne "$PUID" ]; then
|
||||
echo "Updating node UID to $PUID"
|
||||
usermod -o -u "$PUID" node
|
||||
changed=1
|
||||
fi
|
||||
|
||||
if [ "$(id -g node)" -ne "$PGID" ]; then
|
||||
echo "Updating node GID to $PGID"
|
||||
groupmod -o -g "$PGID" node
|
||||
usermod -g "$PGID" node
|
||||
changed=1
|
||||
fi
|
||||
|
||||
if [ "$changed" = "1" ]; then
|
||||
chown -R node:node /paperclip
|
||||
# Ensure the app home is owned by the runtime user BEFORE dropping
|
||||
# privileges -- not only after a UID/GID remap. A freshly mounted volume
|
||||
# (Docker named volume, Railway volume, Kubernetes PV) arrives root-owned
|
||||
# and shadows the image's build-time chown, so with the default UID the old
|
||||
# remap-only condition dropped privileges onto an unwritable home and the
|
||||
# server crashed on its first mkdir. The probe is a first-mismatch find
|
||||
# over the WHOLE tree (uid and gid): a root-owned mount or descendant
|
||||
# (init containers, backup restores, files written before a remap) is
|
||||
# found immediately and repaired recursively, a GID-only remap is caught,
|
||||
# and a fully-correct tree costs one metadata-only walk with no chown.
|
||||
home_dir="${PAPERCLIP_HOME:-/paperclip}"
|
||||
if [ -d "$home_dir" ] && [ -n "$(find "$home_dir" \( ! -user node -o ! -group node \) -print -quit 2>/dev/null)" ]; then
|
||||
chown -R node:node "$home_dir"
|
||||
fi
|
||||
|
||||
exec gosu node "$@"
|
||||
@@ -1,8 +1,10 @@
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { instanceUserRoles } from "@paperclipai/db";
|
||||
import { companyMemberships, instanceUserRoles } from "@paperclipai/db";
|
||||
import { actorMiddleware } from "../middleware/auth.js";
|
||||
import { errorHandler } from "../middleware/error-handler.js";
|
||||
import { assertCompanyAccess } from "../routes/authz.js";
|
||||
|
||||
function createSelectChain(rows: unknown[]) {
|
||||
return {
|
||||
@@ -138,6 +140,83 @@ describe("actorMiddleware authenticated session profile", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("lets the cloud tenant actor through assertCompanyAccess for a company it holds a membership row in", async () => {
|
||||
process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = "tenant-token";
|
||||
// A company created on the instance after provisioning (e.g. by a company
|
||||
// import) — the user has a real membership row, but it is not the stack's
|
||||
// seeded primary company.
|
||||
const importedCompanyId = "33333333-3333-4333-8333-333333333333";
|
||||
const unrelatedCompanyId = "44444444-4444-4444-8444-444444444444";
|
||||
const insertChain = {
|
||||
values() {
|
||||
return insertChain;
|
||||
},
|
||||
onConflictDoUpdate() {
|
||||
return insertChain;
|
||||
},
|
||||
onConflictDoNothing() {
|
||||
return insertChain;
|
||||
},
|
||||
returning() {
|
||||
return Promise.resolve([{ companyId: "company-1", membershipRole: "member", status: "active" }]);
|
||||
},
|
||||
then(resolve: (value: unknown) => unknown) {
|
||||
return Promise.resolve(undefined).then(resolve);
|
||||
},
|
||||
};
|
||||
const db = {
|
||||
select: vi.fn(() => ({
|
||||
from: (table: unknown) => ({
|
||||
where: () =>
|
||||
Promise.resolve(
|
||||
table === companyMemberships
|
||||
? [{ companyId: importedCompanyId, membershipRole: "member", status: "active" }]
|
||||
: [],
|
||||
),
|
||||
}),
|
||||
})),
|
||||
insert: vi.fn(() => insertChain),
|
||||
delete: vi.fn(() => ({ where: () => Promise.resolve(undefined) })),
|
||||
} as any;
|
||||
const app = express();
|
||||
app.use(
|
||||
actorMiddleware(db, {
|
||||
deploymentMode: "authenticated",
|
||||
resolveSession: async () => null,
|
||||
}),
|
||||
);
|
||||
app.get("/companies/:companyId/resource", (req, res) => {
|
||||
assertCompanyAccess(req, req.params.companyId);
|
||||
res.json({ ok: true, companyIds: req.actor.companyIds });
|
||||
});
|
||||
app.post("/companies/:companyId/resource", (req, res) => {
|
||||
assertCompanyAccess(req, req.params.companyId);
|
||||
res.json({ ok: true });
|
||||
});
|
||||
app.use(errorHandler);
|
||||
|
||||
const cloudHeaders = {
|
||||
"x-paperclip-cloud-tenant-token": "tenant-token",
|
||||
"x-paperclip-cloud-user-id": "global-user-1",
|
||||
"x-paperclip-cloud-user-email": "owner@example.com",
|
||||
"x-paperclip-cloud-stack-id": "stack-alpha",
|
||||
"x-paperclip-cloud-stack-role": "member",
|
||||
};
|
||||
|
||||
// Reads and writes both reach the imported company through the real
|
||||
// membership row (write access also consults actor.memberships).
|
||||
const read = await request(app).get(`/companies/${importedCompanyId}/resource`).set(cloudHeaders);
|
||||
expect(read.status).toBe(200);
|
||||
expect(read.body.companyIds).toContain(importedCompanyId);
|
||||
|
||||
const write = await request(app).post(`/companies/${importedCompanyId}/resource`).set(cloudHeaders);
|
||||
expect(write.status).toBe(200);
|
||||
|
||||
// Companies the user holds no membership row in stay unreachable.
|
||||
const denied = await request(app).get(`/companies/${unrelatedCompanyId}/resource`).set(cloudHeaders);
|
||||
expect(denied.status).toBe(403);
|
||||
});
|
||||
|
||||
it("purges a stale instance_admin row so the session path stops elevating the cloud-tenant user", async () => {
|
||||
process.env.PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN = "tenant-token";
|
||||
// Simulates a deployment that previously ran the pre-hardening cloud_tenant
|
||||
|
||||
@@ -1,334 +0,0 @@
|
||||
import { generateKeyPairSync, randomUUID } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { companies, cloudUpstreamConnections, cloudUpstreamRuns, companySkills, createDb } from "@paperclipai/db";
|
||||
|
||||
import { HttpError } from "../errors.js";
|
||||
import {
|
||||
cloudUpstreamRemoteFailureReport,
|
||||
cloudUpstreamService,
|
||||
reconcileCloudUpstreamRunsOnStartup,
|
||||
sealCloudUpstreamCredential,
|
||||
unsealCloudUpstreamCredential,
|
||||
} from "../services/cloud-upstreams.js";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping embedded Postgres cloud upstream tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
||||
);
|
||||
}
|
||||
|
||||
describe("cloud upstream remote failures", () => {
|
||||
it("preserves the cloud response body and message on run reports", () => {
|
||||
const body = {
|
||||
error: "bad_request",
|
||||
message: "entities[42].body must be an object",
|
||||
errors: [{ path: "entities[42].body" }],
|
||||
};
|
||||
|
||||
expect(cloudUpstreamRemoteFailureReport(new HttpError(400, "bad_request", body))).toEqual({
|
||||
error: "bad_request",
|
||||
errorMessage: "entities[42].body must be an object",
|
||||
details: body,
|
||||
});
|
||||
});
|
||||
|
||||
it("falls back to the thrown error message for non-remote failures", () => {
|
||||
expect(cloudUpstreamRemoteFailureReport(new Error("network failed"))).toEqual({
|
||||
error: "network failed",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("cloud upstream credential storage", () => {
|
||||
const previousMasterKey = process.env.PAPERCLIP_SECRETS_MASTER_KEY;
|
||||
|
||||
afterEach(() => {
|
||||
if (previousMasterKey === undefined) {
|
||||
delete process.env.PAPERCLIP_SECRETS_MASTER_KEY;
|
||||
} else {
|
||||
process.env.PAPERCLIP_SECRETS_MASTER_KEY = previousMasterKey;
|
||||
}
|
||||
});
|
||||
|
||||
it("stores new credentials as encrypted envelopes and preserves legacy plaintext reads", async () => {
|
||||
process.env.PAPERCLIP_SECRETS_MASTER_KEY = "12345678901234567890123456789012";
|
||||
const sealed = await sealCloudUpstreamCredential("cloud-access-token");
|
||||
|
||||
expect(sealed).toMatch(/^paperclip-cloud-credential:/);
|
||||
expect(sealed).not.toContain("cloud-access-token");
|
||||
await expect(unsealCloudUpstreamCredential(sealed)).resolves.toBe("cloud-access-token");
|
||||
await expect(unsealCloudUpstreamCredential("legacy-plaintext-token")).resolves.toBe("legacy-plaintext-token");
|
||||
});
|
||||
});
|
||||
|
||||
describeEmbeddedPostgres("cloud upstream persistence", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
const previousMasterKey = process.env.PAPERCLIP_SECRETS_MASTER_KEY;
|
||||
|
||||
beforeAll(async () => {
|
||||
process.env.PAPERCLIP_SECRETS_MASTER_KEY = "12345678901234567890123456789012";
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-cloud-upstreams-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks();
|
||||
await db.delete(cloudUpstreamRuns);
|
||||
await db.delete(cloudUpstreamConnections);
|
||||
await db.delete(companySkills);
|
||||
await db.delete(companies);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (previousMasterKey === undefined) {
|
||||
delete process.env.PAPERCLIP_SECRETS_MASTER_KEY;
|
||||
} else {
|
||||
process.env.PAPERCLIP_SECRETS_MASTER_KEY = previousMasterKey;
|
||||
}
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
it("encrypts stored upstream credentials while keeping connection flows usable", async () => {
|
||||
const companyId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
const tokenUrl = "https://cloud.example.test/oauth/token";
|
||||
vi.spyOn(globalThis, "fetch").mockImplementation(async (input, init) => {
|
||||
const url = String(input);
|
||||
if (url.startsWith("https://cloud.example.test/.well-known/paperclip-upstream")) {
|
||||
return jsonResponse({
|
||||
product: "Paperclip Cloud",
|
||||
stack: {
|
||||
id: "stack-1",
|
||||
companyId: "cloud-company-1",
|
||||
origin: "https://cloud.example.test",
|
||||
primaryHost: "cloud.example.test",
|
||||
},
|
||||
transfer: {
|
||||
supportedSchemaMajor: 1,
|
||||
maxChunkBytes: 8192,
|
||||
},
|
||||
auth: {
|
||||
scopes: ["upstream_import:write"],
|
||||
pkce: {
|
||||
authorizeUrl: "https://cloud.example.test/oauth/authorize",
|
||||
tokenUrl,
|
||||
},
|
||||
},
|
||||
});
|
||||
}
|
||||
if (url === tokenUrl && init?.method === "POST") {
|
||||
const payload = JSON.parse(String(init.body));
|
||||
expect(payload.codeVerifier).toEqual(expect.any(String));
|
||||
expect(payload.codeVerifier).not.toContain("paperclip-cloud-credential:");
|
||||
return jsonResponse({
|
||||
accessToken: "cloud-access-token",
|
||||
token: {
|
||||
id: "token-1",
|
||||
expiresAt: "2026-05-22T13:00:00.000Z",
|
||||
globalUserId: "user-1",
|
||||
},
|
||||
});
|
||||
}
|
||||
throw new Error(`Unexpected fetch: ${url}`);
|
||||
});
|
||||
|
||||
const service = cloudUpstreamService(db, { instanceId: "test" });
|
||||
const started = await service.startConnect({
|
||||
companyId,
|
||||
remoteUrl: "https://cloud.example.test",
|
||||
redirectUri: "http://localhost:3100/callback",
|
||||
});
|
||||
await service.finishConnect({
|
||||
pendingConnectionId: started.pendingConnectionId,
|
||||
code: "auth-code",
|
||||
state: new URL(started.authorizationUrl).searchParams.get("state") ?? "",
|
||||
});
|
||||
|
||||
const [row] = await db.select().from(cloudUpstreamConnections);
|
||||
expect(row.privateKeyPem).toMatch(/^paperclip-cloud-credential:/);
|
||||
expect(row.privateKeyPem).not.toContain("BEGIN PRIVATE KEY");
|
||||
expect(row.accessToken).toMatch(/^paperclip-cloud-credential:/);
|
||||
expect(row.accessToken).not.toContain("cloud-access-token");
|
||||
});
|
||||
|
||||
it("marks orphaned running runs failed during startup reconciliation", async () => {
|
||||
const companyId = randomUUID();
|
||||
const connectionId = randomUUID();
|
||||
const runningRunId = randomUUID();
|
||||
const succeededRunId = randomUUID();
|
||||
const reconciledAt = new Date("2026-05-22T13:00:00.000Z");
|
||||
await seedCompany(companyId);
|
||||
await db.insert(cloudUpstreamConnections).values({
|
||||
id: connectionId,
|
||||
companyId,
|
||||
remoteUrl: "https://cloud.example.test",
|
||||
sourceInstanceId: "source-1",
|
||||
sourceInstanceFingerprint: "sha256:test",
|
||||
sourcePublicKey: "public-key",
|
||||
privateKeyPem: "legacy-private-key",
|
||||
tokenStatus: "connected",
|
||||
scopes: ["upstream_import:write"],
|
||||
authorizedGlobalUserId: "user-1",
|
||||
accessToken: "legacy-token",
|
||||
tokenId: "token-1",
|
||||
targetStackId: "stack-1",
|
||||
targetCompanyId: "cloud-company-1",
|
||||
targetOrigin: "https://cloud.example.test",
|
||||
targetPrimaryHost: "cloud.example.test",
|
||||
targetProduct: "Paperclip Cloud",
|
||||
targetSchemaMajor: 1,
|
||||
targetMaxChunkBytes: 8192,
|
||||
});
|
||||
await db.insert(cloudUpstreamRuns).values([
|
||||
cloudRunRow({ id: runningRunId, connectionId, companyId, status: "running" }),
|
||||
cloudRunRow({ id: succeededRunId, connectionId, companyId, status: "succeeded", completedAt: reconciledAt }),
|
||||
]);
|
||||
|
||||
await expect(reconcileCloudUpstreamRunsOnStartup(db, reconciledAt)).resolves.toEqual({ reconciled: 1 });
|
||||
|
||||
const rows = await db.select().from(cloudUpstreamRuns);
|
||||
const running = rows.find((row) => row.id === runningRunId);
|
||||
const succeeded = rows.find((row) => row.id === succeededRunId);
|
||||
expect(running?.status).toBe("failed");
|
||||
expect(running?.completedAt?.toISOString()).toBe(reconciledAt.toISOString());
|
||||
expect(running?.events.at(-1)?.message).toContain("server startup");
|
||||
expect(running?.report).toMatchObject({
|
||||
error: "orphaned_running_run",
|
||||
reconciledAt: reconciledAt.toISOString(),
|
||||
});
|
||||
expect(succeeded?.status).toBe("succeeded");
|
||||
});
|
||||
|
||||
it("rejects a new run when the connection already has a running run", async () => {
|
||||
const companyId = randomUUID();
|
||||
const connectionId = randomUUID();
|
||||
const runningRunId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
await db.insert(cloudUpstreamConnections).values(cloudConnectionRow({ id: connectionId, companyId }));
|
||||
await db.insert(cloudUpstreamRuns).values(
|
||||
cloudRunRow({ id: runningRunId, connectionId, companyId, status: "running" }),
|
||||
);
|
||||
|
||||
await expect(cloudUpstreamService(db).createRun({ connectionId, companyId })).rejects.toMatchObject({
|
||||
status: 409,
|
||||
details: { runId: runningRunId },
|
||||
});
|
||||
});
|
||||
|
||||
it("preserves a cancelled run when an in-flight createRun tries to finish", async () => {
|
||||
const companyId = randomUUID();
|
||||
const connectionId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
await db.insert(cloudUpstreamConnections).values(cloudConnectionRow({ id: connectionId, companyId }));
|
||||
|
||||
const service = cloudUpstreamService(db);
|
||||
const remoteCalls: string[] = [];
|
||||
globalThis.fetch = vi.fn(async (input) => {
|
||||
const path = new URL(String(input)).pathname;
|
||||
remoteCalls.push(path);
|
||||
if (path.endsWith("/upstream-imports/runs")) {
|
||||
return jsonResponse({ run: { id: "remote-run-1" } });
|
||||
}
|
||||
if (path.endsWith("/chunks")) {
|
||||
const run = await db.select().from(cloudUpstreamRuns).then((rows) => rows[0]);
|
||||
expect(run?.status).toBe("running");
|
||||
await service.cancelRun(connectionId, run.id, companyId);
|
||||
return jsonResponse({ ok: true });
|
||||
}
|
||||
if (path.endsWith("/cancel")) {
|
||||
return jsonResponse({ ok: true });
|
||||
}
|
||||
if (path.endsWith("/apply")) {
|
||||
return jsonResponse({ ok: true });
|
||||
}
|
||||
if (path.endsWith("/events")) {
|
||||
return jsonResponse({ events: [] });
|
||||
}
|
||||
return jsonResponse({ error: "not_found" }, 404);
|
||||
}) as typeof fetch;
|
||||
|
||||
const result = await service.createRun({ connectionId, companyId });
|
||||
|
||||
expect(result.status).toBe("cancelled");
|
||||
expect(remoteCalls.some((path) => path.endsWith("/apply"))).toBe(false);
|
||||
const rows = await db.select().from(cloudUpstreamRuns);
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]?.status).toBe("cancelled");
|
||||
});
|
||||
|
||||
async function seedCompany(companyId: string) {
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
function jsonResponse(body: unknown): Response {
|
||||
return new Response(JSON.stringify(body), {
|
||||
status: 200,
|
||||
headers: { "Content-Type": "application/json" },
|
||||
});
|
||||
}
|
||||
|
||||
function cloudConnectionRow(input: { id: string; companyId: string }) {
|
||||
const { privateKey } = generateKeyPairSync("ed25519");
|
||||
return {
|
||||
id: input.id,
|
||||
companyId: input.companyId,
|
||||
remoteUrl: "https://cloud.example.test",
|
||||
sourceInstanceId: "source-1",
|
||||
sourceInstanceFingerprint: "sha256:test",
|
||||
sourcePublicKey: "public-key",
|
||||
privateKeyPem: privateKey.export({ type: "pkcs8", format: "pem" }).toString(),
|
||||
tokenStatus: "connected",
|
||||
scopes: ["upstream_import:write"],
|
||||
authorizedGlobalUserId: "user-1",
|
||||
accessToken: "legacy-token",
|
||||
tokenId: "token-1",
|
||||
targetStackId: "stack-1",
|
||||
targetCompanyId: "cloud-company-1",
|
||||
targetOrigin: "https://cloud.example.test",
|
||||
targetPrimaryHost: "cloud.example.test",
|
||||
targetProduct: "Paperclip Cloud",
|
||||
targetSchemaMajor: 1,
|
||||
targetMaxChunkBytes: 8192,
|
||||
};
|
||||
}
|
||||
|
||||
function cloudRunRow(input: {
|
||||
id: string;
|
||||
connectionId: string;
|
||||
companyId: string;
|
||||
status: string;
|
||||
completedAt?: Date;
|
||||
}) {
|
||||
return {
|
||||
id: input.id,
|
||||
connectionId: input.connectionId,
|
||||
companyId: input.companyId,
|
||||
status: input.status,
|
||||
activeStep: "push",
|
||||
progressPercent: input.status === "running" ? 45 : 100,
|
||||
dryRun: false,
|
||||
summary: [],
|
||||
warnings: [],
|
||||
conflicts: [],
|
||||
events: [],
|
||||
report: {},
|
||||
idempotencyKey: `key-${input.id}`,
|
||||
manifestHash: `sha256:${input.id.replace(/-/g, "")}`,
|
||||
targetUrl: "https://cloud.example.test",
|
||||
completedAt: input.completedAt,
|
||||
};
|
||||
}
|
||||
@@ -253,6 +253,10 @@ describe.sequential("company route cross-company authorization", () => {
|
||||
label: "POST /api/companies/:companyId/exports/preview",
|
||||
request: (app: express.Express) => request(app).post(`/api/companies/${companyBId}/exports/preview`).send(exportRequest),
|
||||
},
|
||||
{
|
||||
label: "GET /api/companies/:companyId/export/fidelity",
|
||||
request: (app: express.Express) => request(app).get(`/api/companies/${companyBId}/export/fidelity`),
|
||||
},
|
||||
{
|
||||
label: "POST /api/companies/:companyId/imports/preview",
|
||||
request: (app: express.Express) => request(app).post(`/api/companies/${companyBId}/imports/preview`).send(importRequest()),
|
||||
|
||||
@@ -38,6 +38,9 @@ vi.mock("../services/index.js", () => ({
|
||||
getFeedbackTraceById: vi.fn(),
|
||||
saveIssueVote: vi.fn(),
|
||||
}),
|
||||
instanceSettingsService: () => ({
|
||||
getExperimental: vi.fn(),
|
||||
}),
|
||||
logActivity: vi.fn(),
|
||||
}));
|
||||
|
||||
|
||||
@@ -51,6 +51,9 @@ vi.mock("../services/index.js", () => ({
|
||||
companyPortabilityService: () => mockCompanyPortabilityService,
|
||||
companyService: () => mockCompanyService,
|
||||
feedbackService: () => mockFeedbackService,
|
||||
instanceSettingsService: () => ({
|
||||
getExperimental: vi.fn(),
|
||||
}),
|
||||
logActivity: mockLogActivity,
|
||||
}));
|
||||
|
||||
|
||||
@@ -0,0 +1,485 @@
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
assets,
|
||||
createDb,
|
||||
documentRevisions,
|
||||
documents,
|
||||
issueAttachments,
|
||||
issueComments,
|
||||
issueDocuments,
|
||||
issueInboxArchives,
|
||||
issueLabels,
|
||||
issueRelations,
|
||||
issueWorkProducts,
|
||||
issues,
|
||||
} from "@paperclipai/db";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { companyPortabilityService } from "../services/company-portability.js";
|
||||
import { issueService } from "../services/issues.js";
|
||||
import { workProductService } from "../services/work-products.js";
|
||||
import type { ImportIssueWorkProductRow } from "../services/import-write-types.js";
|
||||
import type { CompanyPortabilityFileEntry } from "@paperclipai/shared";
|
||||
import { randomUUID } from "node:crypto";
|
||||
|
||||
// This suite proves the company-import writers batch their inserts: it runs a
|
||||
// real import against embedded Postgres and counts the SQL insert statements
|
||||
// issued per table. The non-benchmark test guards against a regression to the
|
||||
// old one-insert-per-row pattern in normal CI; the benchmark (opt-in via
|
||||
// PORTABILITY_IMPORT_BENCH=1) reports wall-clock and statement counts at scale.
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
// Tables whose insert statements we count, keyed by a readable label.
|
||||
const COUNTED_TABLES: Array<[string, unknown]> = [
|
||||
["issues", issues],
|
||||
["issue_labels", issueLabels],
|
||||
["issue_comments", issueComments],
|
||||
["documents", documents],
|
||||
["document_revisions", documentRevisions],
|
||||
["issue_documents", issueDocuments],
|
||||
["issue_work_products", issueWorkProducts],
|
||||
["assets", assets],
|
||||
["issue_attachments", issueAttachments],
|
||||
["issue_relations", issueRelations],
|
||||
];
|
||||
|
||||
interface InsertCounts {
|
||||
total: number;
|
||||
byTable: Map<string, number>;
|
||||
reset: () => void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Patch a drizzle db so every `insert()` call (on the db and on any
|
||||
* transaction handed to `db.transaction`) is counted. Each `insert(table)`
|
||||
* call corresponds to exactly one SQL statement, so this counts statements.
|
||||
*/
|
||||
function instrumentInserts(db: Record<string, unknown>): InsertCounts {
|
||||
const counts: InsertCounts = {
|
||||
total: 0,
|
||||
byTable: new Map(),
|
||||
reset() {
|
||||
this.total = 0;
|
||||
this.byTable.clear();
|
||||
},
|
||||
};
|
||||
const labelFor = (table: unknown): string | null => {
|
||||
for (const [label, ref] of COUNTED_TABLES) {
|
||||
if (table === ref) return label;
|
||||
}
|
||||
return null;
|
||||
};
|
||||
const patchInsert = (target: Record<string, unknown>) => {
|
||||
const original = (target.insert as (table: unknown) => unknown).bind(target);
|
||||
target.insert = (table: unknown) => {
|
||||
counts.total += 1;
|
||||
const label = labelFor(table);
|
||||
if (label) counts.byTable.set(label, (counts.byTable.get(label) ?? 0) + 1);
|
||||
return original(table);
|
||||
};
|
||||
};
|
||||
patchInsert(db);
|
||||
const originalTransaction = (db.transaction as (fn: unknown, config?: unknown) => unknown).bind(db);
|
||||
db.transaction = (fn: (tx: Record<string, unknown>) => unknown, config?: unknown) =>
|
||||
originalTransaction((tx: Record<string, unknown>) => {
|
||||
patchInsert(tx);
|
||||
return fn(tx);
|
||||
}, config);
|
||||
return counts;
|
||||
}
|
||||
|
||||
// The `.paperclip.yaml` extension is consumed by the importer's own hand-rolled
|
||||
// block-YAML parser (not a general YAML/JSON reader), so render it with the same
|
||||
// block algorithm the exporter uses to guarantee a round-trip-compatible bundle.
|
||||
function renderYamlScalar(value: unknown): string {
|
||||
if (value === null) return "null";
|
||||
if (typeof value === "boolean" || typeof value === "number") return String(value);
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
function isRenderableScalar(value: unknown): boolean {
|
||||
return (
|
||||
value === null
|
||||
|| typeof value === "string"
|
||||
|| typeof value === "boolean"
|
||||
|| typeof value === "number"
|
||||
|| (Array.isArray(value) && value.length === 0)
|
||||
|| (value !== null && typeof value === "object" && !Array.isArray(value) && Object.keys(value).length === 0)
|
||||
);
|
||||
}
|
||||
|
||||
function renderYamlBlock(value: unknown, indentLevel: number): string[] {
|
||||
const indent = " ".repeat(indentLevel);
|
||||
if (Array.isArray(value)) {
|
||||
if (value.length === 0) return [`${indent}[]`];
|
||||
const lines: string[] = [];
|
||||
for (const entry of value) {
|
||||
if (isRenderableScalar(entry)) {
|
||||
lines.push(`${indent}- ${renderYamlScalar(entry)}`);
|
||||
continue;
|
||||
}
|
||||
lines.push(`${indent}-`);
|
||||
lines.push(...renderYamlBlock(entry, indentLevel + 1));
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
if (value !== null && typeof value === "object") {
|
||||
const entries = Object.entries(value as Record<string, unknown>);
|
||||
if (entries.length === 0) return [`${indent}{}`];
|
||||
const lines: string[] = [];
|
||||
for (const [key, entry] of entries) {
|
||||
if (isRenderableScalar(entry)) {
|
||||
lines.push(`${indent}${key}: ${renderYamlScalar(entry)}`);
|
||||
continue;
|
||||
}
|
||||
lines.push(`${indent}${key}:`);
|
||||
lines.push(...renderYamlBlock(entry, indentLevel + 1));
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
return [`${indent}${renderYamlScalar(value)}`];
|
||||
}
|
||||
|
||||
function renderPaperclipYaml(value: Record<string, unknown>): string {
|
||||
return `${renderYamlBlock(value, 0).join("\n")}\n`;
|
||||
}
|
||||
|
||||
interface SyntheticBundleOptions {
|
||||
issueCount: number;
|
||||
commentsPerIssue: number;
|
||||
documentsPerIssue: number;
|
||||
}
|
||||
|
||||
/** Build an inline import bundle with the requested shape, valid at schemaVersion 6. */
|
||||
function buildSyntheticBundle(options: SyntheticBundleOptions): {
|
||||
rootPath: string;
|
||||
files: Record<string, CompanyPortabilityFileEntry>;
|
||||
} {
|
||||
const files: Record<string, CompanyPortabilityFileEntry> = {};
|
||||
files["COMPANY.md"] = ['---', 'schema: "agentcompanies/v1"', 'name: "Batching Bench Co"', '---', '', 'Synthetic import bundle.', ''].join("\n");
|
||||
|
||||
const tasks: Record<string, unknown> = {};
|
||||
for (let i = 1; i <= options.issueCount; i += 1) {
|
||||
const slug = `task-${String(i).padStart(4, "0")}`;
|
||||
files[`tasks/${slug}/TASK.md`] = ['---', `name: "Task ${i}"`, "kind: task", '---', '', `Description body for task ${i}. `.repeat(6), ''].join("\n");
|
||||
|
||||
const comments = [];
|
||||
for (let c = 0; c < options.commentsPerIssue; c += 1) {
|
||||
comments.push({
|
||||
body: `Comment ${c + 1} on task ${i}. `.repeat(4),
|
||||
authorType: "system",
|
||||
});
|
||||
}
|
||||
const taskDocuments = [];
|
||||
for (let d = 0; d < options.documentsPerIssue; d += 1) {
|
||||
const key = d === 0 ? "spec" : `doc-${d + 1}`;
|
||||
const docPath = `tasks/${slug}/documents/${key}.md`;
|
||||
files[docPath] = `# ${key}\n\n${`Document body for task ${i} ${key}. `.repeat(8)}`;
|
||||
taskDocuments.push({ key, title: key, format: "markdown", path: docPath });
|
||||
}
|
||||
|
||||
tasks[slug] = {
|
||||
status: "todo",
|
||||
priority: "medium",
|
||||
comments,
|
||||
documents: taskDocuments,
|
||||
};
|
||||
}
|
||||
|
||||
files[".paperclip.yaml"] = renderPaperclipYaml({ schemaVersion: 6, tasks });
|
||||
|
||||
return { rootPath: "batching-bench", files };
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a bundle whose every task carries a *user*-authored comment. On import,
|
||||
* user comments are re-attributed to the importing user, so every issue becomes
|
||||
* "touched by me" — the exact shape that used to flood the inbox after an
|
||||
* import. The inbox seeding is what must keep them hidden.
|
||||
*/
|
||||
function buildTouchedIssuesBundle(issueCount: number): {
|
||||
rootPath: string;
|
||||
files: Record<string, CompanyPortabilityFileEntry>;
|
||||
} {
|
||||
const files: Record<string, CompanyPortabilityFileEntry> = {};
|
||||
files["COMPANY.md"] = ['---', 'schema: "agentcompanies/v1"', 'name: "Inbox Flood Co"', '---', '', 'Synthetic import bundle.', ''].join("\n");
|
||||
|
||||
const tasks: Record<string, unknown> = {};
|
||||
for (let i = 1; i <= issueCount; i += 1) {
|
||||
const slug = `task-${String(i).padStart(4, "0")}`;
|
||||
files[`tasks/${slug}/TASK.md`] = ['---', `name: "Task ${i}"`, "kind: task", '---', '', `Body for task ${i}.`, ''].join("\n");
|
||||
tasks[slug] = {
|
||||
status: "todo",
|
||||
priority: "medium",
|
||||
comments: [{ body: `A human note on task ${i}.`, authorType: "user" }],
|
||||
documents: [],
|
||||
};
|
||||
}
|
||||
|
||||
files[".paperclip.yaml"] = renderPaperclipYaml({ schemaVersion: 6, tasks });
|
||||
return { rootPath: "inbox-flood", files };
|
||||
}
|
||||
|
||||
describeEmbeddedPostgres("company import batches inserts", () => {
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let counts!: InsertCounts;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-import-batching-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
counts = instrumentInserts(db as unknown as Record<string, unknown>);
|
||||
}, 30_000);
|
||||
|
||||
afterAll(async () => {
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
counts.reset();
|
||||
});
|
||||
|
||||
it("imports many issues with a bounded number of issue-insert statements", async () => {
|
||||
const issueCount = 50;
|
||||
const commentsPerIssue = 2;
|
||||
const documentsPerIssue = 1;
|
||||
const bundle = buildSyntheticBundle({ issueCount, commentsPerIssue, documentsPerIssue });
|
||||
const portability = companyPortabilityService(db);
|
||||
|
||||
const result = await portability.importBundle(
|
||||
{
|
||||
source: { type: "inline", rootPath: bundle.rootPath, files: bundle.files },
|
||||
include: { company: true, agents: false, projects: false, issues: true },
|
||||
target: { mode: "new_company", newCompanyName: "Batching Regression Co" },
|
||||
collisionStrategy: "rename",
|
||||
},
|
||||
"user-batching-regression",
|
||||
);
|
||||
|
||||
const companyId = result.company.id;
|
||||
const issueInserts = counts.byTable.get("issues") ?? 0;
|
||||
const commentInserts = counts.byTable.get("issue_comments") ?? 0;
|
||||
|
||||
// The whole point: a 50-issue import must not issue one insert per issue.
|
||||
expect(issueInserts).toBeLessThan(issueCount);
|
||||
// 50 issues fit in a single chunk; the exact number is small and stable.
|
||||
expect(issueInserts).toBeLessThanOrEqual(2);
|
||||
expect(commentInserts).toBeLessThanOrEqual(2);
|
||||
|
||||
// And the data must actually land, unchanged in shape.
|
||||
const [{ issueRows }] = await db
|
||||
.select({ issueRows: sql<number>`count(*)::int` })
|
||||
.from(issues)
|
||||
.where(eq(issues.companyId, companyId));
|
||||
const [{ commentRows }] = await db
|
||||
.select({ commentRows: sql<number>`count(*)::int` })
|
||||
.from(issueComments)
|
||||
.where(eq(issueComments.companyId, companyId));
|
||||
// Count the issue-document links so company-bootstrap routine documents
|
||||
// (seeded on company creation) don't skew the imported-document count.
|
||||
const [{ documentLinkRows }] = await db
|
||||
.select({ documentLinkRows: sql<number>`count(*)::int` })
|
||||
.from(issueDocuments)
|
||||
.where(eq(issueDocuments.companyId, companyId));
|
||||
|
||||
expect(issueRows).toBe(issueCount);
|
||||
expect(commentRows).toBe(issueCount * commentsPerIssue);
|
||||
expect(documentLinkRows).toBe(issueCount * documentsPerIssue);
|
||||
expect(result.warnings).toEqual([]);
|
||||
|
||||
// Identifiers are a contiguous range allocated from the company counter.
|
||||
const identifiers = await db
|
||||
.select({ identifier: issues.identifier })
|
||||
.from(issues)
|
||||
.where(eq(issues.companyId, companyId));
|
||||
const uniqueIdentifiers = new Set(identifiers.map((row) => row.identifier));
|
||||
expect(uniqueIdentifiers.size).toBe(issueCount);
|
||||
});
|
||||
|
||||
it("keeps imported issues out of the importing user's inbox", async () => {
|
||||
const importingUserId = "board-inbox-user";
|
||||
const bundle = buildTouchedIssuesBundle(3);
|
||||
const portability = companyPortabilityService(db);
|
||||
|
||||
const result = await portability.importBundle(
|
||||
{
|
||||
source: { type: "inline", rootPath: bundle.rootPath, files: bundle.files },
|
||||
include: { company: true, agents: false, projects: false, issues: true },
|
||||
target: { mode: "new_company", newCompanyName: "Inbox Flood Co" },
|
||||
collisionStrategy: "rename",
|
||||
},
|
||||
importingUserId,
|
||||
);
|
||||
const companyId = result.company.id;
|
||||
|
||||
const importedIssues = await db
|
||||
.select({ id: issues.id })
|
||||
.from(issues)
|
||||
.where(eq(issues.companyId, companyId));
|
||||
expect(importedIssues.length).toBe(3);
|
||||
const importedIds = new Set(importedIssues.map((row) => row.id));
|
||||
|
||||
// Every imported issue is seeded with a per-user inbox archive, attributed
|
||||
// to the importing board user (not an agent).
|
||||
const archives = await db
|
||||
.select({
|
||||
issueId: issueInboxArchives.issueId,
|
||||
archivedByActorType: issueInboxArchives.archivedByActorType,
|
||||
})
|
||||
.from(issueInboxArchives)
|
||||
.where(and(
|
||||
eq(issueInboxArchives.companyId, companyId),
|
||||
eq(issueInboxArchives.userId, importingUserId),
|
||||
));
|
||||
expect(new Set(archives.map((row) => row.issueId))).toEqual(importedIds);
|
||||
expect(archives.every((row) => row.archivedByActorType === "user")).toBe(true);
|
||||
|
||||
const issuesSvc = issueService(db);
|
||||
|
||||
// Sanity: the imported issues really are "touched by me" (their user comment
|
||||
// was re-attributed to the importing user), so without the archive filter
|
||||
// they would all flood the inbox.
|
||||
const touched = await issuesSvc.list(companyId, { touchedByUserId: importingUserId });
|
||||
expect(new Set(touched.map((issue) => issue.id))).toEqual(importedIds);
|
||||
|
||||
// The inbox "mine" query (touched AND not inbox-archived) returns none of
|
||||
// them — the seeding is load-bearing.
|
||||
const mineInbox = await issuesSvc.list(companyId, {
|
||||
touchedByUserId: importingUserId,
|
||||
inboxArchivedByUserId: importingUserId,
|
||||
});
|
||||
expect(mineInbox.filter((issue) => importedIds.has(issue.id))).toEqual([]);
|
||||
|
||||
// Normal (non-import) creation is unaffected: an issue the same user creates
|
||||
// after the import is not archived and shows up in their inbox.
|
||||
const fresh = await issuesSvc.create(companyId, {
|
||||
title: "Freshly created work",
|
||||
status: "todo",
|
||||
priority: "medium",
|
||||
createdByUserId: importingUserId,
|
||||
});
|
||||
const mineAfterCreate = await issuesSvc.list(companyId, {
|
||||
touchedByUserId: importingUserId,
|
||||
inboxArchivedByUserId: importingUserId,
|
||||
});
|
||||
expect(mineAfterCreate.map((issue) => issue.id)).toContain(fresh.id);
|
||||
expect(mineAfterCreate.filter((issue) => importedIds.has(issue.id))).toEqual([]);
|
||||
});
|
||||
|
||||
it("rolls back the whole work-product batch when a later chunk fails", async () => {
|
||||
// Seed a real company + issue to hang work products off of.
|
||||
const bundle = buildSyntheticBundle({ issueCount: 1, commentsPerIssue: 0, documentsPerIssue: 0 });
|
||||
const portability = companyPortabilityService(db);
|
||||
const seeded = await portability.importBundle(
|
||||
{
|
||||
source: { type: "inline", rootPath: bundle.rootPath, files: bundle.files },
|
||||
include: { company: true, agents: false, projects: false, issues: true },
|
||||
target: { mode: "new_company", newCompanyName: "Work Product Atomicity Co" },
|
||||
collisionStrategy: "rename",
|
||||
},
|
||||
"user-work-product-atomicity",
|
||||
);
|
||||
const companyId = seeded.company.id;
|
||||
const [issueRow] = await db
|
||||
.select({ id: issues.id })
|
||||
.from(issues)
|
||||
.where(eq(issues.companyId, companyId))
|
||||
.limit(1);
|
||||
expect(issueRow?.id).toBeTruthy();
|
||||
|
||||
const buildRow = (issueId: string, index: number): ImportIssueWorkProductRow => ({
|
||||
companyId,
|
||||
issueId,
|
||||
projectId: null,
|
||||
type: "pull_request",
|
||||
provider: "github",
|
||||
externalId: null,
|
||||
title: `Work product ${index}`,
|
||||
url: null,
|
||||
status: "open",
|
||||
reviewState: "none",
|
||||
isPrimary: false,
|
||||
healthStatus: "unknown",
|
||||
summary: null,
|
||||
metadata: null,
|
||||
executionWorkspaceId: null,
|
||||
runtimeServiceId: null,
|
||||
createdByRunId: null,
|
||||
sourceTrust: null,
|
||||
});
|
||||
|
||||
// 501 rows span two chunks (chunk size is 500). The first 500 are valid;
|
||||
// the 501st points at a non-existent issue so the *second* chunk's insert
|
||||
// fails after the first chunk already ran. Without a wrapping transaction
|
||||
// the first 500 would have committed; with it, nothing may persist.
|
||||
const rows: ImportIssueWorkProductRow[] = [];
|
||||
for (let i = 0; i < 500; i += 1) rows.push(buildRow(issueRow!.id, i));
|
||||
rows.push(buildRow(randomUUID(), 500));
|
||||
|
||||
await expect(workProductService(db).createManyForImport(rows)).rejects.toThrow();
|
||||
|
||||
const [{ workProductRows }] = await db
|
||||
.select({ workProductRows: sql<number>`count(*)::int` })
|
||||
.from(issueWorkProducts)
|
||||
.where(eq(issueWorkProducts.companyId, companyId));
|
||||
expect(workProductRows).toBe(0);
|
||||
});
|
||||
|
||||
const benchmark = process.env.PORTABILITY_IMPORT_BENCH === "1" ? it : it.skip;
|
||||
benchmark(
|
||||
"benchmark: large import stays in the tens of statements, not thousands",
|
||||
async () => {
|
||||
const issueCount = 500;
|
||||
const commentsPerIssue = 5;
|
||||
const documentsPerIssue = 1;
|
||||
const bundle = buildSyntheticBundle({ issueCount, commentsPerIssue, documentsPerIssue });
|
||||
const totalRows =
|
||||
issueCount + issueCount * commentsPerIssue + issueCount * documentsPerIssue * 3;
|
||||
const portability = companyPortabilityService(db);
|
||||
|
||||
const startedAt = performance.now();
|
||||
const result = await portability.importBundle(
|
||||
{
|
||||
source: { type: "inline", rootPath: bundle.rootPath, files: bundle.files },
|
||||
include: { company: true, agents: false, projects: false, issues: true },
|
||||
target: { mode: "new_company", newCompanyName: "Batching Benchmark Co" },
|
||||
collisionStrategy: "rename",
|
||||
},
|
||||
"user-batching-benchmark",
|
||||
);
|
||||
const elapsedMs = performance.now() - startedAt;
|
||||
|
||||
const perTable = Object.fromEntries(counts.byTable);
|
||||
// Analytical "before": the old path inserted one row per statement.
|
||||
const beforeStatements = totalRows;
|
||||
const afterStatements = counts.total;
|
||||
|
||||
// eslint-disable-next-line no-console
|
||||
console.log(
|
||||
`[import-batching-bench] ${issueCount} issues x ${commentsPerIssue} comments x ${documentsPerIssue} doc\n`
|
||||
+ ` wall-clock: ${elapsedMs.toFixed(0)}ms\n`
|
||||
+ ` insert statements (after batching): ${afterStatements}\n`
|
||||
+ ` insert statements (before, one per row): ~${beforeStatements}\n`
|
||||
+ ` reduction: ${(beforeStatements / Math.max(1, afterStatements)).toFixed(0)}x\n`
|
||||
+ ` per-table insert statements: ${JSON.stringify(perTable)}`,
|
||||
);
|
||||
|
||||
const companyId = result.company.id;
|
||||
const [{ issueRows }] = await db
|
||||
.select({ issueRows: sql<number>`count(*)::int` })
|
||||
.from(issues)
|
||||
.where(eq(issues.companyId, companyId));
|
||||
expect(issueRows).toBe(issueCount);
|
||||
|
||||
// Two orders of magnitude fewer statements than the per-row baseline.
|
||||
expect(afterStatements * 50).toBeLessThan(beforeStatements);
|
||||
expect(counts.byTable.get("issues") ?? 0).toBeLessThanOrEqual(2);
|
||||
},
|
||||
120_000,
|
||||
);
|
||||
});
|
||||
@@ -43,6 +43,10 @@ const mockFeedbackService = vi.hoisted(() => ({
|
||||
saveIssueVote: vi.fn(),
|
||||
}));
|
||||
|
||||
const mockInstanceSettingsService = vi.hoisted(() => ({
|
||||
getExperimental: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock("../services/access.js", () => ({
|
||||
accessService: () => mockAccessService,
|
||||
}));
|
||||
@@ -79,6 +83,7 @@ vi.mock("../services/index.js", () => ({
|
||||
companyPortabilityService: () => mockCompanyPortabilityService,
|
||||
companyService: () => mockCompanyService,
|
||||
feedbackService: () => mockFeedbackService,
|
||||
instanceSettingsService: () => mockInstanceSettingsService,
|
||||
logActivity: mockLogActivity,
|
||||
}));
|
||||
|
||||
@@ -91,6 +96,7 @@ function registerCompanyRouteMocks() {
|
||||
companyPortabilityService: () => mockCompanyPortabilityService,
|
||||
companyService: () => mockCompanyService,
|
||||
feedbackService: () => mockFeedbackService,
|
||||
instanceSettingsService: () => mockInstanceSettingsService,
|
||||
logActivity: mockLogActivity,
|
||||
}));
|
||||
}
|
||||
@@ -198,6 +204,135 @@ async function waitForCondition(condition: () => boolean, label: string) {
|
||||
throw new Error(`Timed out waiting for ${label}`);
|
||||
}
|
||||
|
||||
const TEST_USER_HEADER = "x-test-user-id";
|
||||
|
||||
function boardActor(userId: string) {
|
||||
return {
|
||||
type: "board",
|
||||
userId,
|
||||
userName: "Board User",
|
||||
userEmail: `${userId}@example.com`,
|
||||
companyIds: [companyId],
|
||||
memberships: [{ companyId, membershipRole: "owner", status: "active" }],
|
||||
isInstanceAdmin: true,
|
||||
source: "session",
|
||||
};
|
||||
}
|
||||
|
||||
// A single companyRoutes instance (one shared in-memory job map) whose board
|
||||
// actor is chosen per request from the `x-test-user-id` header. This lets one
|
||||
// app exercise cross-user isolation and the per-user duplicate guard.
|
||||
async function createBoardApp() {
|
||||
registerCompanyRouteMocks();
|
||||
appImportCounter += 1;
|
||||
const routeModulePath = `../routes/companies.js?company-portability-routes-${appImportCounter}`;
|
||||
const middlewareModulePath = `../middleware/index.js?company-portability-routes-${appImportCounter}`;
|
||||
const [{ companyRoutes }, { errorHandler }] = await Promise.all([
|
||||
import(routeModulePath) as Promise<typeof import("../routes/companies.js")>,
|
||||
import(middlewareModulePath) as Promise<typeof import("../middleware/index.js")>,
|
||||
]);
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
const header = req.headers[TEST_USER_HEADER];
|
||||
const userId = typeof header === "string" && header.length > 0 ? header : "board-user-a";
|
||||
(req as any).actor = boardActor(userId);
|
||||
next();
|
||||
});
|
||||
app.use("/api/companies", companyRoutes({} as any));
|
||||
app.use(errorHandler);
|
||||
return app;
|
||||
}
|
||||
|
||||
async function waitForImportJobStatusAs(
|
||||
app: express.Express,
|
||||
statusUrl: string,
|
||||
status: string,
|
||||
headers: Record<string, string>,
|
||||
) {
|
||||
for (let attempt = 0; attempt < 20; attempt += 1) {
|
||||
const res = await request(app).get(statusUrl).set(headers);
|
||||
if (res.body.job?.status === status) {
|
||||
return res;
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
}
|
||||
throw new Error(`Timed out waiting for import job to reach ${status}`);
|
||||
}
|
||||
|
||||
// A minimal STORE-only zip writer so the multipart/zip upload path can be
|
||||
// exercised end-to-end: the route unzips this into the same inline bundle an
|
||||
// application/json caller would send. Layout matches ui/src/lib/zip.ts (local
|
||||
// file headers, central directory, end-of-central-directory).
|
||||
function crc32(bytes: Uint8Array) {
|
||||
let crc = 0xffffffff;
|
||||
for (const byte of bytes) {
|
||||
crc ^= byte;
|
||||
for (let bit = 0; bit < 8; bit += 1) {
|
||||
crc = (crc & 1) === 1 ? (crc >>> 1) ^ 0xedb88320 : crc >>> 1;
|
||||
}
|
||||
}
|
||||
return (crc ^ 0xffffffff) >>> 0;
|
||||
}
|
||||
|
||||
function buildStoreZip(files: Record<string, string>, rootPath: string): Buffer {
|
||||
const encoder = new TextEncoder();
|
||||
const localChunks: Buffer[] = [];
|
||||
const centralChunks: Buffer[] = [];
|
||||
let localOffset = 0;
|
||||
const entries = Object.entries(files);
|
||||
|
||||
for (const [relativePath, content] of entries) {
|
||||
const fileName = encoder.encode(`${rootPath}/${relativePath}`);
|
||||
const body = Buffer.from(encoder.encode(content));
|
||||
const checksum = crc32(body);
|
||||
|
||||
const localHeader = Buffer.alloc(30 + fileName.length);
|
||||
localHeader.writeUInt32LE(0x04034b50, 0);
|
||||
localHeader.writeUInt16LE(20, 4);
|
||||
localHeader.writeUInt16LE(0x0800, 6);
|
||||
localHeader.writeUInt32LE(checksum, 14);
|
||||
localHeader.writeUInt32LE(body.length, 18);
|
||||
localHeader.writeUInt32LE(body.length, 22);
|
||||
localHeader.writeUInt16LE(fileName.length, 26);
|
||||
Buffer.from(fileName).copy(localHeader, 30);
|
||||
|
||||
const centralHeader = Buffer.alloc(46 + fileName.length);
|
||||
centralHeader.writeUInt32LE(0x02014b50, 0);
|
||||
centralHeader.writeUInt16LE(20, 4);
|
||||
centralHeader.writeUInt16LE(20, 6);
|
||||
centralHeader.writeUInt16LE(0x0800, 8);
|
||||
centralHeader.writeUInt32LE(checksum, 16);
|
||||
centralHeader.writeUInt32LE(body.length, 20);
|
||||
centralHeader.writeUInt32LE(body.length, 24);
|
||||
centralHeader.writeUInt16LE(fileName.length, 28);
|
||||
centralHeader.writeUInt32LE(localOffset, 42);
|
||||
Buffer.from(fileName).copy(centralHeader, 46);
|
||||
|
||||
localChunks.push(localHeader, body);
|
||||
centralChunks.push(centralHeader);
|
||||
localOffset += localHeader.length + body.length;
|
||||
}
|
||||
|
||||
const centralDirectory = Buffer.concat(centralChunks);
|
||||
const eocd = Buffer.alloc(22);
|
||||
eocd.writeUInt32LE(0x06054b50, 0);
|
||||
eocd.writeUInt16LE(entries.length, 8);
|
||||
eocd.writeUInt16LE(entries.length, 10);
|
||||
eocd.writeUInt32LE(centralDirectory.length, 12);
|
||||
eocd.writeUInt32LE(localOffset, 16);
|
||||
|
||||
return Buffer.concat([...localChunks, centralDirectory, eocd]);
|
||||
}
|
||||
|
||||
// The import fields a multipart caller ships as the JSON `meta` field: the same
|
||||
// object an inline caller sends, minus `source` (the source is the uploaded zip).
|
||||
const importMeta = {
|
||||
include: importRequest.include,
|
||||
target: importRequest.target,
|
||||
collisionStrategy: importRequest.collisionStrategy,
|
||||
};
|
||||
|
||||
describe.sequential("company portability routes", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
@@ -635,7 +770,7 @@ describe.sequential("company portability routes", () => {
|
||||
expect(accepted.body.statusUrl).toMatch(/^\/api\/companies\/import\/jobs\/tenant-import-/);
|
||||
expect(accepted.body.retryAfterMs).toBe(1000);
|
||||
await waitForCondition(() => mockCompanyPortabilityService.importBundle.mock.calls.length === 1, "import job start");
|
||||
expect(mockCompanyPortabilityService.importBundle).toHaveBeenCalledWith(importRequest, "cloud-user-1");
|
||||
expect(mockCompanyPortabilityService.importBundle).toHaveBeenCalledWith(importRequest, "cloud-user-1", { pauseAutomations: false });
|
||||
expect(mockLogActivity).not.toHaveBeenCalled();
|
||||
|
||||
resolveImport(createImportResult("updated"));
|
||||
@@ -655,7 +790,23 @@ describe.sequential("company portability routes", () => {
|
||||
}),
|
||||
}));
|
||||
|
||||
const nowSpy = vi.spyOn(Date, "now").mockReturnValue(Date.parse(succeeded.body.job.completedAt) + (5 * 60 * 1000) + 1);
|
||||
const completedAtMs = Date.parse(succeeded.body.job.completedAt);
|
||||
|
||||
// A completed job stays resolvable well past the old 5-minute retention so a
|
||||
// user who steps away during a long import never polls into a false 404.
|
||||
const withinWindow = vi.spyOn(Date, "now").mockReturnValue(completedAtMs + (5 * 60 * 1000) + 1);
|
||||
try {
|
||||
const stillThere = await request(app).get(accepted.body.statusUrl).set(cloudHeaders);
|
||||
expect(stillThere.status).toBe(200);
|
||||
expect(stillThere.body.job.status).toBe("succeeded");
|
||||
expect(stillThere.body.job.result.companyId).toBe(companyId);
|
||||
} finally {
|
||||
withinWindow.mockRestore();
|
||||
}
|
||||
|
||||
// Only past the extended 60-minute window is the in-memory record finally
|
||||
// dropped, and only then does the status route report the job as gone.
|
||||
const nowSpy = vi.spyOn(Date, "now").mockReturnValue(completedAtMs + (60 * 60 * 1000) + 1);
|
||||
try {
|
||||
const expired = await request(app).get(accepted.body.statusUrl).set(cloudHeaders);
|
||||
expect(expired.status).toBe(404);
|
||||
@@ -721,10 +872,360 @@ describe.sequential("company portability routes", () => {
|
||||
expect(res.body.company.id).toBe(companyId);
|
||||
expect(res.body.company.action).toBe("created");
|
||||
expect(res.body.job).toBeUndefined();
|
||||
expect(mockCompanyPortabilityService.importBundle).toHaveBeenCalledWith(importRequest, "cloud-user-1");
|
||||
expect(mockCompanyPortabilityService.importBundle).toHaveBeenCalledWith(importRequest, "cloud-user-1", { pauseAutomations: false });
|
||||
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
|
||||
action: "company.imported",
|
||||
companyId,
|
||||
}));
|
||||
});
|
||||
|
||||
it.sequential("forwards pauseAutomations from the global import body to the portability service", async () => {
|
||||
mockCompanyPortabilityService.importBundle.mockResolvedValueOnce(createImportResult("created"));
|
||||
const app = await createApp(cloudTenantActor());
|
||||
|
||||
const res = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set(cloudHeaders)
|
||||
.send({ ...importRequest, pauseAutomations: true });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(mockCompanyPortabilityService.importBundle).toHaveBeenCalledWith(
|
||||
{ ...importRequest, pauseAutomations: true },
|
||||
"cloud-user-1",
|
||||
{ pauseAutomations: true },
|
||||
);
|
||||
});
|
||||
|
||||
it.sequential("runs board-session async imports as jobs and reports the full result by job id", async () => {
|
||||
let resolveImport: (value: ReturnType<typeof createImportResult>) => void = () => undefined;
|
||||
const pendingImport = new Promise<ReturnType<typeof createImportResult>>((resolve) => {
|
||||
resolveImport = resolve;
|
||||
});
|
||||
mockCompanyPortabilityService.importBundle.mockReturnValueOnce(pendingImport);
|
||||
const app = await createBoardApp();
|
||||
|
||||
const accepted = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set("x-paperclip-cloud-async-import", "1")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.send(importRequest);
|
||||
|
||||
expect(accepted.status).toBe(202);
|
||||
expect(accepted.body.job.status).toBe("running");
|
||||
// Board jobs get their own id prefix, distinct from the tenant-import prefix.
|
||||
expect(accepted.body.statusUrl).toMatch(/^\/api\/companies\/import\/jobs\/import-/);
|
||||
expect(accepted.body.statusUrl).not.toMatch(/\/jobs\/tenant-import-/);
|
||||
await waitForCondition(() => mockCompanyPortabilityService.importBundle.mock.calls.length === 1, "board import start");
|
||||
expect(mockCompanyPortabilityService.importBundle).toHaveBeenCalledWith(importRequest, "board-user-a", { pauseAutomations: false });
|
||||
|
||||
const fullResult = createImportResult("created");
|
||||
resolveImport(fullResult);
|
||||
const succeeded = await waitForImportJobStatusAs(app, accepted.body.statusUrl, "succeeded", {
|
||||
[TEST_USER_HEADER]: "board-user-a",
|
||||
});
|
||||
|
||||
expect(succeeded.status).toBe(200);
|
||||
expect(succeeded.body.job.status).toBe("succeeded");
|
||||
expect(succeeded.body.job.result.companyId).toBe(companyId);
|
||||
// Parity with the synchronous response: the board job carries the full
|
||||
// import result so the import page can run the same activation path.
|
||||
expect(succeeded.body.job.importResult).toEqual(fullResult);
|
||||
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
|
||||
action: "company.imported",
|
||||
companyId,
|
||||
}));
|
||||
});
|
||||
|
||||
it.sequential("hides a board import job from other board users", async () => {
|
||||
mockCompanyPortabilityService.importBundle.mockReturnValueOnce(new Promise(() => undefined));
|
||||
const app = await createBoardApp();
|
||||
|
||||
const accepted = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set("x-paperclip-cloud-async-import", "1")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.send(importRequest);
|
||||
|
||||
expect(accepted.status).toBe(202);
|
||||
|
||||
const crossUser = await request(app).get(accepted.body.statusUrl).set(TEST_USER_HEADER, "board-user-b");
|
||||
expect(crossUser.status).toBe(404);
|
||||
expect(crossUser.body.error).toBe("Import job not found");
|
||||
|
||||
const owner = await request(app).get(accepted.body.statusUrl).set(TEST_USER_HEADER, "board-user-a");
|
||||
expect(owner.status).toBe(200);
|
||||
expect(owner.body.job.status).toBe("running");
|
||||
});
|
||||
|
||||
it.sequential("returns 409 with the running job when a board user resubmits an import", async () => {
|
||||
mockCompanyPortabilityService.importBundle.mockReturnValueOnce(new Promise(() => undefined));
|
||||
const app = await createBoardApp();
|
||||
|
||||
const first = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set("x-paperclip-cloud-async-import", "1")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.send(importRequest);
|
||||
|
||||
expect(first.status).toBe(202);
|
||||
|
||||
const duplicate = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set("x-paperclip-cloud-async-import", "1")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.send(importRequest);
|
||||
|
||||
expect(duplicate.status).toBe(409);
|
||||
expect(duplicate.body.job.id).toBe(first.body.job.id);
|
||||
expect(duplicate.body.statusUrl).toBe(first.body.statusUrl);
|
||||
// The duplicate submit must not start a second import of the same bundle.
|
||||
expect(mockCompanyPortabilityService.importBundle).toHaveBeenCalledTimes(1);
|
||||
|
||||
// A different board user is not blocked by the first user's running job.
|
||||
mockCompanyPortabilityService.importBundle.mockReturnValueOnce(new Promise(() => undefined));
|
||||
const otherUser = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set("x-paperclip-cloud-async-import", "1")
|
||||
.set(TEST_USER_HEADER, "board-user-b")
|
||||
.send(importRequest);
|
||||
|
||||
expect(otherUser.status).toBe(202);
|
||||
expect(otherUser.body.job.id).not.toBe(first.body.job.id);
|
||||
});
|
||||
|
||||
it.sequential(
|
||||
"rejects a concurrent different import without adopting the running job",
|
||||
async () => {
|
||||
mockCompanyPortabilityService.importBundle.mockReturnValueOnce(new Promise(() => undefined));
|
||||
const app = await createBoardApp();
|
||||
|
||||
const first = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set("x-paperclip-cloud-async-import", "1")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.send(importRequest);
|
||||
|
||||
expect(first.status).toBe(202);
|
||||
|
||||
// Same user, a *different* import (another destination) while the first
|
||||
// still runs. It must NOT adopt the first job — adopting would show the
|
||||
// wrong result and switch to the wrong company — so the 409 carries no
|
||||
// job to watch, and no second import starts.
|
||||
const different = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set("x-paperclip-cloud-async-import", "1")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.send({ ...importRequest, target: { mode: "new_company", newCompanyName: "A Different Destination" } });
|
||||
|
||||
expect(different.status).toBe(409);
|
||||
expect(different.body.job).toBeUndefined();
|
||||
expect(different.body.statusUrl).toBeUndefined();
|
||||
expect(different.body.error).toMatch(/different import is already running/i);
|
||||
expect(mockCompanyPortabilityService.importBundle).toHaveBeenCalledTimes(1);
|
||||
},
|
||||
);
|
||||
|
||||
it.sequential("fails a board async import job when the bundle import throws", async () => {
|
||||
mockCompanyPortabilityService.importBundle.mockRejectedValueOnce(new Error("import payload is incomplete"));
|
||||
const app = await createBoardApp();
|
||||
|
||||
const accepted = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set("x-paperclip-cloud-async-import", "1")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.send(importRequest);
|
||||
|
||||
expect(accepted.status).toBe(202);
|
||||
|
||||
const failed = await waitForImportJobStatusAs(app, accepted.body.statusUrl, "failed", {
|
||||
[TEST_USER_HEADER]: "board-user-a",
|
||||
});
|
||||
|
||||
expect(failed.status).toBe(200);
|
||||
expect(failed.body.job.status).toBe("failed");
|
||||
expect(failed.body.job.error.message).toBe("import payload is incomplete");
|
||||
expect(failed.body.job.importResult).toBeUndefined();
|
||||
// A resubmit is allowed once the job is terminal.
|
||||
expect(mockLogActivity).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.sequential("keeps the import job status route board-only", async () => {
|
||||
const app = await createApp({
|
||||
type: "agent",
|
||||
agentId: engineerAgentId,
|
||||
companyId,
|
||||
source: "agent_key",
|
||||
runId: "run-1",
|
||||
});
|
||||
|
||||
const res = await request(app).get("/api/companies/import/jobs/import-does-not-exist");
|
||||
|
||||
expect(res.status).toBe(403);
|
||||
expect(res.body.error).toContain("Board access required");
|
||||
});
|
||||
|
||||
it.sequential("returns 404 for an unknown import job id", async () => {
|
||||
const app = await createBoardApp();
|
||||
|
||||
const res = await request(app)
|
||||
.get("/api/companies/import/jobs/import-missing")
|
||||
.set(TEST_USER_HEADER, "board-user-a");
|
||||
|
||||
expect(res.status).toBe(404);
|
||||
expect(res.body.error).toBe("Import job not found");
|
||||
});
|
||||
|
||||
it.sequential("forwards pauseAutomations from CEO-safe import apply bodies to the portability service", async () => {
|
||||
mockCompanyPortabilityService.importBundle.mockResolvedValueOnce(createImportResult("created"));
|
||||
const app = await createApp({
|
||||
type: "agent",
|
||||
agentId: ceoAgentId,
|
||||
companyId,
|
||||
source: "agent_key",
|
||||
runId: "run-1",
|
||||
});
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/companies/${companyId}/imports/apply`)
|
||||
.send({ ...importRequest, pauseAutomations: true });
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(mockCompanyPortabilityService.importBundle).toHaveBeenCalledWith(
|
||||
{ ...importRequest, pauseAutomations: true },
|
||||
null,
|
||||
{ mode: "agent_safe", sourceCompanyId: companyId, pauseAutomations: true },
|
||||
);
|
||||
});
|
||||
|
||||
it.sequential("imports a company from a multipart zip upload, unzipping into the same inline bundle", async () => {
|
||||
const app = await createBoardApp();
|
||||
const files = { "COMPANY.md": "---\nname: Test\n---\n", "agents/ceo/AGENTS.md": "---\nname: CEO\n---\n" };
|
||||
const zip = buildStoreZip(files, "paperclip");
|
||||
|
||||
const res = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.field("meta", JSON.stringify(importMeta))
|
||||
.attach("package", zip, "paperclip-demo.zip");
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(mockCompanyPortabilityService.importBundle).toHaveBeenCalledTimes(1);
|
||||
const call = mockCompanyPortabilityService.importBundle.mock.calls[0]!;
|
||||
// The uploaded zip is unzipped into the exact inline source the importer
|
||||
// consumes; the other import fields ride along from the JSON meta field.
|
||||
expect(call[0]).toEqual({ ...importMeta, source: { type: "inline", rootPath: "paperclip", files } });
|
||||
expect(call[1]).toBe("board-user-a");
|
||||
expect(call[2]).toEqual({ pauseAutomations: false });
|
||||
});
|
||||
|
||||
it.sequential("previews a company from a multipart zip upload", async () => {
|
||||
const app = await createBoardApp();
|
||||
const files = { "COMPANY.md": "---\nname: Test\n---\n" };
|
||||
const zip = buildStoreZip(files, "paperclip");
|
||||
|
||||
const res = await request(app)
|
||||
.post("/api/companies/import/preview")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.field("meta", JSON.stringify(importMeta))
|
||||
.attach("package", zip, "paperclip-demo.zip");
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(mockCompanyPortabilityService.previewImport).toHaveBeenCalledTimes(1);
|
||||
const call = mockCompanyPortabilityService.previewImport.mock.calls[0]!;
|
||||
expect(call[0]).toEqual({ ...importMeta, source: { type: "inline", rootPath: "paperclip", files } });
|
||||
});
|
||||
|
||||
it.sequential("runs a multipart zip import as an async board job via ?async=1", async () => {
|
||||
let resolveImport: (value: ReturnType<typeof createImportResult>) => void = () => undefined;
|
||||
const pendingImport = new Promise<ReturnType<typeof createImportResult>>((resolve) => {
|
||||
resolveImport = resolve;
|
||||
});
|
||||
mockCompanyPortabilityService.importBundle.mockReturnValueOnce(pendingImport);
|
||||
const app = await createBoardApp();
|
||||
const files = { "COMPANY.md": "---\nname: Test\n---\n" };
|
||||
const zip = buildStoreZip(files, "paperclip");
|
||||
|
||||
const accepted = await request(app)
|
||||
.post("/api/companies/import?async=1")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.field("meta", JSON.stringify(importMeta))
|
||||
.attach("package", zip, "paperclip-demo.zip");
|
||||
|
||||
expect(accepted.status).toBe(202);
|
||||
expect(accepted.body.job.status).toBe("running");
|
||||
expect(accepted.body.statusUrl).toMatch(/^\/api\/companies\/import\/jobs\/import-/);
|
||||
await waitForCondition(
|
||||
() => mockCompanyPortabilityService.importBundle.mock.calls.length === 1,
|
||||
"multipart async import start",
|
||||
);
|
||||
expect(mockCompanyPortabilityService.importBundle.mock.calls[0]![0]).toEqual({
|
||||
...importMeta,
|
||||
source: { type: "inline", rootPath: "paperclip", files },
|
||||
});
|
||||
|
||||
const fullResult = createImportResult("created");
|
||||
resolveImport(fullResult);
|
||||
const succeeded = await waitForImportJobStatusAs(app, accepted.body.statusUrl, "succeeded", {
|
||||
[TEST_USER_HEADER]: "board-user-a",
|
||||
});
|
||||
expect(succeeded.body.job.status).toBe("succeeded");
|
||||
expect(succeeded.body.job.importResult).toEqual(fullResult);
|
||||
});
|
||||
|
||||
it.sequential("engages the async path for board sessions via ?async=1, not the stripped cloud header", async () => {
|
||||
mockCompanyPortabilityService.importBundle.mockReturnValueOnce(new Promise(() => undefined));
|
||||
const app = await createBoardApp();
|
||||
|
||||
// The Cloud harness strips inbound x-paperclip-cloud-* headers, so a browser
|
||||
// can only opt into async with the proxy-safe query parameter.
|
||||
const accepted = await request(app)
|
||||
.post("/api/companies/import?async=1")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.send(importRequest);
|
||||
|
||||
expect(accepted.status).toBe(202);
|
||||
expect(accepted.body.job.status).toBe("running");
|
||||
expect(accepted.body.statusUrl).toMatch(/^\/api\/companies\/import\/jobs\/import-/);
|
||||
});
|
||||
|
||||
it.sequential("keeps a board import synchronous when neither async signal is present", async () => {
|
||||
const app = await createBoardApp();
|
||||
|
||||
const res = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.send(importRequest);
|
||||
|
||||
expect(res.status).toBe(200);
|
||||
expect(res.body.company.id).toBe(companyId);
|
||||
});
|
||||
|
||||
it.sequential("still engages the async path for cloud tenants via the x-paperclip-cloud-async-import header", async () => {
|
||||
mockCompanyPortabilityService.importBundle.mockReturnValueOnce(new Promise(() => undefined));
|
||||
const app = await createApp(cloudTenantActor());
|
||||
|
||||
const accepted = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set("x-paperclip-cloud-async-import", "1")
|
||||
.set(cloudHeaders)
|
||||
.send(importRequest);
|
||||
|
||||
expect(accepted.status).toBe(202);
|
||||
expect(accepted.body.statusUrl).toMatch(/^\/api\/companies\/import\/jobs\/tenant-import-/);
|
||||
});
|
||||
|
||||
it.sequential("rejects a truncated zip upload without importing anything", async () => {
|
||||
const app = await createBoardApp();
|
||||
const zip = buildStoreZip({ "COMPANY.md": "---\nname: Test\n---\n" }, "paperclip");
|
||||
const truncated = zip.subarray(0, 40);
|
||||
|
||||
const res = await request(app)
|
||||
.post("/api/companies/import")
|
||||
.set(TEST_USER_HEADER, "board-user-a")
|
||||
.field("meta", JSON.stringify(importMeta))
|
||||
.attach("package", truncated, "paperclip-demo.zip");
|
||||
|
||||
expect(res.status).toBe(400);
|
||||
expect(mockCompanyPortabilityService.importBundle).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
File diff suppressed because it is too large.
Load diff
@@ -22,6 +22,7 @@ const mockCompanySkillService = vi.hoisted(() => ({
|
||||
starSkill: vi.fn(),
|
||||
unstarSkill: vi.fn(),
|
||||
forkSkill: vi.fn(),
|
||||
renameSkill: vi.fn(),
|
||||
forkPrecheck: vi.fn(),
|
||||
listComments: vi.fn(),
|
||||
createComment: vi.fn(),
|
||||
@@ -315,6 +316,15 @@ describe("company skill mutation permissions", () => {
|
||||
},
|
||||
reassignments: [],
|
||||
});
|
||||
mockCompanySkillService.renameSkill.mockResolvedValue({
|
||||
skill: { ...forkedSkill, id: "skill-1", name: "Ship PR", slug: "ship-pr", key: "company/company-1/ship-pr" },
|
||||
previousName: "Review",
|
||||
previousSlug: "review",
|
||||
previousKey: "company/company-1/review",
|
||||
reassignments: [
|
||||
{ agentId: "11111111-1111-4111-8111-111111111111", previousSkillKey: "company/company-1/review", nextSkillKey: "company/company-1/ship-pr" },
|
||||
],
|
||||
});
|
||||
mockCompanySkillService.forkPrecheck.mockResolvedValue({
|
||||
skillId: "skill-1",
|
||||
original: {
|
||||
@@ -1641,6 +1651,106 @@ describe("company skill mutation permissions", () => {
|
||||
}));
|
||||
});
|
||||
|
||||
it("renames a skill and logs the rename activity", async () => {
|
||||
const app = await createApp({ type: "board", source: "local_implicit", userId: "user-1" });
|
||||
|
||||
const res = await request(app)
|
||||
.post("/api/companies/company-1/skills/skill-1/rename")
|
||||
.send({ name: "Ship PR", slug: "ship-pr" })
|
||||
.expect(200);
|
||||
|
||||
expect(res.body).toMatchObject({
|
||||
skill: { id: "skill-1", name: "Ship PR", slug: "ship-pr", key: "company/company-1/ship-pr" },
|
||||
previousName: "Review",
|
||||
previousSlug: "review",
|
||||
previousKey: "company/company-1/review",
|
||||
});
|
||||
expect(mockCompanySkillService.renameSkill).toHaveBeenCalledWith(
|
||||
"company-1",
|
||||
"skill-1",
|
||||
{ name: "Ship PR", slug: "ship-pr" },
|
||||
);
|
||||
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
|
||||
action: "company.skill_renamed",
|
||||
entityType: "company_skill",
|
||||
entityId: "skill-1",
|
||||
details: expect.objectContaining({
|
||||
previousSlug: "review",
|
||||
slug: "ship-pr",
|
||||
reassignedAgentIds: ["11111111-1111-4111-8111-111111111111"],
|
||||
}),
|
||||
}));
|
||||
});
|
||||
|
||||
it("does not log rename activity for a normalized no-op", async () => {
|
||||
mockCompanySkillService.renameSkill.mockResolvedValueOnce({
|
||||
skill: {
|
||||
id: "skill-1",
|
||||
name: "Review",
|
||||
slug: "review",
|
||||
key: "company/company-1/review",
|
||||
},
|
||||
previousName: "Review",
|
||||
previousSlug: "review",
|
||||
previousKey: "company/company-1/review",
|
||||
reassignments: [],
|
||||
});
|
||||
const app = await createApp({ type: "board", source: "local_implicit", userId: "user-1" });
|
||||
|
||||
await request(app)
|
||||
.post("/api/companies/company-1/skills/skill-1/rename")
|
||||
.send({ name: "Review", slug: "review" })
|
||||
.expect(200);
|
||||
|
||||
expect(mockLogActivity).not.toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
|
||||
action: "company.skill_renamed",
|
||||
}));
|
||||
});
|
||||
|
||||
it("attributes rename activity to the calling agent API key", async () => {
|
||||
const app = await createApp({
|
||||
type: "agent",
|
||||
agentId: "55555555-5555-4555-8555-555555555555",
|
||||
companyId: "company-1",
|
||||
runId: "run-1",
|
||||
keyId: "agent-key-1",
|
||||
source: "agent_key",
|
||||
});
|
||||
|
||||
await request(app)
|
||||
.post("/api/companies/company-1/skills/skill-1/rename")
|
||||
.send({ name: "Ship PR", slug: "ship-pr" })
|
||||
.expect(200);
|
||||
|
||||
expect(mockLogActivity).toHaveBeenCalledWith(expect.anything(), expect.objectContaining({
|
||||
action: "company.skill_renamed",
|
||||
actorType: "agent",
|
||||
agentId: "55555555-5555-4555-8555-555555555555",
|
||||
runId: "run-1",
|
||||
agentApiKeyId: "agent-key-1",
|
||||
}));
|
||||
});
|
||||
|
||||
it("rejects a rename request with a missing name", async () => {
|
||||
const app = await createApp({ type: "board", source: "local_implicit", userId: "user-1" });
|
||||
|
||||
await request(app)
|
||||
.post("/api/companies/company-1/skills/skill-1/rename")
|
||||
.send({ slug: "ship-pr" })
|
||||
.expect(400);
|
||||
expect(mockCompanySkillService.renameSkill).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("rejects a rename request with a multiline name", async () => {
|
||||
const app = await createApp({ type: "board", source: "local_implicit", userId: "user-1" });
|
||||
|
||||
await request(app)
|
||||
.post("/api/companies/company-1/skills/skill-1/rename")
|
||||
.send({ name: "Ship PR\nslug: injected" })
|
||||
.expect(400);
|
||||
expect(mockCompanySkillService.renameSkill).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not synthesize a shared board user id for board actors without user ids", async () => {
|
||||
const app = await createApp({ type: "board", source: "local_implicit" });
|
||||
|
||||
|
||||
@@ -15,6 +15,7 @@ import {
|
||||
projects,
|
||||
projectWorkspaces,
|
||||
} from "@paperclipai/db";
|
||||
import { parseFrontmatterMarkdown } from "@paperclipai/shared";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
@@ -2523,7 +2524,6 @@ describeEmbeddedPostgres("companySkillService.list", () => {
|
||||
const persisted = await db.select().from(companySkills).where(eq(companySkills.companyId, companyId));
|
||||
expect(persisted.filter((skill) => skill.metadata?.sourceKind === "project_scan")).toEqual([]);
|
||||
});
|
||||
|
||||
it("files new project imports without moving them back on re-import", async () => {
|
||||
const companyId = randomUUID();
|
||||
const projectId = randomUUID();
|
||||
@@ -2579,4 +2579,303 @@ describeEmbeddedPostgres("companySkillService.list", () => {
|
||||
});
|
||||
});
|
||||
|
||||
async function seedCompany(companyId: string) {
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
}
|
||||
|
||||
function runtimeSkillName(key: string, slug: string) {
|
||||
if (key.startsWith("paperclipai/paperclip/")) return slug;
|
||||
return `${slug}--${createHash("sha256").update(key).digest("hex").slice(0, 10)}`;
|
||||
}
|
||||
|
||||
it("renames a Paperclip-managed skill, moving the directory and rewriting SKILL.md frontmatter", async () => {
|
||||
const companyId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
const skill = await svc.createLocalSkill(
|
||||
companyId,
|
||||
{ name: "Prepare PR", slug: "prepare-pr", description: "Prep pull requests" },
|
||||
{ type: "user", userId: "board" },
|
||||
);
|
||||
const oldDir = skill.sourceLocator!;
|
||||
const managedRoot = path.dirname(oldDir);
|
||||
|
||||
const result = await svc.renameSkill(companyId, skill.id, { name: "Ship PR", slug: "ship-pr" });
|
||||
|
||||
expect(result).toMatchObject({
|
||||
previousName: "Prepare PR",
|
||||
previousSlug: "prepare-pr",
|
||||
previousKey: `company/${companyId}/prepare-pr`,
|
||||
reassignments: [],
|
||||
skill: {
|
||||
id: skill.id,
|
||||
name: "Ship PR",
|
||||
slug: "ship-pr",
|
||||
key: `company/${companyId}/ship-pr`,
|
||||
},
|
||||
});
|
||||
|
||||
const newDir = path.join(managedRoot, "ship-pr");
|
||||
expect(result.skill.sourceLocator).toBe(newDir);
|
||||
await expect(fs.stat(oldDir)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
const renamedMarkdown = await fs.readFile(path.join(newDir, "SKILL.md"), "utf8");
|
||||
expect(renamedMarkdown).toContain("name: Ship PR");
|
||||
expect(renamedMarkdown).not.toContain("name: Prepare PR");
|
||||
|
||||
// Description and version history are preserved.
|
||||
const persisted = await svc.getById(companyId, skill.id);
|
||||
expect(persisted).toMatchObject({ description: "Prep pull requests" });
|
||||
expect(persisted?.markdown).toBe(renamedMarkdown);
|
||||
const versions = await svc.listVersions(companyId, skill.id);
|
||||
expect(versions).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("quotes YAML-special names while keeping stored and on-disk markdown synchronized", async () => {
|
||||
const companyId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
const skill = await svc.createLocalSkill(companyId, { name: "Prepare PR", slug: "prepare-pr" });
|
||||
|
||||
const result = await svc.renameSkill(companyId, skill.id, { name: "Ship: PR #1", slug: "ship-pr" });
|
||||
const markdown = await fs.readFile(path.join(result.skill.sourceLocator!, "SKILL.md"), "utf8");
|
||||
const persisted = await svc.getById(companyId, skill.id);
|
||||
|
||||
expect(markdown).toContain('name: "Ship: PR #1"');
|
||||
expect(parseFrontmatterMarkdown(markdown).frontmatter.name).toBe("Ship: PR #1");
|
||||
expect(persisted?.markdown).toBe(markdown);
|
||||
});
|
||||
|
||||
it("supports a name-only rename that keeps the slug/key and directory", async () => {
|
||||
const companyId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
const skill = await svc.createLocalSkill(
|
||||
companyId,
|
||||
{ name: "Prepare PR", slug: "prepare-pr" },
|
||||
{ type: "user", userId: "board" },
|
||||
);
|
||||
const dir = skill.sourceLocator!;
|
||||
|
||||
// Keeping the slug requires passing it explicitly: an omitted slug is
|
||||
// re-derived from the new name.
|
||||
const result = await svc.renameSkill(companyId, skill.id, { name: "Prepare Pull Request", slug: "prepare-pr" });
|
||||
expect(result.skill).toMatchObject({
|
||||
name: "Prepare Pull Request",
|
||||
slug: "prepare-pr",
|
||||
key: `company/${companyId}/prepare-pr`,
|
||||
sourceLocator: dir,
|
||||
});
|
||||
const markdown = await fs.readFile(path.join(dir, "SKILL.md"), "utf8");
|
||||
expect(markdown).toContain("name: Prepare Pull Request");
|
||||
});
|
||||
|
||||
it("returns the unchanged skill for a normalized no-op rename", async () => {
|
||||
const companyId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
const skill = await svc.createLocalSkill(
|
||||
companyId,
|
||||
{ name: "Prepare PR", slug: "prepare-pr" },
|
||||
{ type: "user", userId: "board" },
|
||||
);
|
||||
|
||||
const result = await svc.renameSkill(companyId, skill.id, { name: "Prepare PR", slug: "Prepare-PR" });
|
||||
expect(result).toMatchObject({
|
||||
reassignments: [],
|
||||
skill: { id: skill.id, slug: "prepare-pr", key: `company/${companyId}/prepare-pr` },
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects a rename whose slug conflicts with another skill", async () => {
|
||||
const companyId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
const source = await svc.createLocalSkill(companyId, { name: "Source", slug: "source" });
|
||||
await svc.createLocalSkill(companyId, { name: "Taken", slug: "taken" });
|
||||
|
||||
await expect(svc.renameSkill(companyId, source.id, { name: "Taken", slug: "taken" })).rejects.toMatchObject({
|
||||
status: 409,
|
||||
details: { conflict: "slug", slug: "taken" },
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects a rename whose derived key conflicts with another skill", async () => {
|
||||
const companyId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
const source = await svc.createLocalSkill(companyId, { name: "Source", slug: "source" });
|
||||
// A sibling whose key already matches the derived target key but whose slug
|
||||
// differs, so only the key-conflict branch fires. It needs a real on-disk
|
||||
// source so inventory reconciliation does not prune it before the check.
|
||||
const squatterDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-key-squatter-"));
|
||||
cleanupDirs.add(squatterDir);
|
||||
await fs.writeFile(path.join(squatterDir, "SKILL.md"), "---\nname: Key Squatter\n---\n# Key Squatter\n", "utf8");
|
||||
await db.insert(companySkills).values({
|
||||
id: randomUUID(),
|
||||
companyId,
|
||||
key: `company/${companyId}/renamed`,
|
||||
slug: "different-slug",
|
||||
name: "Key Squatter",
|
||||
description: null,
|
||||
markdown: "# Key Squatter\n",
|
||||
sourceType: "local_path",
|
||||
sourceLocator: squatterDir,
|
||||
trustLevel: "markdown_only",
|
||||
compatibility: "compatible",
|
||||
fileInventory: [{ path: "SKILL.md", kind: "skill" }],
|
||||
metadata: { sourceKind: "managed_local" },
|
||||
});
|
||||
|
||||
await expect(svc.renameSkill(companyId, source.id, { name: "Renamed", slug: "renamed" })).rejects.toMatchObject({
|
||||
status: 409,
|
||||
details: { conflict: "key", key: `company/${companyId}/renamed` },
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects renaming non Paperclip-managed skill sources with 422", async () => {
|
||||
const companyId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
|
||||
const unmanagedDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-unmanaged-skill-"));
|
||||
cleanupDirs.add(unmanagedDir);
|
||||
await fs.writeFile(path.join(unmanagedDir, "SKILL.md"), "---\nname: Unmanaged\n---\n# Unmanaged\n", "utf8");
|
||||
|
||||
const rows = [
|
||||
{
|
||||
slug: "github-skill",
|
||||
sourceType: "github" as const,
|
||||
sourceLocator: "https://github.com/acme/github-skill",
|
||||
metadata: { sourceKind: "github", owner: "acme", repo: "github-skill" },
|
||||
},
|
||||
{
|
||||
slug: "url-skill",
|
||||
sourceType: "url" as const,
|
||||
sourceLocator: "https://example.com/url.md",
|
||||
metadata: { sourceKind: "url" },
|
||||
},
|
||||
{
|
||||
slug: "skills-sh-skill",
|
||||
sourceType: "skills_sh" as const,
|
||||
sourceLocator: "https://github.com/acme/skills-sh-skill",
|
||||
metadata: { sourceKind: "skills_sh", owner: "acme", repo: "skills-sh-skill" },
|
||||
},
|
||||
{
|
||||
slug: "catalog-skill",
|
||||
sourceType: "catalog" as const,
|
||||
sourceLocator: null,
|
||||
metadata: { sourceKind: "catalog" },
|
||||
},
|
||||
{
|
||||
slug: "project-scan-skill",
|
||||
sourceType: "local_path" as const,
|
||||
sourceLocator: unmanagedDir,
|
||||
metadata: { sourceKind: "project_scan" },
|
||||
},
|
||||
{
|
||||
slug: "unmanaged-local-skill",
|
||||
sourceType: "local_path" as const,
|
||||
sourceLocator: unmanagedDir,
|
||||
metadata: { sourceKind: "local_path" },
|
||||
},
|
||||
];
|
||||
const inserted = rows.map((row) => ({ id: randomUUID(), ...row }));
|
||||
await db.insert(companySkills).values(inserted.map((row) => ({
|
||||
id: row.id,
|
||||
companyId,
|
||||
key: `company/${companyId}/${row.slug}`,
|
||||
slug: row.slug,
|
||||
name: row.slug,
|
||||
description: null,
|
||||
markdown: `# ${row.slug}\n`,
|
||||
sourceType: row.sourceType,
|
||||
sourceLocator: row.sourceLocator,
|
||||
trustLevel: "markdown_only" as const,
|
||||
compatibility: "compatible" as const,
|
||||
fileInventory: [{ path: "SKILL.md", kind: "skill" as const }],
|
||||
metadata: row.metadata,
|
||||
})));
|
||||
|
||||
for (const row of inserted) {
|
||||
await expect(svc.renameSkill(companyId, row.id, { name: "Renamed" })).rejects.toMatchObject({
|
||||
status: 422,
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("rewrites agent desired-skill keys on rename while preserving version pins", async () => {
|
||||
const companyId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
const skill = await svc.createLocalSkill(companyId, { name: "Shared", slug: "shared" });
|
||||
const pinnedVersionId = randomUUID();
|
||||
|
||||
const pinnedAgentId = randomUUID();
|
||||
const looseAgentId = randomUUID();
|
||||
const otherAgentId = randomUUID();
|
||||
await db.insert(agents).values([
|
||||
{
|
||||
id: pinnedAgentId,
|
||||
companyId,
|
||||
name: "Pinned",
|
||||
role: "engineer",
|
||||
adapterType: "codex_local",
|
||||
adapterConfig: {
|
||||
paperclipSkillSync: {
|
||||
desiredSkills: [{ key: `company/${companyId}/shared`, versionId: pinnedVersionId }],
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
id: looseAgentId,
|
||||
companyId,
|
||||
name: "Loose",
|
||||
role: "engineer",
|
||||
adapterType: "codex_local",
|
||||
adapterConfig: {
|
||||
paperclipSkillSync: { desiredSkills: [`company/${companyId}/shared`] },
|
||||
},
|
||||
},
|
||||
{
|
||||
id: otherAgentId,
|
||||
companyId,
|
||||
name: "Other",
|
||||
role: "engineer",
|
||||
adapterType: "codex_local",
|
||||
adapterConfig: {
|
||||
paperclipSkillSync: { desiredSkills: [`company/${companyId}/unrelated`] },
|
||||
},
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await svc.renameSkill(companyId, skill.id, { name: "Shared Renamed", slug: "shared-renamed" });
|
||||
expect(result.reassignments).toEqual(
|
||||
expect.arrayContaining([
|
||||
{ agentId: pinnedAgentId, previousSkillKey: `company/${companyId}/shared`, nextSkillKey: `company/${companyId}/shared-renamed` },
|
||||
{ agentId: looseAgentId, previousSkillKey: `company/${companyId}/shared`, nextSkillKey: `company/${companyId}/shared-renamed` },
|
||||
]),
|
||||
);
|
||||
expect(result.reassignments).toHaveLength(2);
|
||||
|
||||
const after = await db.select().from(agents).where(eq(agents.companyId, companyId));
|
||||
const pinned = after.find((agent) => agent.id === pinnedAgentId)!.adapterConfig as Record<string, any>;
|
||||
const loose = after.find((agent) => agent.id === looseAgentId)!.adapterConfig as Record<string, any>;
|
||||
const other = after.find((agent) => agent.id === otherAgentId)!.adapterConfig as Record<string, any>;
|
||||
expect(pinned.paperclipSkillSync.desiredSkills).toEqual([
|
||||
{ key: `company/${companyId}/shared-renamed`, versionId: pinnedVersionId },
|
||||
]);
|
||||
expect(loose.paperclipSkillSync.desiredSkills).toEqual([`company/${companyId}/shared-renamed`]);
|
||||
expect(other.paperclipSkillSync.desiredSkills).toEqual([`company/${companyId}/unrelated`]);
|
||||
});
|
||||
|
||||
it("removes the old runtime materialization when the key/slug changes", async () => {
|
||||
const companyId = randomUUID();
|
||||
await seedCompany(companyId);
|
||||
const skill = await svc.createLocalSkill(companyId, { name: "Runtime Skill", slug: "runtime-skill" });
|
||||
const managedRoot = path.dirname(skill.sourceLocator!);
|
||||
const oldRuntimeDir = path.join(managedRoot, "__runtime__", runtimeSkillName(skill.key, skill.slug));
|
||||
await fs.mkdir(oldRuntimeDir, { recursive: true });
|
||||
await fs.writeFile(path.join(oldRuntimeDir, "SKILL.md"), "# stale\n", "utf8");
|
||||
|
||||
await svc.renameSkill(companyId, skill.id, { name: "Runtime Skill", slug: "runtime-renamed" });
|
||||
|
||||
await expect(fs.stat(oldRuntimeDir)).rejects.toMatchObject({ code: "ENOENT" });
|
||||
});
|
||||
});
|
||||
@@ -32,7 +32,7 @@ function writeStub(name: string, body: string) {
|
||||
writeFileSync(path, `#!/bin/sh\n${body}\n`, { mode: 0o755 });
|
||||
}
|
||||
|
||||
function installStubs(ids: { uid: number; gid: number; nodeUid?: number; nodeGid?: number }) {
|
||||
function installStubs(ids: { uid: number; gid: number; nodeUid?: number; nodeGid?: number; homeMismatch?: boolean }) {
|
||||
writeStub(
|
||||
"id",
|
||||
[
|
||||
@@ -46,6 +46,12 @@ function installStubs(ids: { uid: number; gid: number; nodeUid?: number; nodeGid
|
||||
for (const cmd of ["usermod", "groupmod", "chown"]) {
|
||||
writeStub(cmd, `echo "${cmd} $*" >> "${logFile}"`);
|
||||
}
|
||||
// The entrypoint's ownership probe is a first-mismatch find over the app
|
||||
// home. An empty result models a fully node-owned tree (image-baked dir,
|
||||
// healthy volume); a path models any uid OR gid mismatch anywhere in the
|
||||
// tree (fresh root-owned mount, root-owned descendant, stale group after
|
||||
// a GID-only remap).
|
||||
writeStub("find", ids.homeMismatch ? `echo "$1/mismatched-entry"` : `true`);
|
||||
writeStub("gosu", `echo "gosu $*" >> "${logFile}"\nshift\nexec "$@"`);
|
||||
}
|
||||
|
||||
@@ -79,17 +85,60 @@ describe("docker-entrypoint.sh", () => {
|
||||
});
|
||||
|
||||
it("remaps the node user and chowns /paperclip before gosu when root requests a different UID/GID", async () => {
|
||||
installStubs({ uid: 0, gid: 0 });
|
||||
// The stubbed node uid stays 1000 while the stat probe reports the old
|
||||
// ownership, modelling the post-remap mismatch that must trigger chown.
|
||||
installStubs({ uid: 0, gid: 0, homeMismatch: true });
|
||||
|
||||
const { stdout, calls } = await runEntrypoint({ USER_UID: "1001", USER_GID: "1001" });
|
||||
const { stdout, calls } = await runEntrypoint({ USER_UID: "1001", USER_GID: "1001", PAPERCLIP_HOME: stubDir });
|
||||
|
||||
expect(stdout).toContain("ENTRYPOINT-CMD-RAN");
|
||||
expect(calls).toContain("usermod -o -u 1001 node");
|
||||
expect(calls).toContain("groupmod -o -g 1001 node");
|
||||
expect(calls).toContain("chown -R node:node /paperclip");
|
||||
expect(calls).toContain(`chown -R node:node ${stubDir}`);
|
||||
expect(calls).toContain("gosu node echo ENTRYPOINT-CMD-RAN");
|
||||
});
|
||||
|
||||
it("chowns a root-owned home before gosu even with the default UID/GID (fresh volume mount)", async () => {
|
||||
// A freshly mounted volume arrives root-owned and shadows the image's
|
||||
// build-time chown; with no remap requested the old entrypoint dropped
|
||||
// privileges onto an unwritable home and the server crashed on its
|
||||
// first mkdir.
|
||||
installStubs({ uid: 0, gid: 0, homeMismatch: true });
|
||||
|
||||
const { stdout, calls } = await runEntrypoint({ PAPERCLIP_HOME: stubDir });
|
||||
|
||||
expect(stdout).toContain("ENTRYPOINT-CMD-RAN");
|
||||
expect(calls).toContain(`chown -R node:node ${stubDir}`);
|
||||
expect(calls).not.toContain("usermod");
|
||||
expect(calls).toContain("gosu node echo ENTRYPOINT-CMD-RAN");
|
||||
});
|
||||
|
||||
it("repairs ownership on a GID-only remap (stale group on persisted descendants)", async () => {
|
||||
installStubs({ uid: 0, gid: 0, homeMismatch: true });
|
||||
|
||||
const { calls } = await runEntrypoint({ USER_GID: "1001", PAPERCLIP_HOME: stubDir });
|
||||
|
||||
expect(calls).toContain("groupmod -o -g 1001 node");
|
||||
expect(calls).toContain(`chown -R node:node ${stubDir}`);
|
||||
});
|
||||
|
||||
it("keeps a fully node-owned tree chown-free (no per-boot recursive chown)", async () => {
|
||||
installStubs({ uid: 0, gid: 0, homeMismatch: false });
|
||||
|
||||
const { calls } = await runEntrypoint({ PAPERCLIP_HOME: stubDir });
|
||||
|
||||
expect(calls).not.toContain("chown");
|
||||
expect(calls).toContain("gosu node echo ENTRYPOINT-CMD-RAN");
|
||||
});
|
||||
|
||||
it("honours PAPERCLIP_HOME for the ownership probe", async () => {
|
||||
installStubs({ uid: 0, gid: 0, homeMismatch: true });
|
||||
|
||||
const { calls } = await runEntrypoint({ PAPERCLIP_HOME: stubDir });
|
||||
|
||||
expect(calls).toContain(`chown -R node:node ${stubDir}`);
|
||||
});
|
||||
|
||||
it("execs directly and silently when already running as the requested user (restricted PodSecurity)", async () => {
|
||||
installStubs({ uid: 1000, gid: 1000 });
|
||||
|
||||
|
||||
@@ -176,6 +176,106 @@ describe("resolveEnvironmentExecutionTarget", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("resolves sandbox targets for every remote-managed adapter, including grok_local", async () => {
|
||||
for (const adapterType of [
|
||||
"claude_local",
|
||||
"codex_local",
|
||||
"cursor",
|
||||
"gemini_local",
|
||||
"grok_local",
|
||||
"opencode_local",
|
||||
"pi_local",
|
||||
]) {
|
||||
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
|
||||
driver: "sandbox",
|
||||
config: {
|
||||
provider: "fake-plugin",
|
||||
reuseLease: false,
|
||||
timeoutMs: 30_000,
|
||||
},
|
||||
});
|
||||
|
||||
const target = await resolveEnvironmentExecutionTarget({
|
||||
db: {} as never,
|
||||
companyId: "company-1",
|
||||
adapterType,
|
||||
environment: {
|
||||
id: "env-1",
|
||||
driver: "sandbox",
|
||||
config: { provider: "fake-plugin" },
|
||||
},
|
||||
leaseId: "lease-1",
|
||||
leaseMetadata: {},
|
||||
lease: null,
|
||||
environmentRuntime: null,
|
||||
});
|
||||
|
||||
expect(target, `adapter ${adapterType}`).toMatchObject({
|
||||
kind: "remote",
|
||||
transport: "sandbox",
|
||||
providerKey: "fake-plugin",
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("returns null for adapters without remote-managed environment support", async () => {
|
||||
for (const driver of ["sandbox", "ssh"] as const) {
|
||||
const target = await resolveEnvironmentExecutionTarget({
|
||||
db: {} as never,
|
||||
companyId: "company-1",
|
||||
adapterType: "process",
|
||||
environment: {
|
||||
id: "env-1",
|
||||
driver,
|
||||
config: { provider: "fake-plugin" },
|
||||
},
|
||||
leaseId: "lease-1",
|
||||
leaseMetadata: {},
|
||||
lease: null,
|
||||
environmentRuntime: null,
|
||||
});
|
||||
|
||||
expect(target, `driver ${driver}`).toBeNull();
|
||||
}
|
||||
expect(mockResolveEnvironmentDriverConfigForRuntime).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("resolves SSH execution targets for grok_local", async () => {
|
||||
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
|
||||
driver: "ssh",
|
||||
config: {
|
||||
host: "ssh.example.test",
|
||||
port: 22,
|
||||
username: "paperclip",
|
||||
remoteWorkspacePath: "/srv/paperclip",
|
||||
privateKey: "PRIVATE KEY",
|
||||
knownHosts: "[ssh.example.test]:22 ssh-ed25519 AAAA",
|
||||
strictHostKeyChecking: true,
|
||||
},
|
||||
});
|
||||
|
||||
const target = await resolveEnvironmentExecutionTarget({
|
||||
db: {} as never,
|
||||
companyId: "company-1",
|
||||
adapterType: "grok_local",
|
||||
environment: {
|
||||
id: "env-ssh-1",
|
||||
driver: "ssh",
|
||||
config: {},
|
||||
},
|
||||
leaseId: "lease-ssh-1",
|
||||
leaseMetadata: {},
|
||||
lease: null,
|
||||
environmentRuntime: null,
|
||||
});
|
||||
|
||||
expect(target).toMatchObject({
|
||||
kind: "remote",
|
||||
transport: "ssh",
|
||||
remoteCwd: "/srv/paperclip",
|
||||
});
|
||||
});
|
||||
|
||||
it("resolves SSH execution targets in bridge mode", async () => {
|
||||
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
|
||||
driver: "ssh",
|
||||
@@ -320,4 +420,180 @@ describe("resolveEnvironmentExecutionTarget", () => {
|
||||
expect(runner!.providerExecMs()).toBe(0);
|
||||
expect(runner!.providerGetMs()).toBe(0);
|
||||
});
|
||||
|
||||
// A recording tracer that captures each provider-exec span's name, attribute
|
||||
// map, and end. It satisfies the structural tracer the seam calls.
|
||||
function createRecordingExecTracer() {
|
||||
const spans: Array<{ name: string; attributes: Record<string, unknown>; ended: boolean }> = [];
|
||||
const tracer = {
|
||||
startSpan(name: string) {
|
||||
const span = {
|
||||
name,
|
||||
attributes: {} as Record<string, unknown>,
|
||||
ended: false,
|
||||
setAttribute(key: string, value: unknown) {
|
||||
span.attributes[key] = value;
|
||||
},
|
||||
end() {
|
||||
span.ended = true;
|
||||
},
|
||||
};
|
||||
spans.push(span);
|
||||
return span;
|
||||
},
|
||||
};
|
||||
return { tracer, spans };
|
||||
}
|
||||
|
||||
// The closed span-attribute allowlist for a provider-exec span (Phase 4).
|
||||
const ALLOWED_EXEC_SPAN_ATTRIBUTE_KEYS = new Set([
|
||||
"provider",
|
||||
"exit",
|
||||
"provider.exec.duration_ms",
|
||||
"provider.get.duration_ms",
|
||||
]);
|
||||
|
||||
async function runnerFor(input: {
|
||||
provider: string;
|
||||
execResult: Record<string, unknown>;
|
||||
tracer: unknown;
|
||||
}) {
|
||||
mockResolveEnvironmentDriverConfigForRuntime.mockResolvedValue({
|
||||
driver: "sandbox",
|
||||
config: { provider: input.provider, reuseLease: false, timeoutMs: 30_000 },
|
||||
});
|
||||
const environmentRuntime = {
|
||||
execute: vi.fn().mockResolvedValue(input.execResult),
|
||||
supportsSync: vi.fn().mockReturnValue(false),
|
||||
};
|
||||
const target = await resolveEnvironmentExecutionTarget({
|
||||
db: {} as never,
|
||||
companyId: "company-1",
|
||||
adapterType: "codex_local",
|
||||
environment: { id: "env-1", driver: "sandbox", config: { provider: input.provider } },
|
||||
leaseId: "lease-1",
|
||||
leaseMetadata: { remoteCwd: "/workspace" },
|
||||
lease: { id: "lease-1" } as never,
|
||||
environmentRuntime: environmentRuntime as never,
|
||||
tracer: input.tracer as never,
|
||||
});
|
||||
return (target as { runner?: {
|
||||
execute(input: { command: string; args?: string[] }): Promise<unknown>;
|
||||
} }).runner!;
|
||||
}
|
||||
|
||||
it("sets the provider duration attributes from finite Daytona-shaped metadata", async () => {
|
||||
const { tracer, spans } = createRecordingExecTracer();
|
||||
const runner = await runnerFor({
|
||||
provider: "daytona",
|
||||
execResult: {
|
||||
exitCode: 0,
|
||||
signal: null,
|
||||
timedOut: false,
|
||||
stdout: "ok",
|
||||
stderr: "",
|
||||
metadata: { durationMs: 600, getDurationMs: 15 },
|
||||
},
|
||||
tracer,
|
||||
});
|
||||
|
||||
await runner.execute({ command: "echo", args: ["a"] });
|
||||
|
||||
expect(spans).toHaveLength(1);
|
||||
const span = spans[0]!;
|
||||
expect(span.name).toBe("provider.execute");
|
||||
expect(span.ended).toBe(true);
|
||||
expect(span.attributes["provider.exec.duration_ms"]).toBe(600);
|
||||
expect(span.attributes["provider.get.duration_ms"]).toBe(15);
|
||||
expect(span.attributes.provider).toBe("daytona");
|
||||
expect(span.attributes.exit).toBe("ok");
|
||||
});
|
||||
|
||||
it("omits each duration attribute when a provider returns no timing (does not throw, keeps provider)", async () => {
|
||||
const { tracer, spans } = createRecordingExecTracer();
|
||||
const runner = await runnerFor({
|
||||
provider: "kubernetes",
|
||||
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
|
||||
tracer,
|
||||
});
|
||||
|
||||
await expect(runner.execute({ command: "echo" })).resolves.toBeTruthy();
|
||||
|
||||
expect(spans).toHaveLength(1);
|
||||
const span = spans[0]!;
|
||||
expect("provider.exec.duration_ms" in span.attributes).toBe(false);
|
||||
expect("provider.get.duration_ms" in span.attributes).toBe(false);
|
||||
// The provider attribute is always present so a trace shows which provider ran.
|
||||
expect(span.attributes.provider).toBe("kubernetes");
|
||||
});
|
||||
|
||||
it("never emits a `0` duration attribute for a Daytona timeout that omits durationMs", async () => {
|
||||
const { tracer, spans } = createRecordingExecTracer();
|
||||
const runner = await runnerFor({
|
||||
provider: "daytona",
|
||||
execResult: {
|
||||
exitCode: 124,
|
||||
signal: null,
|
||||
timedOut: true,
|
||||
stdout: "",
|
||||
stderr: "",
|
||||
// The Daytona timeout branch may leave durationMs undefined.
|
||||
metadata: { getDurationMs: 20 },
|
||||
},
|
||||
tracer,
|
||||
});
|
||||
|
||||
await runner.execute({ command: "sleep", args: ["999"] });
|
||||
|
||||
const span = spans[0]!;
|
||||
expect("provider.exec.duration_ms" in span.attributes).toBe(false);
|
||||
expect(span.attributes["provider.get.duration_ms"]).toBe(20);
|
||||
expect(span.attributes.exit).toBe("error");
|
||||
});
|
||||
|
||||
it("never sets a command, arg, or non-allowlisted key as an indexed span attribute", async () => {
|
||||
const { tracer, spans } = createRecordingExecTracer();
|
||||
const runner = await runnerFor({
|
||||
provider: "daytona",
|
||||
execResult: {
|
||||
exitCode: 0,
|
||||
signal: null,
|
||||
timedOut: false,
|
||||
stdout: "",
|
||||
stderr: "",
|
||||
metadata: { durationMs: 5, getDurationMs: 1 },
|
||||
},
|
||||
tracer,
|
||||
});
|
||||
|
||||
await runner.execute({ command: "bash -lc 'rm -rf /secret/path'", args: ["--token", "s3cr3t"] });
|
||||
|
||||
const span = spans[0]!;
|
||||
for (const key of Object.keys(span.attributes)) {
|
||||
expect(ALLOWED_EXEC_SPAN_ATTRIBUTE_KEYS.has(key), `non-allowlisted key "${key}"`).toBe(true);
|
||||
}
|
||||
const values = Object.values(span.attributes).map(String);
|
||||
expect(values.some((value) => value.includes("rm -rf"))).toBe(false);
|
||||
expect(values.some((value) => value.includes("s3cr3t"))).toBe(false);
|
||||
});
|
||||
|
||||
it("normalizes a plugin-backed provider key to `plugin` and keeps a built-in family as-is", async () => {
|
||||
const plugin = createRecordingExecTracer();
|
||||
const pluginRunner = await runnerFor({
|
||||
provider: "acme-custom-sandbox",
|
||||
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
|
||||
tracer: plugin.tracer,
|
||||
});
|
||||
await pluginRunner.execute({ command: "echo" });
|
||||
expect(plugin.spans[0]!.attributes.provider).toBe("plugin");
|
||||
|
||||
const builtIn = createRecordingExecTracer();
|
||||
const builtInRunner = await runnerFor({
|
||||
provider: "e2b",
|
||||
execResult: { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "" },
|
||||
tracer: builtIn.tracer,
|
||||
});
|
||||
await builtInRunner.execute({ command: "echo" });
|
||||
expect(builtIn.spans[0]!.attributes.provider).toBe("e2b");
|
||||
});
|
||||
});
|
||||
Loaded 100 of 205 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user