Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-25 17:45:53 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge origin/master into feat/managed-install-integration

Browse Source 
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-07-31 18:53:11 +00:00
2 parents 54f10dba44 c4f62644b0
commit d641914741
417 files changed
+41155 -7991
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.agents/skills/create-paperclip-bundled-skill/SKILL.md
+265
View File
@@ -0,0 +1,265 @@
---
name: create-paperclip-bundled-skill
description: >
Turn an idea, tweet, or task into a skill in the Paperclip skills catalog
(packages/skills-catalog). Use when asked to FIND or MAKE a skill and publish
it as a bundled/optional catalog skill: research prior art, reference or
author it, add examples, regenerate the manifest, open a PR.
---
# Create a Paperclip Bundled Skill
Take source material — a tweet, a task description, a blog post, "make a skill
that does X" — and land it as a skill in the Paperclip skills catalog
(`packages/skills-catalog/`), delivered as a reviewed PR. The catalog is the
shelf every Paperclip company browses and installs from, so the bar is: correct
metadata, useful instructions, worked examples, and a clean validation run.
The core rule is **FIND before MAKE**: if a good skill already exists (in the
catalog, in this repo, or published on GitHub), reference or adapt it instead
of writing a duplicate from scratch.
## When to use
- A human sends a tweet/link/idea and asks for it to become a Paperclip skill.
- A task asks to bundle an existing repo skill into the catalog.
- A task asks to add an external published skill to the catalog.
## When not to use
- The skill is company-private (belongs in that company's library via the
Skills UI/API, not the shipped catalog).
- You only need a repo-internal agent skill for working on Paperclip itself —
that goes in `.agents/skills/` or `skills/`, with no catalog machinery.
## Step 0 — Capture the source material
Understand exactly what the skill should teach before writing anything.
**Tweets / X links.** Use the `xc` CLI (X API client). Paperclip engineering
agent environments ship it preinstalled and pre-authenticated; it is not a
tool you install or mint credentials for yourself. Check availability before
relying on it:
```sh
command -v xc && xc whoami # on PATH and authenticated? if not, use the fallback below
```
```sh
xc get <post-url-or-id> --json # the post itself (conversation_id, author)
xc search 'conversation_id:<id>' --archive --json # rest of the thread (>7 days old needs --archive)
xc user <username> # author context
xc search '<topic keywords>' -n 30 # related discussion
```
If `xc` is not on PATH, is unauthenticated, or the account lacks read access
(the check above fails for any reason), delegate the
fetch to a teammate with X/Twitter access (e.g. the Content Strategist agent)
via a child issue: give them the URL and ask for full text of the post + thread
+ any linked content.
**Other sources.** Fetch linked articles/READMEs directly. Record the source
URL — it goes in the skill body or PR description as attribution.
Distill: what is the repeatable procedure? What inputs does it take? What does
"done" look like? If the source is just an aspiration ("agents should write
better commit messages"), you are authoring the procedure yourself — say so in
the PR.
## Step 1 — FIND: search for an existing skill
Search in this order; stop when you have a clear winner.
1. **Already in the catalog?** Avoid duplicates (duplicate slugs fail the
build):
```sh
grep -i '<topic>' packages/skills-catalog/generated/catalog.json
ls packages/skills-catalog/catalog/{bundled,optional}/*/
```
2. **Already in this repo?** Check `.agents/skills/`, `skills/`, and issue
history (`gh search issues` / Paperclip board) for prior work on the topic.
3. **Published on GitHub?** Skills are conventionally a directory with a
`SKILL.md`:
```sh
gh search code --filename SKILL.md "<topic>" --limit 20
gh search repos "<topic> skill" --limit 20
```
Also check known collections (e.g. `anthropics/skills`) and do a web search
for `<topic> agent skill SKILL.md`.
Judge candidates by: does the SKILL.md actually contain the procedure (not a
stub)? Is it maintained? What does it bundle (scripts raise the trust level)?
Is the license compatible with redistribution? Then pick a path:
- **Good external skill exists** → add it as an **external reference**
(Step 2A). It stays attributed to and pinned at the upstream repo.
- **Partial match** → author a local skill (Step 2B) that adapts the idea;
credit the source with a link in the SKILL.md body.
- **Nothing usable** → author a new local skill (Step 2B).
## Step 2 — Choose kind, category, and slug
- **kind**: default to `optional`. Use `bundled` only when the skill should
ship to every Paperclip company by default — that needs explicit human/board
direction, not your judgment call.
- **category**: reuse an existing directory when one fits (`browser`,
`content`, `docs`, `finance`, `paperclip-operations`, `product`, `quality`,
`research`, `software-development`). New categories are allowed but must be
lowercase kebab-case slugs.
- **slug**: lowercase kebab-case (`^[a-z0-9]+(-[a-z0-9]+)*$`), unique across
the whole catalog (both kinds).
The skill lives at
`packages/skills-catalog/catalog/<kind>/<category>/<slug>/` and its canonical
key is `paperclipai/<kind>/<category>/<slug>`.
## Step 2A — External reference path (`catalog-ref.json`)
The directory contains **only** `catalog-ref.json` (a directory with both
`catalog-ref.json` and `SKILL.md` fails the build). The manifest builder
fetches the pinned files from GitHub at build time and inventories them.
```sh
# Pin the exact commit for the chosen ref (tag or branch)
gh api repos/<owner>/<repo>/commits/<ref> --jq .sha
```
```json
{
"source": {
"type": "github",
"hostname": "github.com",
"owner": "<owner>",
"repo": "<repo>",
"ref": "<tag-or-branch>",
"commit": "<40-char sha from above>",
"path": "<dir inside the repo containing SKILL.md, or ''>"
},
"files": ["SKILL.md", "references/**", "scripts/run.py"],
"defaultInstall": false,
"recommendedForRoles": ["researcher"],
"requires": ["python3"],
"tags": ["topic", "keywords"]
}
```
Rules the builder enforces:
- `files` entries are exact relative paths or `dir/**` globs; `SKILL.md` must
be included and must have frontmatter with `name` and `description`.
- If the upstream frontmatter declares `key`/`slug`, they must match the
catalog placement — otherwise pick a matching slug or use the local path.
- `commit` must be a full 40-hex SHA; every listed file must be ≤ 1 MiB.
- `recommendedForRoles`, `requires`, `tags` live in the JSON (there is no
local SKILL.md to carry them).
See `catalog/optional/research/last30days/catalog-ref.json` for the live
example, and `examples/external-reference.md` next to this skill.
## Step 2B — Author a local catalog skill
Layout:
```
catalog/<kind>/<category>/<slug>/
├── SKILL.md # required entrypoint
├── examples/ # 1–2 worked examples (Step 3)
├── references/ # optional deep-dive docs
├── scripts/ # optional — raises trust level, avoid unless needed
└── assets/ # optional templates/images
```
`SKILL.md` frontmatter (all validated by the builder):
```markdown
---
name: <slug>
description: >
40–300 chars. Routing logic, not marketing: what it does, when to use it,
when not to.
key: paperclipai/<kind>/<category>/<slug>
recommendedForRoles:
- engineer # non-empty; used for staffing suggestions
tags:
- topic # non-empty; used for browse/search
---
```
Optional frontmatter: `defaultInstall: true` (only for skills every new
company should get), `requires: [node, python3, ...]` for runtime deps.
Body: follow `docs/guides/agent-developer/writing-a-skill.md` — "When to use"
/ "When not to use" sections, concrete commands over prose, supporting detail
in `references/`. If the skill came from a tweet or external source, link it
in the body for attribution.
Trust level is derived from files, not declared: any `scripts/` file makes the
skill `scripts_executables` (install becomes audit-gated and you must extend
the `scriptBearing` expectation in `src/shipped-catalog.test.ts`); `assets/`
or non-markdown files make it `assets`; markdown-only skills stay
`markdown_only`. Prefer markdown-only.
## Step 3 — Write 1–2 worked examples
Create `examples/` inside the skill directory with one or two markdown files,
each a complete input → application → output walkthrough (realistic input, the
skill's steps applied, the finished artifact). These ship with the skill so
installers can judge it before running it, and they keep the trust level at
`markdown_only` because they are `.md` files.
Name them by scenario, e.g. `examples/rewrite-release-note.md`.
## Step 4 — Regenerate the manifest and update tests
Never hand-edit `generated/catalog.json`; it is deterministic build output.
```sh
pnpm --filter @paperclipai/skills-catalog build:manifest # regenerates generated/catalog.json
pnpm --filter @paperclipai/skills-catalog validate # must report no errors
```
(External references need network access to GitHub during these steps.)
Then update `packages/skills-catalog/src/shipped-catalog.test.ts`:
- add the new key to `EXPECTED_BUNDLED_KEYS` or `EXPECTED_OPTIONAL_KEYS`
(alphabetical order);
- if the skill bears scripts, add it to the `scriptBearing` expectation.
```sh
pnpm --filter @paperclipai/skills-catalog test
```
The test suite also enforces the ≤300-char frontmatter description budget
across the repo and the ≥40-char description / non-empty roles+tags rules for
every catalog skill.
## Step 5 — Open the PR
Follow the `prepare-paperclip-pr` skill (`.agents/skills/prepare-paperclip-pr/`)
against `paperclipai/paperclip` master. The diff should contain exactly:
1. the new skill directory (SKILL.md + examples/ + supporting files, **or**
catalog-ref.json),
2. the regenerated `generated/catalog.json`,
3. the `shipped-catalog.test.ts` expectation update.
In the PR body: link the source material (tweet URL, upstream repo), state
whether this is a new skill / adaptation / external reference, and note the
trust level. Reference PR #10410 (simplified-english) as the shape of a
minimal optional-skill PR.
## Gotchas
- `generated/catalog.json` staleness is a validation error — always rerun
`build:manifest` after any file change inside the skill directory (the
inventory carries per-file sha256 hashes).
- Duplicate `slug` across bundled *and* optional fails the build, not just
duplicate keys.
- Symlinks inside a skill directory must resolve within it; directory
symlinks are rejected — copy files in.
- The `bundled` kind and `defaultInstall` are independent axes; don't set
`defaultInstall: true` casually on optional skills.
- For external references the builder fetches from GitHub on every manifest
build; a moved/deleted upstream breaks the build, which is why `commit` is
pinned — prefer upstream tags for `ref`.
.agents/skills/create-paperclip-bundled-skill/examples/external-reference.md
+84
View File
@@ -0,0 +1,84 @@
# Example — FIND path: tweet → existing skill → external reference
Real artifact: `packages/skills-catalog/catalog/optional/research/last30days/`.
## Input
Dotta sends a tweet praising a "last 30 days" research workflow that sweeps
Reddit/X/YouTube for what changed recently on a topic.
## Step 0 — Capture
```sh
xc get https://x.com/<author>/status/<id> --json # post text + conversation_id
xc search 'conversation_id:<id>' --archive --json # the rest of the thread
```
The thread links a GitHub repo: `mvanhorn/last30days-skill`, which already
contains a proper skill (`skills/last30days/SKILL.md` plus scripts and
references).
## Step 1 — FIND
- Not in the catalog, not in this repo.
- The upstream repo IS the skill — maintained, tagged releases, real SKILL.md.
- Verdict: **FIND** — add it as an external reference, keep attribution and
updates upstream.
## Step 2A — catalog-ref.json
Placement: `optional` / `research` / `last30days`. Pin the release tag to an
exact commit:
```sh
gh api repos/mvanhorn/last30days-skill/commits/v3.3.0 --jq .sha
# → daca71f89eb71d0d56d01a43ed7627aa919dba4f
```
`catalog/optional/research/last30days/catalog-ref.json` (the only file in the
directory):
```json
{
"source": {
"type": "github",
"hostname": "github.com",
"owner": "mvanhorn",
"repo": "last30days-skill",
"ref": "v3.3.0",
"commit": "daca71f89eb71d0d56d01a43ed7627aa919dba4f",
"path": "skills/last30days"
},
"files": [
"SKILL.md",
"agents/openai.yaml",
"references/**",
"scripts/briefing.py",
"scripts/compare.sh",
"scripts/last30days.py",
"scripts/lib/**",
"scripts/setup-keychain.sh",
"scripts/store.py",
"scripts/watchlist.py"
],
"defaultInstall": false,
"recommendedForRoles": ["researcher", "marketer", "product-manager", "analyst"],
"requires": ["node", "python3"],
"tags": ["research", "last-30-days", "social-media", "trends", "citations", "reddit", "x", "youtube"]
}
```
Metadata (`recommendedForRoles`, `requires`, `tags`) lives in the JSON because
there is no local SKILL.md to carry it.
## Steps 4–5 — Manifest, tests, PR
- `pnpm --filter @paperclipai/skills-catalog build:manifest` fetches the
pinned files from GitHub and inventories them (network required).
- The skill bundles `scripts/`, so trust level derives to
`scripts_executables` → it must also be added to the `scriptBearing`
expectation in `src/shipped-catalog.test.ts`, alongside
`EXPECTED_OPTIONAL_KEYS`.
- PR diff: `catalog-ref.json`, regenerated `generated/catalog.json`, test
expectations. PR body links both the tweet and the upstream repo, and calls
out the elevated trust level so review is deliberate.
.agents/skills/create-paperclip-bundled-skill/examples/new-local-skill.md
+90
View File
@@ -0,0 +1,90 @@
# Example — MAKE path: idea → new optional catalog skill
Real run: PAP-15684 → PR #10410 (`feat(skills-catalog): add optional
/simplified-english skill`).
## Input
Task from Dotta: "make a skill that has agents write user-facing text in
Simplified English." No tweet this time — the source is a known public
specification (ASD-STE100 Simplified Technical English).
## Step 1 — FIND
- `grep -i 'simplified\|plain.english' packages/skills-catalog/generated/catalog.json`
→ no hits; nothing in `.agents/skills/` or `skills/` either.
- `gh search code --filename SKILL.md "simplified technical english"` → no
usable published skill (only STE checker tools, no SKILL.md procedure).
- Verdict: **MAKE** a new local skill.
## Step 2 — Placement
- kind: `optional` (useful, but not something every company must ship with).
- category: `content` (existing category, fits writing/communication).
- slug: `simplified-english`.
- Path: `packages/skills-catalog/catalog/optional/content/simplified-english/`.
## Step 2B — Authoring
One markdown-only `SKILL.md` (trust level stays `markdown_only`):
```markdown
---
name: simplified-english
description: Write user-facing comments, plans, and documents in ASD-STE100 Simplified Technical English — short, unambiguous sentences with approved words and one meaning each — so readers understand them the first time.
key: paperclipai/optional/content/simplified-english
recommendedForRoles:
- engineer
- product
- writer
- devrel
tags:
- writing
- communication
- clarity
- style
---
# Simplified English
For user-facing comments, plans, and documents, write using only ASD-STE100
Simplified Technical English.
## Core rules
- Use short sentences (procedures ≤ 20 words, descriptions ≤ 25 words).
- Give one instruction per sentence.
- Use approved words with one meaning each; avoid synonyms and jargon.
...
```
Note the frontmatter hits every builder rule: description is 40–300 chars,
`key` matches the placement, roles and tags are non-empty.
## Steps 3–4 — Examples, manifest, tests
- Add `examples/rewrite-status-comment.md` showing a jargon-heavy status
comment rewritten under the rules (before/after).
- `pnpm --filter @paperclipai/skills-catalog build:manifest` → regenerates
`generated/catalog.json` with the new entry.
- Add `"paperclipai/optional/content/simplified-english"` to
`EXPECTED_OPTIONAL_KEYS` in `src/shipped-catalog.test.ts` (alphabetical).
- `pnpm --filter @paperclipai/skills-catalog test` → green.
## Step 5 — PR
Four-part diff:
```
packages/skills-catalog/catalog/optional/content/simplified-english/SKILL.md
packages/skills-catalog/catalog/optional/content/simplified-english/examples/rewrite-status-comment.md
packages/skills-catalog/generated/catalog.json
packages/skills-catalog/src/shipped-catalog.test.ts
```
(The historical PR #10410 predates this skill and shipped as a three-part
diff without the `examples/` file; a run that follows this skill includes the
worked example from Step 3 in the same PR.)
PR body links the ASD-STE100 spec as the source and states trust level
`markdown_only`.
.agents/skills/garden-inbox/SKILL.md
+98
View File
@@ -0,0 +1,98 @@
---
name: garden-inbox
description: Scan a Paperclip user's Mine inbox, classify reversible archive candidates, request checkbox confirmation, and archive only accepted selections. Use when asked to garden, clean up, prune, or tidy a Paperclip inbox without changing issues, branches, or workspaces.
---
# Garden Inbox
Use the bundled script for every stage. Keep the workflow strictly ordered: `scan` → `confirm` → `apply`.
## Safety contract
- Treat `scan` as read-only. It may read inbox, workspace, close-readiness, and local Git metadata only.
- Treat `confirm` as a confirmation-card write only. It must not archive inbox entries or mutate issue fields, branches, or workspaces.
- Run `apply` only after a resolved `request_checkbox_confirmation` interaction. It archives only accepted option IDs that also occur in the originating `candidates.json`.
- Remember that inbox archive state is per-user presentation state. It is reversible and does not change the underlying issue.
- Never substitute issue status changes, branch deletion, workspace cleanup, or issue deletion for inbox archiving.
## Inputs
Require `PAPERCLIP_API_URL` and `PAPERCLIP_API_KEY`. The script strips a trailing `/api` from the URL. It resolves the target user from the run JWT payload's `responsible_user_id`; use `--user-id <uuid>` only for an explicit target override.
Use a run-owned output directory when available:
```bash
RUN_DIR="${PAPERCLIP_RUN_SCRATCH_DIR:-${PAPERCLIP_TASK_SCRATCH_DIR:-.}}/garden-inbox"
mkdir -p "$RUN_DIR"
```
## 1. Scan
```bash
node .agents/skills/garden-inbox/scripts/garden-inbox.mjs scan \
--output-dir "$RUN_DIR" \
--stale-days 60
```
Inspect `garden-inbox-report.md` and `candidates.json`. The report groups every inbox row into exactly one bucket:
The scan reads Mine separately for each included issue status so the endpoint's global 500-row cap does not silently omit older rows. If any single-status query reaches that cap, both outputs mark coverage as possibly truncated; do not treat that scan as complete.
- A: merged/archived workspace and all linked work terminal; selected by default.
- B: terminal or workspace-gone work idle beyond the threshold; selected by default.
- C: stale work with commits ahead of base; never selected by default.
- D: keep; never offered for archiving.
Do not manually promote bucket D entries into the candidate file.
## 2. Confirm
Post checkbox interactions on the driving issue:
```bash
node .agents/skills/garden-inbox/scripts/garden-inbox.mjs confirm \
--issue-id "$PAPERCLIP_TASK_ID" \
--candidates "$RUN_DIR/candidates.json"
```
The script posts sequential cards when a scan has more than 200 candidates. Re-running `confirm` with the same scan file is idempotent. Leave the driving issue in the waiting posture required by the surrounding Paperclip heartbeat workflow.
For development or payload review, suppress the POST:
```bash
node .agents/skills/garden-inbox/scripts/garden-inbox.mjs confirm \
--issue-id "$PAPERCLIP_TASK_ID" \
--candidates "$RUN_DIR/candidates.json" \
--dry-run
```
## 3. Apply accepted selections
After an interaction-resolution wake, take the resolved interaction ID from the wake payload and run:
```bash
node .agents/skills/garden-inbox/scripts/garden-inbox.mjs apply \
--issue-id "$PAPERCLIP_TASK_ID" \
--interaction-id "$INTERACTION_ID" \
--candidates "$RUN_DIR/candidates.json"
```
On rejection, expiry, or an accepted empty selection, the script archives nothing. Apply preserves the scan file's target user, including an explicit `--user-id` override. Its summary includes the API undo path and target-user body for every archived row.
Test `apply` without API writes by supplying a saved interaction response:
```bash
node .agents/skills/garden-inbox/scripts/garden-inbox.mjs apply \
--issue-id "$PAPERCLIP_TASK_ID" \
--interaction-file resolved-interaction.json \
--candidates "$RUN_DIR/candidates.json" \
--dry-run
```
## Verify the bundled logic
Run the zero-dependency Node tests after changing classification or selection safety:
```bash
node --test .agents/skills/garden-inbox/scripts/garden-inbox.test.mjs
```
.agents/skills/garden-inbox/scripts/garden-inbox.mjs
Executable
+699
View File
@@ -0,0 +1,699 @@
#!/usr/bin/env node
import { createHash } from "node:crypto";
import { execFileSync } from "node:child_process";
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { dirname, resolve } from "node:path";
import { pathToFileURL } from "node:url";
const TERMINAL_STATUSES = new Set(["done", "cancelled"]);
const OFFERED_BUCKETS = new Set(["A", "B", "C"]);
const DEFAULT_STALE_DAYS = 60;
const INTERACTION_LIMIT = 200;
const INBOX_MINE_STATUSES = ["backlog", "todo", "in_progress", "in_review", "blocked", "done"];
const INBOX_QUERY_CAP = 500;
class ApiError extends Error {
constructor(status, message, body = null) {
super(`${status}: ${message}`);
this.name = "ApiError";
this.status = status;
this.body = body;
}
}
function usage() {
return `Usage:
garden-inbox.mjs [scan] [--user-id UUID] [--stale-days 60] [--output-dir DIR]
garden-inbox.mjs confirm [--issue-id ID] [--candidates FILE] [--dry-run]
garden-inbox.mjs apply [--issue-id ID] (--interaction-id ID | --interaction-file FILE) [--candidates FILE] [--dry-run]
Common environment:
PAPERCLIP_API_URL, PAPERCLIP_API_KEY, PAPERCLIP_RUN_ID, PAPERCLIP_TASK_ID`;
}
function parseArgs(argv) {
const args = [...argv];
let command = "scan";
if (args[0] && !args[0].startsWith("-")) command = args.shift();
if (!["scan", "confirm", "apply"].includes(command)) throw new Error(`Unknown command: ${command}`);
const options = {};
while (args.length > 0) {
const token = args.shift();
if (token === "--help" || token === "-h") {
options.help = true;
continue;
}
if (!token.startsWith("--")) throw new Error(`Unexpected argument: ${token}`);
const key = token.slice(2).replaceAll("-", "_");
if (key === "dry_run") {
options[key] = true;
continue;
}
const value = args.shift();
if (value === undefined || value.startsWith("--")) throw new Error(`${token} requires a value`);
if (options[key] === undefined) options[key] = value;
else options[key] = Array.isArray(options[key]) ? [...options[key], value] : [options[key], value];
}
return { command, options };
}
function required(value, name) {
if (typeof value !== "string" || value.trim() === "") throw new Error(`${name} is required`);
return value.trim();
}
function normalizeApiBase(value) {
let base = required(value, "PAPERCLIP_API_URL").replace(/\/+$/, "");
if (base.endsWith("/api")) base = base.slice(0, -4);
return base;
}
function runtime(options, { requireApi = true } = {}) {
const apiUrl = options.api_url ?? process.env.PAPERCLIP_API_URL;
const apiKey = options.api_key ?? process.env.PAPERCLIP_API_KEY;
return {
apiBase: requireApi ? normalizeApiBase(apiUrl) : (apiUrl ? normalizeApiBase(apiUrl) : null),
apiKey: requireApi ? required(apiKey, "PAPERCLIP_API_KEY") : apiKey ?? null,
runId: options.run_id ?? process.env.PAPERCLIP_RUN_ID ?? null,
};
}
async function apiRequest(context, path, { method = "GET", body } = {}) {
const headers = { Accept: "application/json", Authorization: `Bearer ${context.apiKey}` };
if (body !== undefined) headers["Content-Type"] = "application/json";
if (method !== "GET" && context.runId) headers["X-Paperclip-Run-Id"] = context.runId;
const response = await fetch(`${context.apiBase}/api${path}`, {
method,
headers,
body: body === undefined ? undefined : JSON.stringify(body),
});
const text = await response.text();
let parsed = null;
if (text) {
try { parsed = JSON.parse(text); } catch { parsed = text; }
}
if (!response.ok) {
const message = parsed && typeof parsed === "object"
? parsed.error ?? parsed.message ?? response.statusText
: parsed ?? response.statusText;
throw new ApiError(response.status, String(message), parsed);
}
return parsed;
}
function decodeJwtPayload(token) {
const parts = required(token, "PAPERCLIP_API_KEY").split(".");
if (parts.length < 2) throw new Error("PAPERCLIP_API_KEY is not a JWT; pass --user-id explicitly");
try {
return JSON.parse(Buffer.from(parts[1], "base64url").toString("utf8"));
} catch {
throw new Error("Could not decode the PAPERCLIP_API_KEY JWT payload; pass --user-id explicitly");
}
}
function resolveUserId(options, apiKey) {
if (options.user_id) return required(options.user_id, "--user-id");
const userId = decodeJwtPayload(apiKey).responsible_user_id;
if (typeof userId !== "string" || userId.trim() === "") {
throw new Error("Run JWT has no responsible_user_id; pass --user-id explicitly");
}
return userId.trim();
}
function parsePositiveInteger(value, fallback, name) {
if (value === undefined) return fallback;
const parsed = Number(value);
if (!Number.isInteger(parsed) || parsed < 1 || parsed > 3650) {
throw new Error(`${name} must be an integer from 1 to 3650`);
}
return parsed;
}
function validDate(value) {
if (!value) return null;
const date = new Date(value);
return Number.isNaN(date.getTime()) ? null : date;
}
function latestDate(...values) {
return values.flat().map(validDate).filter(Boolean).sort((left, right) => right.getTime() - left.getTime())[0] ?? null;
}
function runGit(args, cwd) {
try {
return execFileSync("git", ["-C", cwd, ...args], {
encoding: "utf8",
stdio: ["ignore", "pipe", "ignore"],
timeout: 5_000,
}).trim();
} catch {
return "";
}
}
function gitLastCommitAt(branchName, roots) {
if (!branchName) return null;
for (const root of [...new Set(roots.filter(Boolean))]) {
const parsed = validDate(runGit(["log", "-1", "--format=%cI", branchName], root));
if (parsed) return parsed;
}
return null;
}
function findLocalBranch(identifier, roots) {
if (!identifier) return null;
const needle = identifier.toLowerCase();
for (const root of [...new Set(roots.filter(Boolean))]) {
const branches = runGit(["for-each-ref", "--format=%(refname:short)", "refs/heads"], root)
.split("\n").map((branch) => branch.trim()).filter(Boolean)
.filter((branch) => branch.toLowerCase().includes(needle));
if (branches.length === 1) return branches[0];
}
return null;
}
async function mapLimit(values, limit, mapper) {
const results = new Array(values.length);
let cursor = 0;
async function worker() {
while (cursor < values.length) {
const index = cursor++;
results[index] = await mapper(values[index], index);
}
}
await Promise.all(Array.from({ length: Math.min(limit, values.length) }, worker));
return results;
}
async function fetchMineInboxRows(context, userId) {
const statusResults = await Promise.all(INBOX_MINE_STATUSES.map(async (status) => {
const query = new URLSearchParams({ userId, status });
const rows = await apiRequest(context, `/agents/me/inbox/mine?${query}`);
if (!Array.isArray(rows)) throw new Error(`Inbox API returned a non-array response for status ${status}`);
return { status, rows };
}));
const rowsById = new Map();
const statusCounts = {};
const cappedStatuses = [];
let duplicateCount = 0;
for (const { status, rows } of statusResults) {
statusCounts[status] = rows.length;
if (rows.length >= INBOX_QUERY_CAP) cappedStatuses.push(status);
for (const issue of rows) {
if (rowsById.has(issue.id)) duplicateCount += 1;
rowsById.set(issue.id, issue);
}
}
return {
rows: [...rowsById.values()],
coverage: {
strategy: "per_status",
queryCap: INBOX_QUERY_CAP,
statusCounts,
cappedStatuses,
duplicateCount,
complete: cappedStatuses.length === 0,
},
};
}
async function inspectWorkspace(context, workspaceId, issueIdentifiers, gitRoot) {
let workspace = null;
let readiness = null;
let gone = false;
let error = null;
try {
workspace = await apiRequest(context, `/execution-workspaces/${encodeURIComponent(workspaceId)}`);
} catch (caught) {
if (caught instanceof ApiError && caught.status === 404) gone = true;
else error = caught.message;
}
if (workspace) {
try {
readiness = await apiRequest(context, `/execution-workspaces/${encodeURIComponent(workspaceId)}/close-readiness`);
} catch (caught) {
error = error ?? caught.message;
}
}
const roots = [workspace?.cwd, workspace?.providerRef, readiness?.git?.repoRoot, readiness?.git?.workspacePath, gitRoot];
let branchName = workspace?.branchName ?? readiness?.git?.branchName ?? null;
if (!branchName && gone) {
const matches = issueIdentifiers.map((identifier) => findLocalBranch(identifier, roots)).filter(Boolean);
if (new Set(matches).size === 1) branchName = matches[0];
}
const branchCommitAt = gitLastCommitAt(branchName, roots);
return {
id: workspaceId,
gone,
error,
status: workspace?.status ?? null,
branchName,
lastUsedAt: workspace?.lastUsedAt ?? null,
branchCommitAt: branchCommitAt?.toISOString() ?? null,
readiness,
};
}
function reason(code, message, facts = {}) {
return { code, message, facts };
}
function classify(issue, workspaceInfo, staleDays, now) {
const terminal = TERMINAL_STATUSES.has(issue.status);
const workspaceArchived = workspaceInfo?.status === "archived";
const workspaceGone = Boolean(workspaceInfo?.gone);
const readiness = workspaceInfo?.readiness ?? null;
const linkedIssues = Array.isArray(readiness?.linkedIssues) ? readiness.linkedIssues : [];
const allLinkedIssuesTerminal = Boolean(readiness) && linkedIssues.every((linked) => linked.isTerminal === true);
const mergedIntoBase = readiness?.git?.isMergedIntoBase === true;
const aheadCount = Number.isInteger(readiness?.git?.aheadCount) ? readiness.git.aheadCount : null;
const blockerAttention = issue.blockerAttention?.state;
const hasOpenBlockers = issue.status === "blocked"
|| (Array.isArray(issue.blockedBy) && issue.blockedBy.some((blocker) => !TERMINAL_STATUSES.has(blocker.status)))
|| (typeof blockerAttention === "string" && blockerAttention !== "none");
const awaitingUser = issue.blockedInboxAttention?.state === "awaiting_decision"
|| (issue.status === "in_review" && Boolean(issue.assigneeUserId));
const issueActivity = latestDate(issue.lastActivityAt, issue.updatedAt, issue.createdAt);
const branchActivity = latestDate(workspaceInfo?.branchCommitAt, workspaceInfo?.lastUsedAt);
const lastActivity = latestDate(issueActivity, branchActivity) ?? new Date(0);
const cutoff = new Date(now.getTime() - staleDays * 86_400_000);
const stale = lastActivity.getTime() <= cutoff.getTime();
const facts = {
issueStatus: issue.status,
workspaceStatus: workspaceInfo?.status ?? null,
workspaceGone,
workspaceInspectionError: workspaceInfo?.error ?? null,
mergedIntoBase,
allLinkedIssuesTerminal,
aheadCount,
staleDays,
};
if (workspaceInfo?.error) {
return {
bucket: "D",
reason: reason("workspace_inspection_failed", "Keep: workspace archive safety could not be verified.", facts),
lastActivity,
};
}
if (hasOpenBlockers) return { bucket: "D", reason: reason("open_blockers", "Keep: the issue has unresolved blocker or liveness attention.", facts), lastActivity };
if (awaitingUser) return { bucket: "D", reason: reason("pending_user_action", "Keep: the issue is awaiting a user decision or review.", facts), lastActivity };
if (!terminal) {
const code = issue.status === "in_review" ? "pending_interaction_or_review" : "non_terminal_status";
return { bucket: "D", reason: reason(code, `Keep: issue status ${issue.status} is not terminal.`, facts), lastActivity };
}
if ((mergedIntoBase || workspaceArchived) && allLinkedIssuesTerminal) {
const code = workspaceArchived ? "workspace_archived_finished" : "merged_finished";
return { bucket: "A", reason: reason(code, "Archive candidate: work is merged or archived and every linked issue is terminal.", facts), lastActivity };
}
if (!stale) return { bucket: "D", reason: reason("recent_activity", `Keep: activity is newer than ${staleDays} days.`, facts), lastActivity };
if (aheadCount !== null && aheadCount > 0) {
return { bucket: "C", reason: reason("stale_unmerged_commits", `Review manually: stale branch is ${aheadCount} commit(s) ahead of base.`, facts), lastActivity };
}
if (terminal || workspaceGone) {
return { bucket: "B", reason: reason("stale_terminal_or_workspace_gone", `Archive candidate: no issue or branch activity for at least ${staleDays} days and the issue is terminal or workspace is gone.`, facts), lastActivity };
}
return { bucket: "D", reason: reason("insufficient_archive_evidence", "Keep: archive safety conditions were not met.", facts), lastActivity };
}
function candidateFrom(issue, workspace, classification) {
return {
issueId: issue.id,
identifier: issue.identifier ?? null,
title: issue.title,
bucket: classification.bucket,
reason: classification.reason,
executionWorkspaceId: issue.executionWorkspaceId ?? null,
workspaceStatus: workspace?.status ?? null,
workspaceGone: Boolean(workspace?.gone),
branchName: workspace?.branchName ?? null,
lastActivityAt: classification.lastActivity.toISOString(),
};
}
function markdownEscape(value) {
return String(value ?? "").replaceAll("|", "\\|").replaceAll("\n", " ");
}
function renderReport(scanResult) {
const labels = {
A: "A — merged and finished (default checked)",
B: "B — stale (default checked)",
C: "C — stale but unmerged (default unchecked)",
D: "D — keep (not offered)",
};
const lines = [
"# Garden inbox scan", "",
`- Generated: ${scanResult.generatedAt}`,
`- Target user: \`${scanResult.userId}\``,
`- Stale threshold: ${scanResult.staleDays} days`,
`- Inbox rows classified: ${scanResult.items.length}`,
`- Archive candidates: ${scanResult.candidates.length}`,
`- Scan coverage: ${scanResult.coverage.complete ? "complete across per-status queries" : "possibly truncated"}`,
];
if (!scanResult.coverage.complete) {
lines.push(
"",
"> [!WARNING]",
`> Coverage may be incomplete: ${scanResult.coverage.cappedStatuses.map((status) => `\`${status}\``).join(", ")} returned at least ${scanResult.coverage.queryCap} rows, the Mine endpoint cap.`,
);
}
for (const bucket of ["A", "B", "C", "D"]) {
const items = scanResult.items.filter((item) => item.bucket === bucket);
lines.push("", `## ${labels[bucket]} (${items.length})`, "");
if (items.length === 0) {
lines.push("_None._");
continue;
}
lines.push("| Issue | Title | Reason | Workspace / branch | Last activity |", "|---|---|---|---|---|");
for (const item of items) {
const issue = item.identifier ? `\`${markdownEscape(item.identifier)}\`` : `\`${item.issueId}\``;
const workspace = [item.workspaceStatus, item.branchName].filter(Boolean).join(" / ") || (item.workspaceGone ? "gone" : "none");
lines.push(`| ${issue} | ${markdownEscape(item.title)} | \`${item.reason.code}\` — ${markdownEscape(item.reason.message)} | ${markdownEscape(workspace)} | ${item.lastActivityAt} |`);
}
}
lines.push("");
return lines.join("\n");
}
function writeText(path, contents) {
const target = resolve(path);
mkdirSync(dirname(target), { recursive: true });
writeFileSync(target, contents);
return target;
}
function readJson(path) {
return JSON.parse(readFileSync(resolve(path), "utf8"));
}
function stableScanId(scanResult) {
const fingerprint = {
userId: scanResult.userId,
staleDays: scanResult.staleDays,
generatedAt: scanResult.generatedAt,
candidates: scanResult.candidates.map((item) => ({
issueId: item.issueId,
bucket: item.bucket,
reason: item.reason.code,
lastActivityAt: item.lastActivityAt,
})),
};
return createHash("sha256").update(JSON.stringify(fingerprint)).digest("hex").slice(0, 24);
}
async function scan(options) {
const context = runtime(options);
const userId = resolveUserId(options, context.apiKey);
const staleDays = parsePositiveInteger(options.stale_days, DEFAULT_STALE_DAYS, "--stale-days");
const now = options.now ? validDate(options.now) : new Date();
if (!now) throw new Error("--now must be an ISO date");
const outputDir = options.output_dir ?? ".";
const candidatesPath = options.candidates ?? resolve(outputDir, "candidates.json");
const reportPath = options.report ?? resolve(outputDir, "garden-inbox-report.md");
const { rows, coverage } = await fetchMineInboxRows(context, userId);
const issuesByWorkspace = new Map();
for (const issue of rows) {
if (!issue.executionWorkspaceId) continue;
const identifiers = issuesByWorkspace.get(issue.executionWorkspaceId) ?? [];
if (issue.identifier) identifiers.push(issue.identifier);
issuesByWorkspace.set(issue.executionWorkspaceId, identifiers);
}
const workspaceIds = [...issuesByWorkspace.keys()];
const inspected = await mapLimit(workspaceIds, 8, (workspaceId) => inspectWorkspace(
context, workspaceId, issuesByWorkspace.get(workspaceId), options.git_root ?? process.cwd(),
));
const workspaceById = new Map(inspected.map((entry) => [entry.id, entry]));
const items = rows.map((issue) => {
const workspace = issue.executionWorkspaceId ? workspaceById.get(issue.executionWorkspaceId) ?? null : null;
return candidateFrom(issue, workspace, classify(issue, workspace, staleDays, now));
});
const scanResult = {
schemaVersion: 1,
generatedAt: now.toISOString(),
userId,
staleDays,
sourceCount: rows.length,
coverage,
items,
candidates: items.filter((item) => OFFERED_BUCKETS.has(item.bucket)),
};
scanResult.scanId = stableScanId(scanResult);
const output = {
schemaVersion: 1,
scanId: scanResult.scanId,
generatedAt: scanResult.generatedAt,
userId,
staleDays,
sourceCount: rows.length,
coverage,
candidates: scanResult.candidates,
kept: items.filter((item) => item.bucket === "D"),
};
const candidatesFile = writeText(candidatesPath, `${JSON.stringify(output, null, 2)}\n`);
const report = renderReport(scanResult);
const reportFile = writeText(reportPath, report);
process.stdout.write(`${report}\nCandidates JSON: ${candidatesFile}\nReport: ${reportFile}\n`);
return output;
}
function truncate(value, limit) {
const text = String(value ?? "").trim();
if (text.length <= limit) return text;
return `${text.slice(0, Math.max(0, limit - 1)).trimEnd()}…`;
}
function candidateFile(options) {
const data = readJson(options.candidates ?? "candidates.json");
if (data?.schemaVersion !== 1
|| typeof data.scanId !== "string"
|| typeof data.userId !== "string"
|| data.userId.trim() === ""
|| !Array.isArray(data.candidates)) {
throw new Error("Candidates file is not a garden-inbox schemaVersion 1 scan");
}
for (const item of data.candidates) {
if (!item?.issueId || !OFFERED_BUCKETS.has(item.bucket)) {
throw new Error("Candidates file contains an invalid or non-offered item");
}
}
return data;
}
function archiveTargetBody(scanData) {
return { userId: scanData.userId };
}
function chunk(values, size) {
const chunks = [];
for (let index = 0; index < values.length; index += size) chunks.push(values.slice(index, index + size));
return chunks;
}
function confirmationBody(scanData, candidates, index, count) {
const options = candidates.map((candidate) => ({
id: candidate.issueId,
label: truncate(`${candidate.identifier ?? candidate.issueId} — ${candidate.title}`, 120),
description: truncate(`${candidate.reason.message} Last activity: ${candidate.lastActivityAt}.`, 500),
}));
const part = count > 1 ? ` (${index + 1}/${count})` : "";
return {
kind: "request_checkbox_confirmation",
idempotencyKey: `garden-inbox:${scanData.scanId}:${index + 1}:${count}`,
title: `Confirm inbox archive candidates${part}`,
summary: `Choose which reversible inbox entries to archive${part}.`,
continuationPolicy: "wake_assignee",
payload: {
version: 1,
prompt: `Select the inbox entries to archive${part}. Unchecked entries will remain visible.`,
options,
defaultSelectedOptionIds: candidates
.filter((candidate) => candidate.bucket === "A" || candidate.bucket === "B")
.map((candidate) => candidate.issueId),
minSelected: 0,
acceptLabel: "Archive selected",
rejectLabel: "Keep everything",
detailsMarkdown: `Scan \`${scanData.scanId}\` used a ${scanData.staleDays}-day stale threshold. Buckets A and B start checked; stale branches with unmerged commits (C) start unchecked. Archiving changes only this user's inbox visibility and is reversible.`,
},
};
}
async function confirm(options) {
const data = candidateFile(options);
const drivingIssueId = options.issue_id ?? process.env.PAPERCLIP_TASK_ID;
if (!options.dry_run) required(drivingIssueId, "--issue-id or PAPERCLIP_TASK_ID");
const groups = chunk(data.candidates, INTERACTION_LIMIT);
if (groups.length === 0) {
process.stdout.write("No archive candidates; no confirmation interaction created.\n");
return [];
}
const bodies = groups.map((candidates, index) => confirmationBody(data, candidates, index, groups.length));
if (options.dry_run) {
process.stdout.write(`${JSON.stringify({ dryRun: true, issueId: drivingIssueId ?? null, interactions: bodies }, null, 2)}\n`);
return bodies;
}
const context = runtime(options);
const created = [];
for (const body of bodies) {
created.push(await apiRequest(context, `/issues/${encodeURIComponent(drivingIssueId)}/interactions`, {
method: "POST",
body,
}));
}
process.stdout.write(`${JSON.stringify({
scanId: data.scanId,
createdInteractions: created.map((item) => ({
id: item.id,
status: item.status,
idempotencyKey: item.idempotencyKey,
})),
}, null, 2)}\n`);
return created;
}
function flattenInteractionFile(value) {
if (Array.isArray(value)) return value;
if (Array.isArray(value?.interactions)) return value.interactions;
if (value?.interaction && typeof value.interaction === "object") return [value.interaction];
return value && typeof value === "object" ? [value] : [];
}
function optionValues(value) {
if (value === undefined) return [];
return Array.isArray(value) ? value : [value];
}
async function resolveInteractions(options, context, drivingIssueId) {
const files = optionValues(options.interaction_file);
const ids = optionValues(options.interaction_id);
const fromFiles = files.flatMap((file) => flattenInteractionFile(readJson(file)));
if (ids.length === 0) return fromFiles;
required(drivingIssueId, "--issue-id or PAPERCLIP_TASK_ID");
const live = await apiRequest(context, `/issues/${encodeURIComponent(drivingIssueId)}/interactions`);
if (!Array.isArray(live)) throw new Error("Interactions API returned a non-array response");
const byId = new Map(live.map((interaction) => [interaction.id, interaction]));
for (const id of ids) {
if (!byId.has(id)) throw new Error(`Interaction ${id} was not found on the driving issue`);
fromFiles.push(byId.get(id));
}
return fromFiles;
}
function acceptedCandidates(interaction, scanData, candidateById) {
if (interaction?.kind !== "request_checkbox_confirmation") {
throw new Error(`Interaction ${interaction?.id ?? "(unknown)"} is not request_checkbox_confirmation`);
}
if (typeof interaction.idempotencyKey !== "string"
|| !interaction.idempotencyKey.startsWith(`garden-inbox:${scanData.scanId}:`)) {
throw new Error(`Interaction ${interaction.id ?? "(unknown)"} does not belong to scan ${scanData.scanId}`);
}
if (interaction.status !== "accepted" || interaction.result?.outcome !== "accepted") return [];
const selected = interaction.result?.selectedOptionIds ?? [];
if (!Array.isArray(selected)) throw new Error("Accepted interaction has invalid selectedOptionIds");
const optionIds = new Set((interaction.payload?.options ?? []).map((option) => option.id));
const seen = new Set();
return selected.map((issueId) => {
if (seen.has(issueId)) throw new Error(`Interaction selected duplicate option id ${issueId}`);
seen.add(issueId);
if (!optionIds.has(issueId)) throw new Error(`Selected id ${issueId} was not an option in the interaction`);
const candidate = candidateById.get(issueId);
if (!candidate || !OFFERED_BUCKETS.has(candidate.bucket)) {
throw new Error(`Selected id ${issueId} is not an offered candidate in the originating scan`);
}
return candidate;
});
}
function renderApplySummary(results, interactions, dryRun, userId) {
const lines = [
`# Garden inbox ${dryRun ? "dry-run " : ""}apply summary`, "",
`- Interactions inspected: ${interactions.length}`,
`- Entries ${dryRun ? "that would be archived" : "archived"}: ${results.filter((result) => result.ok).length}`,
`- Failures: ${results.filter((result) => !result.ok).length}`,
];
if (results.length === 0) {
lines.push("", "Nothing was archived: the interaction was rejected, unresolved, expired, or accepted with no selections.");
}
for (const result of results) {
lines.push("", `- ${result.ok ? "OK" : "FAILED"}: ${result.identifier ?? result.issueId} — ${result.message}`);
if (result.ok) {
lines.push(` Undo: \`DELETE /api/issues/${result.issueId}/inbox-archive\` with body \`${JSON.stringify({ userId })}\`, or use **Unarchive** in the issue properties pane.`);
}
}
return `${lines.join("\n")}\n`;
}
async function applyAccepted(options) {
const data = candidateFile(options);
const drivingIssueId = options.issue_id ?? process.env.PAPERCLIP_TASK_ID ?? null;
const needsLiveApi = optionValues(options.interaction_id).length > 0 || !options.dry_run;
const context = runtime(options, { requireApi: needsLiveApi });
const interactions = await resolveInteractions(options, context, drivingIssueId);
if (interactions.length === 0) throw new Error("Provide --interaction-id or --interaction-file");
const candidateById = new Map(data.candidates.map((candidate) => [candidate.issueId, candidate]));
const accepted = [];
const seen = new Set();
for (const interaction of interactions) {
for (const candidate of acceptedCandidates(interaction, data, candidateById)) {
if (!seen.has(candidate.issueId)) accepted.push(candidate);
seen.add(candidate.issueId);
}
}
const results = [];
for (const candidate of accepted) {
if (options.dry_run) {
results.push({ ...candidate, ok: true, message: "would archive this accepted inbox entry" });
continue;
}
try {
await apiRequest(context, `/issues/${encodeURIComponent(candidate.issueId)}/inbox-archive`, {
method: "POST",
body: archiveTargetBody(data),
});
results.push({ ...candidate, ok: true, message: "archived accepted inbox entry" });
} catch (error) {
results.push({ ...candidate, ok: false, message: error.message });
}
}
const summary = renderApplySummary(results, interactions, Boolean(options.dry_run), data.userId);
process.stdout.write(summary);
if (options.summary) writeText(options.summary, summary);
if (results.some((result) => !result.ok)) process.exitCode = 1;
return results;
}
export {
acceptedCandidates,
archiveTargetBody,
classify,
confirm,
decodeJwtPayload,
fetchMineInboxRows,
normalizeApiBase,
scan,
};
async function main() {
const { command, options } = parseArgs(process.argv.slice(2));
if (options.help) {
process.stdout.write(`${usage()}\n`);
return;
}
if (command === "scan") await scan(options);
else if (command === "confirm") await confirm(options);
else await applyAccepted(options);
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
main().catch((error) => {
process.stderr.write(`garden-inbox: ${error.message}\n`);
process.exitCode = 1;
});
}
.agents/skills/garden-inbox/scripts/garden-inbox.test.mjs
+157
View File
@@ -0,0 +1,157 @@
import assert from "node:assert/strict";
import test from "node:test";
import {
acceptedCandidates,
archiveTargetBody,
classify,
decodeJwtPayload,
fetchMineInboxRows,
normalizeApiBase,
} from "./garden-inbox.mjs";
const now = new Date("2026-07-29T00:00:00.000Z");
function issue(overrides = {}) {
return {
id: "issue-1",
identifier: "PAP-1",
title: "Example",
status: "done",
updatedAt: "2025-01-01T00:00:00.000Z",
blockedBy: [],
...overrides,
};
}
function workspace(overrides = {}) {
return {
gone: false,
error: null,
status: "active",
branchCommitAt: "2025-01-01T00:00:00.000Z",
readiness: {
git: { isMergedIntoBase: false, aheadCount: 0 },
linkedIssues: [{ isTerminal: true }],
},
...overrides,
};
}
test("classifies each archive and keep condition into one bucket", () => {
const merged = workspace({
readiness: {
git: { isMergedIntoBase: true, aheadCount: 0 },
linkedIssues: [{ isTerminal: true }],
},
});
assert.equal(classify(issue(), merged, 60, now).bucket, "A");
assert.equal(classify(issue(), workspace(), 60, now).bucket, "B");
assert.equal(classify(issue(), workspace({
readiness: {
git: { isMergedIntoBase: false, aheadCount: 2 },
linkedIssues: [{ isTerminal: true }],
},
}), 60, now).bucket, "C");
assert.equal(classify(issue({ status: "in_progress" }), workspace(), 60, now).bucket, "D");
});
test("keeps candidates when workspace safety inspection fails", () => {
const result = classify(issue(), workspace({ error: "503 Service Unavailable", readiness: null }), 60, now);
assert.equal(result.bucket, "D");
assert.equal(result.reason.code, "workspace_inspection_failed");
});
test("returns only accepted options from the originating scan", () => {
const candidate = { issueId: "issue-1", bucket: "A" };
const scan = { scanId: "scan-1" };
const interaction = {
id: "interaction-1",
kind: "request_checkbox_confirmation",
idempotencyKey: "garden-inbox:scan-1:1:1",
status: "accepted",
payload: { options: [{ id: "issue-1" }] },
result: { outcome: "accepted", selectedOptionIds: ["issue-1"] },
};
assert.deepEqual(acceptedCandidates(interaction, scan, new Map([[candidate.issueId, candidate]])), [candidate]);
});
test("preserves an overridden scan user for archive and undo requests", () => {
assert.deepEqual(archiveTargetBody({ userId: "target-user" }), { userId: "target-user" });
});
test("rejects selected ids that were not offered", () => {
const scan = { scanId: "scan-1" };
const interaction = {
id: "interaction-1",
kind: "request_checkbox_confirmation",
idempotencyKey: "garden-inbox:scan-1:1:1",
status: "accepted",
payload: { options: [{ id: "issue-1" }] },
result: { outcome: "accepted", selectedOptionIds: ["issue-2"] },
};
assert.throws(
() => acceptedCandidates(interaction, scan, new Map()),
/was not an option in the interaction/,
);
});
test("decodes the responsible user and normalizes API URLs locally", () => {
const payload = Buffer.from(JSON.stringify({ responsible_user_id: "user-1" })).toString("base64url");
assert.equal(decodeJwtPayload(`header.${payload}.signature`).responsible_user_id, "user-1");
assert.equal(normalizeApiBase("https://paperclip.example/api/"), "https://paperclip.example");
});
test("scans Mine once per status and merges rows by issue id", async (t) => {
const requestedStatuses = [];
t.mock.method(globalThis, "fetch", async (url) => {
const parsed = new URL(url);
const status = parsed.searchParams.get("status");
requestedStatuses.push(status);
const rows = status === "todo"
? [issue({ id: "shared", status: "todo" })]
: status === "done"
? [issue({ id: "shared", status: "done" }), issue({ id: "done-only" })]
: [];
return new Response(JSON.stringify(rows), { status: 200 });
});
const result = await fetchMineInboxRows({
apiBase: "https://paperclip.example",
apiKey: "secret",
}, "user-1");
assert.deepEqual(requestedStatuses, ["backlog", "todo", "in_progress", "in_review", "blocked", "done"]);
assert.equal(result.rows.length, 2);
assert.equal(result.rows.find((row) => row.id === "shared").status, "done");
assert.equal(result.coverage.duplicateCount, 1);
assert.equal(result.coverage.complete, true);
assert.deepEqual(result.coverage.statusCounts, {
backlog: 0,
todo: 1,
in_progress: 0,
in_review: 0,
blocked: 0,
done: 2,
});
});
test("warns when an individual status query reaches the Mine endpoint cap", async (t) => {
t.mock.method(globalThis, "fetch", async (url) => {
const status = new URL(url).searchParams.get("status");
const rows = status === "done"
? Array.from({ length: 500 }, (_, index) => issue({ id: `done-${index}` }))
: [];
return new Response(JSON.stringify(rows), { status: 200 });
});
const result = await fetchMineInboxRows({
apiBase: "https://paperclip.example",
apiKey: "secret",
}, "user-1");
assert.equal(result.rows.length, 500);
assert.equal(result.coverage.complete, false);
assert.deepEqual(result.coverage.cappedStatuses, ["done"]);
assert.equal(result.coverage.queryCap, 500);
});
.github/PULL_REQUEST_TEMPLATE.md
+18 -8
View File
@@ -1,3 +1,5 @@
<!-- Write all pull request text in Simplified Technical English (ASD-STE100): short sentences, one instruction per sentence, simple approved vocabulary, and the active voice. -->
## Thinking Path
<!--
@@ -17,10 +19,11 @@
## Linked Issues or Issue Description
<!--
Required. Pick ONE of the following two paths:
Required. Pick ONE of the two paths below.
(A) Issue exists — tag each linked issue with `Fixes: #123`, `Closes #123`,
or `Refs #123`. Include duplicates and closely related issues too.
(A) Issue exists — replace the placeholder below with your issue links.
Tag each linked issue with `Fixes: #123`, `Closes #123`, or `Refs #123`.
Include duplicates and closely related issues too.
Only reference PUBLIC GitHub issues/PRs here. Do NOT paste internal,
instance-local Paperclip references — ticket ids like PAPA-123 / PAP-224,
@@ -28,11 +31,18 @@
contributors cannot open them. See CONTRIBUTING.md → "No Internal Issue
References".
(B) No issue exists — describe the underlying problem here, following the
relevant issue template so reviewers get the same fields:
• Bug: .github/ISSUE_TEMPLATE/bug_report.yml
• Feature: .github/ISSUE_TEMPLATE/feature_request.yml
• Adapter: .github/ISSUE_TEMPLATE/adapter_request.yml
(B) No issue exists — describe the underlying problem here. Follow the issue
template that fits your change. Open the matching file and copy its field
labels into your description:
• Bug: .github/ISSUE_TEMPLATE/bug_report.yml
• Feature: .github/ISSUE_TEMPLATE/feature_request.yml
• Adapter: .github/ISSUE_TEMPLATE/adapter_request.yml
• Enhancement: .github/ISSUE_TEMPLATE/enhancement.yml
• Docs: .github/ISSUE_TEMPLATE/docs_issue.yml
An automated check reads the literal bold labels AND the content under
each label. Keep at least three of these labels, each alone on its own
line, and write real content under each. A label with only the bare "-"
placeholder does not count, and the check fails.
See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".
-->
.github/scripts/check-pr-linked-issue.mjs
+91 -14
View File
@@ -47,34 +47,111 @@ const TEMPLATE_FIELDS = {
["Why this adapter is useful", "Why it's useful", 'Why useful', 'Use case'],
['How the agent is invoked', 'How it is invoked', "How it's invoked", 'Invocation'],
],
// Labels below match .github/ISSUE_TEMPLATE/enhancement.yml exactly.
enhancement: [
['What existing behavior does this improve?', 'What existing behavior does this improve'],
['Subsystem affected'],
['Current behavior'],
['Proposed behavior'],
['Reason and benefit'],
['Breaking changes'],
],
// Labels below match .github/ISSUE_TEMPLATE/docs_issue.yml exactly. The
// template has 4 distinct fields, so it meets the 3-field minimum. A
// "docs"-prefixed PR skips this check; this set helps a non-"docs"-prefixed
// PR that describes a documentation issue inline.
docs: [
['Issue type'],
['Where is the issue?', 'Where is the issue'],
["What's wrong?", "What's wrong"],
['Suggested fix'],
],
};
function escapeRegExp(s) {
return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}
function countMatchedFields(body, fieldSet) {
// A generic "label line" is a markdown heading (`## Label`) or a bolded label on
// its own line (`**Label**`). The content scan stops at a label line, because
// that line starts a new field.
const LABEL_LINE = /^\s*(?:#{1,6}\s+\S|(?:\*\*|__)[^*_].*(?:\*\*|__)\s*[:?]?\s*$)/;
// Build the regex that matches one field label on its own line.
function labelLinePattern(label) {
const esc = escapeRegExp(label);
// Accept markdown headings or bolded/plain labels on their own line.
// Examples: "## What happened?", "**Expected behavior**", "Problem:".
return new RegExp(
`^\\s*(?:#{1,6}\\s+|\\*\\*\\s*|__\\s*)?${esc}(?:\\s*[:?])?(?:\\s*\\*\\*|\\s*__)?\\s*$`,
'i'
);
}
// Every known field label from every template, precompiled. The generic
// LABEL_LINE regex sees a heading or a bold label as a field boundary, but not a
// plain "Label:" line. A skeleton of stacked plain labels needs each label to
// act as a boundary. Without this list the scan reads the next label as
// content, so it counts an empty field as filled.
const KNOWN_LABEL_PATTERNS = Object.values(TEMPLATE_FIELDS)
.flat(2)
.map(labelLinePattern);
// Return true if the line starts a new field. The line is a heading, a bold
// label, or a plain line that equals a known field label.
function isFieldBoundary(line) {
return LABEL_LINE.test(line) || KNOWN_LABEL_PATTERNS.some(p => p.test(line));
}
// Return true if the line holds real content, not a bare placeholder. The
// default template skeleton puts a lone "-" under each label, so a label with
// only "-", blank lines, or a "[...]" placeholder does not count as filled.
function lineHasContent(line) {
let text = line.trim();
if (!text) return false;
// Drop a leading list marker ("- ", "* ", "1. ") before the check.
text = text.replace(/^[-*+]\s*/, '').replace(/^\d+[.)]\s*/, '').trim();
if (!text) return false;
// Treat a whole-line bracket placeholder ("[describe here]") as empty.
if (/^\[.*\]$/.test(text)) return false;
return true;
}
// Return true if a label variant appears on its own line AND at least one
// content line follows it before the next label line.
function isFieldFilled(lines, variants) {
const patterns = variants.map(labelLinePattern);
for (let i = 0; i < lines.length; i += 1) {
if (!patterns.some(p => p.test(lines[i]))) continue;
for (let j = i + 1; j < lines.length; j += 1) {
if (isFieldBoundary(lines[j])) break; // next field starts here
if (lineHasContent(lines[j])) return true;
}
}
return false;
}
function countMatchedFields(lines, fieldSet) {
let matched = 0;
for (const variants of fieldSet) {
const hasMatch = variants.some(label => {
const esc = escapeRegExp(label);
// Accept markdown headings or bolded/plain labels on their own line.
// Examples: "## What happened?", "**Expected behavior**", "Problem:".
const pattern = new RegExp(
`^\\s*(?:#{1,6}\\s+|\\*\\*\\s*|__\\s*)?${esc}(?:\\s*[:?])?(?:\\s*\\*\\*|\\s*__)?\\s*$`,
'im'
);
return pattern.test(body);
});
if (hasMatch) matched += 1;
if (isFieldFilled(lines, variants)) matched += 1;
}
return matched;
}
// Remove HTML comments. The PR template puts its guidance and its example
// issue links ("Fixes: #123") inside comments, so the gate must not read them
// as author content.
function stripHtmlComments(body) {
return body.replace(/<!--[\s\S]*?-->/g, '');
}
export function hasInlineIssueDescription(body) {
if (!body || !body.trim()) return false;
// Strip the guidance comments, then scan the body line by line.
const lines = stripHtmlComments(body).split(/\r?\n/);
for (const fieldSet of Object.values(TEMPLATE_FIELDS)) {
if (countMatchedFields(body, fieldSet) >= INLINE_DESCRIPTION_MIN_FIELDS) {
if (countMatchedFields(lines, fieldSet) >= INLINE_DESCRIPTION_MIN_FIELDS) {
return true;
}
}
@@ -98,7 +175,7 @@ export function checkLinkedIssue(body, prTitle = '') {
return { passed: false, failures: ['PR body is empty — please fill out the PR template'] };
}
const linked = ISSUE_PATTERNS.some(p => p.test(body));
const linked = ISSUE_PATTERNS.some(p => p.test(stripHtmlComments(body)));
const inlined = hasInlineIssueDescription(body);
const passed = linked || inlined;
.github/scripts/tests/check-pr-linked-issue.test.mjs
+183
View File
@@ -1,5 +1,7 @@
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { readFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';
import { checkLinkedIssue, hasInlineIssueDescription } from '../check-pr-linked-issue.mjs';
// Existing tests with title parameter added (defaults to no prefix, so still required)
@@ -221,3 +223,184 @@ None.
`;
assert.equal(hasInlineIssueDescription(body), true);
});
// Prose-only description (no template labels) must fail. A good paragraph of
// prose matches zero labels, so the gate rejects it.
test('fails with a prose-only description that has no template labels', () => {
const body = `
This pull request rewrites the retry loop so the worker gives up after five
attempts instead of looping forever. The previous loop could hang a job when
the upstream service was down. I also added a log line for each retry so an
operator can see the backoff in the run output.
`;
const result = checkLinkedIssue(body, 'feat: bounded retry');
assert.equal(result.passed, false);
assert.ok(result.failures.length > 0);
});
// An author who copies the feature template labels into the PR body must pass.
// The labels use the bold-label-on-its-own-line form the gate accepts.
const FEATURE_BOLD_LABEL_BODY = `
**Problem or motivation:**
- The gate rejects a good prose description.
**Proposed solution:**
- Copy the feature template labels into the PR body.
**Alternatives considered:**
- Lower the field threshold — rejected, it weakens the gate.
`;
test('passes with the feature template labels (bold labels)', () => {
assert.equal(checkLinkedIssue(FEATURE_BOLD_LABEL_BODY, 'feat: inline feature description').passed, true);
});
// Enhancement template set (matches .github/ISSUE_TEMPLATE/enhancement.yml).
const ENHANCEMENT_INLINE_BODY = `
## What existing behavior does this improve?
The board task list sort order.
## Current behavior
The list sorts by creation time only.
## Proposed behavior
The list sorts by priority, then creation time.
## Reason and benefit
Users miss high-priority tasks that were created early.
`;
test('passes with inline enhancement description (4 template fields)', () => {
assert.equal(checkLinkedIssue(ENHANCEMENT_INLINE_BODY, 'feat: sort by priority').passed, true);
});
test('hasInlineIssueDescription returns true for ≥3 enhancement fields', () => {
assert.equal(hasInlineIssueDescription(ENHANCEMENT_INLINE_BODY), true);
});
// Empty default skeleton must fail. A label with only the bare "-" placeholder
// under it is not filled, so it must not count toward the field minimum.
const EMPTY_SKELETON_BODY = `
**What happened?**
-
**Expected behavior:**
-
**Steps to reproduce:**
-
`;
test('fails with an empty template skeleton (labels but no content)', () => {
const result = checkLinkedIssue(EMPTY_SKELETON_BODY, 'feat: something');
assert.equal(result.passed, false);
assert.ok(result.failures.length > 0);
});
test('hasInlineIssueDescription returns false for an empty skeleton', () => {
assert.equal(hasInlineIssueDescription(EMPTY_SKELETON_BODY), false);
});
// A filled bug skeleton in the bold-label form must pass, even with list-marker
// content. This proves the fix does not reject real author content.
const FILLED_BUG_SKELETON_BODY = `
**What happened?**
- The login button does nothing.
**Expected behavior:**
- The login button authenticates the user.
**Steps to reproduce:**
- Open the app, then click login.
`;
test('passes with a filled bug skeleton (three filled fields)', () => {
assert.equal(checkLinkedIssue(FILLED_BUG_SKELETON_BODY, 'feat: fix login').passed, true);
});
// Stacked plain labels with no content must fail. Each label sits on its own
// line with the next label directly under it. The scan must treat the next
// label as a field boundary, not as content, so every field stays empty.
const STACKED_FEATURE_LABELS = `
Problem or motivation:
Proposed solution:
Alternatives considered:
Roadmap alignment:
`;
const STACKED_BUG_LABELS = `
What happened?:
Expected behavior:
Steps to reproduce:
Paperclip version:
`;
const STACKED_ENHANCEMENT_LABELS = `
What existing behavior does this improve?
Subsystem affected
Current behavior
Proposed behavior
Reason and benefit
`;
const STACKED_DOCS_LABELS = `
Issue type
Where is the issue?
What's wrong?
Suggested fix
`;
test('fails with stacked plain feature labels and no content', () => {
assert.equal(checkLinkedIssue(STACKED_FEATURE_LABELS, 'feat: x').passed, false);
});
test('fails with stacked plain bug labels and no content', () => {
assert.equal(checkLinkedIssue(STACKED_BUG_LABELS, 'feat: x').passed, false);
});
test('fails with stacked plain enhancement labels and no content', () => {
assert.equal(checkLinkedIssue(STACKED_ENHANCEMENT_LABELS, 'feat: x').passed, false);
});
test('fails with stacked plain docs labels and no content', () => {
assert.equal(checkLinkedIssue(STACKED_DOCS_LABELS, 'feat: x').passed, false);
});
// A plain-label skeleton with real content under each label must still pass.
// The boundary fix must not reject a field that has genuine content.
const FILLED_PLAIN_FEATURE_LABELS = `
Problem or motivation:
- The gate rejects a good prose description.
Proposed solution:
- Copy the feature template labels into the PR body.
Alternatives considered:
- Lower the field threshold — rejected, it weakens the gate.
`;
test('passes with plain feature labels and real content under each', () => {
assert.equal(checkLinkedIssue(FILLED_PLAIN_FEATURE_LABELS, 'feat: inline feature').passed, true);
});
// The real .github/PULL_REQUEST_TEMPLATE.md, submitted unchanged, must fail the
// gate. Its skeleton labels have no content and its example issue links live in
// HTML comments, so neither the inline path nor the linked path may pass it.
const PR_TEMPLATE_PATH = fileURLToPath(
new URL('../../PULL_REQUEST_TEMPLATE.md', import.meta.url)
);
test('fails with the unfilled default PR template body', () => {
const body = readFileSync(PR_TEMPLATE_PATH, 'utf8');
const result = checkLinkedIssue(body, 'feat: unfilled template');
assert.equal(result.passed, false);
});
// An issue link that appears only inside an HTML comment must not satisfy the
// linked-issue check. The template ships such an example ("Fixes: #123").
test('fails when the only issue link is inside an HTML comment', () => {
const body = '<!-- Example: Fixes: #123 -->\n\nSome prose with no real link.';
assert.equal(checkLinkedIssue(body, 'feat: commented link').passed, false);
});
.github/workflows/docker.yml
+121 -8
View File
@@ -11,13 +11,17 @@ permissions:
contents: read
packages: write
# Serialise builds per ref without killing an in-flight one: a newer push
# supersedes only the pending slot, so the image build that is already
# running always finishes and publishes.
concurrency:
group: docker-${{ github.ref }}
cancel-in-progress: false
jobs:
build-and-push:
runs-on: ubuntu-latest
timeout-minutes: 60
concurrency:
group: docker-${{ github.ref }}
cancel-in-progress: true
steps:
- name: Checkout
uses: actions/checkout@v7
@@ -151,11 +155,120 @@ jobs:
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# The cloud variant carries built bundled plugins for managed
# deployments (see the `cloud` stage in the Dockerfile). Published
# under the same tag set with a `-cloud` suffix (sha-<short>-cloud,
# latest-cloud, <version>-cloud). Reuses the layer cache from the
# production build, so this mostly adds the plugin-build layers.
# The cloud variant carries built bundled plugins for managed deployments
# (see the `cloud` stage in the Dockerfile). It runs as its own job with no
# `needs:` on the stock publish above, so the two builds run in parallel and
# a failure or slow build in one never gates, delays, or skips the other.
# Both jobs share only the single top-level concurrency slot. Each job is a
# separate runner, so this one carries its own copy of the prep steps
# (checkout through schema labels) — the accepted cost of that isolation.
build-and-push-cloud:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Checkout
uses: actions/checkout@v7
with:
# Full history and tags so `git describe` below can compute the
# release version to stamp into the image.
fetch-depth: 0
# `.git` is dockerignored, so a running image cannot derive its own
# version and otherwise reports the source package.json placeholder in
# analytics and the debug panel. Compute it here from the pristine
# checkout (real CalVer drift from the nearest release tag) and pass it
# into the build. Empty when no release tag is reachable — the server
# then keeps its existing fallbacks.
- name: Compute build version
id: build-version
run: |
set -euo pipefail
version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)"
echo "version=${version}" >> "$GITHUB_OUTPUT"
echo "Stamping build version: ${version:-<none>}"
- name: Setup pnpm
uses: pnpm/action-setup@v6
with:
version: 9.15.4
run_install: false
# No dependency cache here: this workflow publishes release images, and
# restoring a shared Actions cache into the build inputs would let a
# poisoned cache entry reach the published artifact.
- name: Setup Node.js
uses: actions/setup-node@v7
with:
node-version: 20
- name: Refresh lockfile for Docker build context
run: |
set -euo pipefail
pnpm install --lockfile-only --ignore-scripts --no-frozen-lockfile
changed="$(git status --porcelain)"
if [ -z "$changed" ]; then
echo "Lockfile already matches package metadata."
exit 0
fi
if printf '%s\n' "$changed" | grep -Fvq ' pnpm-lock.yaml'; then
echo "Unexpected files changed during lockfile refresh:"
echo "$changed"
exit 1
fi
echo "Using refreshed pnpm-lock.yaml in the Docker build context."
- name: Free runner disk
run: |
set -euo pipefail
echo "Disk before cleanup:"
df -h
pnpm store prune || true
sudo apt-get clean || true
sudo rm -rf \
/usr/share/dotnet \
/usr/share/swift \
/usr/local/lib/android \
/usr/local/share/boost \
/usr/local/share/powershell \
/opt/ghc \
/opt/hostedtoolcache/CodeQL \
/opt/hostedtoolcache/PyPy \
/opt/hostedtoolcache/Ruby || true
docker system prune -af || true
echo "Disk after cleanup:"
df -h
- name: Login to GitHub Container Registry
uses: docker/login-action@v4
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4
# Deployment tooling reads these labels from the registry to verify an
# image's schema expectations against a migrator before deploying it,
# without pulling the image. The server refuses to start when the
# database is missing bundled migrations, so orchestrators need a cheap
# way to check image/migrator compatibility up front.
- name: Compute schema migration labels
id: schema
run: |
set -euo pipefail
last=$(ls packages/db/src/migrations/*.sql | sed 's|.*/||' | LC_ALL=C sort | tail -1)
count=$(ls packages/db/src/migrations/*.sql | wc -l | tr -d ' ')
echo "last=${last}" >> "$GITHUB_OUTPUT"
echo "count=${count}" >> "$GITHUB_OUTPUT"
# Published under the same tag set with a `-cloud` suffix
# (sha-<short>-cloud, latest-cloud, <version>-cloud).
- name: Docker meta (cloud)
id: meta-cloud
uses: docker/metadata-action@v6
CONTRIBUTING.md
+2
View File
@@ -157,6 +157,8 @@ Bugs, docs improvements, and small targeted improvements are still the easiest p
## Writing a Good PR message
Write all PR text in Simplified Technical English (ASD-STE100): use short sentences, one instruction per sentence, simple approved vocabulary, and the active voice.
Your PR description must follow the [PR template](.github/PULL_REQUEST_TEMPLATE.md). All sections are required. The "thinking path" at the top explains from the top of the project down to what you fixed. E.g.:
### Thinking Path Example 1:
README.md
+1 -1
View File
@@ -450,7 +450,7 @@ See [doc/DEVELOPING.md](doc/DEVELOPING.md) for the full development guide.
- ⚪ Self-Organization
- ⚪ Automatic Organizational Learning
- ⚪ CEO Chat
- 🟡 Cloud deployments (multi-tenant isolation & local→cloud sync shipped)
- 🟡 Cloud deployments (multi-tenant isolation & company Import/Export shipped)
- ⚪ Desktop App
- ⚪ Bring-your-own-ticket-system (Asana / Linear / Jira as on-ramps)
- ⚪ Connected Apps (one-click integrations, e.g. Vercel)
ROADMAP.md
+2 -2
View File
@@ -108,11 +108,11 @@ Paperclip should get better at turning completed work into reusable organization
We want a lighter-weight way to talk to leadership agents, but those conversations should still resolve to real work objects like plans, issues, approvals, or decisions. This should improve interaction without changing the core task-and-comments model.
### 🟡 Cloud deployments (multi-tenant isolation & local→cloud sync shipped)
### 🟡 Cloud deployments (multi-tenant isolation & company Import/Export shipped)
Local-first remains important, but Paperclip also needs a cleaner shared deployment story. Teams should be able to run the same product in hosted or semi-hosted environments without changing the mental model.
Shipped so far: multi-tenant isolation with per-company JWT keys and company-scoped cloud tenants, local→cloud upstream sync, and cloud-managed instance bootstrap.
Shipped so far: multi-tenant isolation with per-company JWT keys and company-scoped cloud tenants, portable company Import/Export (zip bundles that move a company between instances, local or cloud), and cloud-managed instance bootstrap. Next: a blob-store relay so large instances can move without a hand-carried bundle.
### ⚪ Desktop App
cli/README.md
+6 -9
View File
@@ -332,20 +332,17 @@ By default, agents run on scheduled heartbeats and event-based triggers (task as
<br/>
## Paperclip Cloud Sync
## Importing & Exporting Companies
Cloud upstream sync is behind the `Cloud Sync` experimental setting. Enable it in Instance Settings before pushing.
Export a company to a portable package and import it into any other instance — local or cloud — from a local path or GitHub:
```bash
paperclipai cloud connect https://your-stack.paperclip.app
paperclipai cloud connect https://your-stack.paperclip.app --no-browser
paperclipai cloud push --company <local-company-id> --dry-run
paperclipai cloud push --company <local-company-id>
paperclipai company export <company-id> --out ./my-export
paperclipai company import ./my-export --dry-run
paperclipai company import org/repo --target new
```
`cloud connect` authorizes the local instance against the target stack and stores the upstream token in the local instance secret store. The default path opens a browser for consent; `--no-browser` uses the device-code flow and prints the verification URL and user code.
`cloud push --dry-run` exports the selected local company, sends a preview bundle to the connected Cloud stack, and exits with code `2` when conflicts need user resolution. A schema mismatch exits with code `3`. Running without `--dry-run` stages chunks idempotently, applies the run, and prints the final summary and recent progress events.
The board UI has matching Export and Import pages in company settings: the Export page shows a fidelity panel listing what the bundle will not carry, and the Import page starts imported agents and routines paused by default, with a post-import activation step. See the [Importing & Exporting guide](https://github.com/paperclipai/paperclip/blob/master/docs/guides/board-operator/importing-and-exporting.md) for details.
## Development
cli/package.json
+1 -1
View File
@@ -58,7 +58,7 @@
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"tsx": "^4.23.1",
"typescript": "^5.7.3"
}
cli/src/__tests__/cloud.test.ts
-243
View File
@@ -1,243 +0,0 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { CompanyPortabilityExportResult } from "@paperclipai/shared";
import {
assertDiscoveryCompatible,
buildBundleFromLocalCompany,
cloudCommandExitCodes,
connectCloud,
resolveDeviceCodeExpiresAt,
} from "../commands/client/cloud.js";
import {
LocalUpstreamPushCoordinator,
normalizedContentHash,
type LocalUpstreamExportBundle,
} from "../commands/client/cloud-transfer.js";
import { getCloudConnection } from "../commands/client/cloud-store.js";
const originalEnv = { ...process.env };
const originalFetch = globalThis.fetch;
describe("cloud CLI helpers", () => {
let tempHome: string;
beforeEach(() => {
tempHome = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-cloud-cli-"));
process.env = { ...originalEnv, PAPERCLIP_HOME: tempHome };
});
afterEach(() => {
process.env = { ...originalEnv };
globalThis.fetch = originalFetch;
vi.restoreAllMocks();
fs.rmSync(tempHome, { recursive: true, force: true });
});
it("connects with the device-code flow and stores the resulting cloud connection", async () => {
globalThis.fetch = vi.fn(async (url, init) => {
const requestUrl = String(url);
if (requestUrl.endsWith("/.well-known/paperclip-upstream")) {
return jsonResponse(discovery());
}
if (requestUrl.endsWith("/api/upstream-sync/device-code")) {
expect(JSON.parse(String(init?.body))).toMatchObject({
stackId: "stack-1",
scopes: ["upstream_import:preview", "upstream_import:write", "upstream_import:read"],
});
return jsonResponse({
deviceCode: "device-1",
userCode: "ABCD-EFGH",
verificationUri: "https://cloud.example.test/api/upstream-sync/device-code/approve",
expiresAt: new Date(Date.now() + 60_000).toISOString(),
intervalSeconds: 0,
});
}
if (requestUrl.endsWith("/api/upstream-sync/token")) {
return jsonResponse({
accessToken: "upt_test",
scopes: ["upstream_import:preview"],
token: {
id: "token-1",
companyStackId: "stack-1",
targetOrigin: "https://cloud.example.test",
sourceInstanceId: "paperclip-local-default",
sourceInstanceFingerprint: "sha256:test",
scopes: ["upstream_import:preview"],
expiresAt: new Date(Date.now() + 60_000).toISOString(),
},
});
}
return jsonResponse({ error: "not_found" }, 404);
}) as typeof fetch;
const connection = await connectCloud("https://cloud.example.test", { noBrowser: true, json: true });
expect(connection.accessToken).toBe("upt_test");
expect(getCloudConnection("https://cloud.example.test")?.token.id).toBe("token-1");
});
it("hard-blocks incompatible transfer schema versions with the stable schema exit code", () => {
expect(() => assertDiscoveryCompatible(discovery({ supportedSchemaMajor: 99 }))).toThrow(/schema mismatch/i);
expect(cloudCommandExitCodes.schemaMismatch).toBe(3);
});
it("falls back to a bounded device-code expiry when the cloud omits or malforms expiresAt", () => {
const now = Date.UTC(2026, 4, 22, 13, 0, 0);
const validExpiry = "2026-05-22T13:05:00.000Z";
expect(resolveDeviceCodeExpiresAt(validExpiry, now)).toBe(Date.parse(validExpiry));
expect(resolveDeviceCodeExpiresAt(undefined, now)).toBe(now + 15 * 60_000);
expect(resolveDeviceCodeExpiresAt("not-a-date", now)).toBe(now + 15 * 60_000);
});
it("builds deterministic chunks with validated payload hashes", async () => {
const bundle = await buildTestBundle();
expect(bundle.chunks).toHaveLength(2);
expect(bundle.chunks[0]?.sha256).toBe(normalizedContentHash(bundle.chunks[0]?.payload));
expect(bundle.manifest.chunks[0]?.manifestHash).toBe(bundle.manifest.manifestHash);
expect(bundle.manifest.idempotencyKey).toBe((await buildTestBundle()).manifest.idempotencyKey);
});
it("reuses the same manifest and chunk identity when an interrupted apply is retried", async () => {
const bundle = await buildTestBundle();
const calls: Array<{ path: string; body: unknown }> = [];
const coordinator = new LocalUpstreamPushCoordinator({
targetOrigin: "https://cloud.example.test",
paperclipCompanyId: "target-company-1",
fetch: async (url, init) => {
const parsed = new URL(String(url));
const body = init?.body ? JSON.parse(String(init.body)) as unknown : {};
calls.push({ path: parsed.pathname, body });
if (parsed.pathname.endsWith("/runs")) return jsonResponse({ run: { id: "run-1" } });
return jsonResponse({ run: { id: "run-1" }, summary: { create: 0, update: 0, adopt: 0, skip: 2, conflict: 0, staleMapping: 0 } });
},
});
await coordinator.apply(bundle);
await coordinator.apply(bundle);
const runBodies = calls.filter((call) => call.path.endsWith("/runs")).map((call) => call.body as { manifest: { idempotencyKey: string } });
const chunkBodies = calls.filter((call) => call.path.endsWith("/chunks")).map((call) => call.body as { chunkIndex: number; sha256: string });
expect(runBodies).toHaveLength(2);
expect(runBodies[0]?.manifest.idempotencyKey).toBe(runBodies[1]?.manifest.idempotencyKey);
expect(chunkBodies[0]).toEqual(chunkBodies[2]);
expect(chunkBodies[1]).toEqual(chunkBodies[3]);
});
});
async function buildTestBundle(): Promise<LocalUpstreamExportBundle> {
return buildBundleFromLocalCompany({
localCompanyId: "local-company-1",
connection: {
id: "conn-1",
remoteUrl: "https://cloud.example.test",
targetOrigin: "https://cloud.example.test",
targetHost: "cloud.example.test",
stackId: "stack-1",
targetCompanyId: "target-company-1",
accessToken: "upt_test",
token: {
id: "token-1",
companyStackId: "stack-1",
targetOrigin: "https://cloud.example.test",
sourceInstanceId: "paperclip-local-default",
sourceInstanceFingerprint: "sha256:test",
scopes: ["upstream_import:preview"],
expiresAt: new Date(Date.now() + 60_000).toISOString(),
},
privateKeyPem: "unused",
sourcePublicKey: "unused",
sourceInstanceId: "paperclip-local-default",
sourceInstanceFingerprint: "sha256:test",
scopes: ["upstream_import:preview"],
createdAt: "2026-05-18T00:00:00.000Z",
updatedAt: "2026-05-18T00:00:00.000Z",
},
discovery: discovery(),
localApi: {
post: async <T>() => portabilityExport() as T,
},
maxEntitiesPerChunk: 1,
mode: "apply",
});
}
function discovery(overrides: Partial<{ supportedSchemaMajor: number }> = {}) {
return {
schema: "paperclip-upstream-discovery-v1",
stack: {
id: "stack-1",
slug: "cloud-test",
displayName: "Cloud Test",
companyId: "target-company-1",
origin: "https://cloud.example.test",
},
auth: {
deviceCode: {
deviceCodeUrl: "https://cloud.example.test/api/upstream-sync/device-code",
verificationUrl: "https://cloud.example.test/api/upstream-sync/device-code/approve",
tokenUrl: "https://cloud.example.test/api/upstream-sync/token",
},
scopes: ["upstream_import:preview", "upstream_import:write", "upstream_import:read"],
},
transfer: {
supportedSchemaMajor: overrides.supportedSchemaMajor ?? 1,
featureFlags: ["cloud_sync"],
},
};
}
function portabilityExport(): CompanyPortabilityExportResult {
return {
rootPath: ".",
paperclipExtensionPath: ".paperclip.yaml",
manifest: {
schemaVersion: 1,
generatedAt: "2026-05-18T00:00:00.000Z",
source: {
companyId: "local-company-1",
companyName: "Local Company",
},
includes: {
company: true,
agents: true,
projects: true,
issues: true,
skills: true,
},
company: {
path: "company.json",
name: "Local Company",
description: null,
brandColor: null,
logoPath: null,
attachmentMaxBytes: null,
requireBoardApprovalForNewAgents: false,
feedbackDataSharingEnabled: false,
feedbackDataSharingConsentAt: null,
feedbackDataSharingConsentByUserId: null,
feedbackDataSharingTermsVersion: null,
},
sidebar: null,
agents: [],
skills: [],
projects: [],
issues: [],
envInputs: [],
},
files: {
"README.md": "Local Company",
},
warnings: [],
};
}
function jsonResponse(body: unknown, status = 200): Response {
return new Response(JSON.stringify(body), {
status,
headers: { "Content-Type": "application/json" },
});
}
cli/src/__tests__/company-import-zip.test.ts
+7
View File
@@ -18,12 +18,14 @@ describe("resolveInlineSourceFromPath", () => {
const tempDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-company-import-zip-"));
tempDirs.push(tempDir);
const blobBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff]);
const archivePath = path.join(tempDir, "paperclip-demo.zip");
const archive = createStoredZipArchive(
{
"COMPANY.md": "# Company\n",
".paperclip.yaml": "schema: paperclip/v1\n",
"agents/ceo/AGENT.md": "# CEO\n",
"blobs/4f2d1c9a": blobBytes,
"notes/todo.txt": "ignore me\n",
},
"paperclip-demo",
@@ -38,6 +40,11 @@ describe("resolveInlineSourceFromPath", () => {
"COMPANY.md": "# Company\n",
".paperclip.yaml": "schema: paperclip/v1\n",
"agents/ceo/AGENT.md": "# CEO\n",
"blobs/4f2d1c9a": {
encoding: "base64",
data: Buffer.from(blobBytes).toString("base64"),
contentType: "application/octet-stream",
},
},
});
});
cli/src/__tests__/company.test.ts
+3
View File
@@ -509,6 +509,9 @@ describe("renderCompanyImportResult", () => {
{ slug: "ops", id: "project-2", action: "updated", name: "Operations", reason: "replace strategy" },
{ slug: "archive", id: null, action: "skipped", name: "Archive", reason: "skip strategy" },
],
routines: [
{ slug: "weekly-report", id: "routine-1", action: "created", title: "Weekly report", status: "paused" },
],
envInputs: [],
warnings: ["Review API keys"],
},
cli/src/__tests__/helpers/zip.ts
+2 -2
View File
@@ -21,7 +21,7 @@ function crc32(bytes: Uint8Array) {
return (crc ^ 0xffffffff) >>> 0;
}
export function createStoredZipArchive(files: Record<string, string>, rootPath: string) {
export function createStoredZipArchive(files: Record<string, string | Uint8Array>, rootPath: string) {
const encoder = new TextEncoder();
const localChunks: Uint8Array[] = [];
const centralChunks: Uint8Array[] = [];
@@ -30,7 +30,7 @@ export function createStoredZipArchive(files: Record<string, string>, rootPath:
for (const [relativePath, content] of Object.entries(files).sort(([left], [right]) => left.localeCompare(right))) {
const fileName = encoder.encode(`${rootPath}/${relativePath}`);
const body = encoder.encode(content);
const body = typeof content === "string" ? encoder.encode(content) : content;
const checksum = crc32(body);
const localHeader = new Uint8Array(30 + fileName.length);
cli/src/__tests__/worktree.test.ts
+2
View File
@@ -276,6 +276,7 @@ describe("worktree helpers", () => {
path.resolve("/tmp/paperclip-worktrees", "instances", "feature-worktree-support", "db"),
);
expect(config.database.embeddedPostgresPort).toBe(54339);
expect(config.database.backup.enabled).toBe(false);
expect(config.server.port).toBe(3110);
expect(config.auth.publicBaseUrl).toBe("http://127.0.0.1:3110/");
expect(config.storage.localDisk.baseDir).toBe(
@@ -289,6 +290,7 @@ describe("worktree helpers", () => {
expect(env.PAPERCLIP_HOME).toBe(path.resolve("/tmp/paperclip-worktrees"));
expect(env.PAPERCLIP_INSTANCE_ID).toBe("feature-worktree-support");
expect(env.PAPERCLIP_IN_WORKTREE).toBe("true");
expect(env.PAPERCLIP_DB_BACKUP_ENABLED).toBe("false");
expect(env.PAPERCLIP_WORKTREE_NAME).toBe("feature-worktree-support");
expect(env.PAPERCLIP_WORKTREE_COLOR).toBe("#3abf7a");
expect(formatShellExports(env)).toContain("export PAPERCLIP_INSTANCE_ID='feature-worktree-support'");
cli/src/__tests__/zip-codec.test.ts
+69
View File
@@ -0,0 +1,69 @@
import { describe, expect, it } from "vitest";
import { bytesToPortableFileEntry, isBlobStorePath, readZipArchive } from "../commands/client/zip.js";
import { createStoredZipArchive } from "./helpers/zip.js";
describe("isBlobStorePath", () => {
it("matches blobs/ entries at the archive root and under a package root", () => {
expect(isBlobStorePath("blobs/4f2d1c9a")).toBe(true);
expect(isBlobStorePath("paperclip-demo/blobs/4f2d1c9a")).toBe(true);
expect(isBlobStorePath("tasks/pap-1/TASK.md")).toBe(false);
expect(isBlobStorePath("blobs/nested/file")).toBe(false);
});
});
describe("bytesToPortableFileEntry", () => {
it("keeps blobs/ entries as base64 octet streams regardless of extension", () => {
const bytes = new Uint8Array([0x00, 0x01, 0x80, 0xfe, 0xff]);
expect(bytesToPortableFileEntry("blobs/4f2d1c9a", bytes)).toEqual({
encoding: "base64",
data: Buffer.from(bytes).toString("base64"),
contentType: "application/octet-stream",
});
});
it("falls back to base64 when bytes are not valid UTF-8", () => {
const invalidUtf8 = new Uint8Array([0x68, 0x69, 0xff, 0xfe, 0xc0]);
expect(bytesToPortableFileEntry("tasks/pap-1/raw-notes", invalidUtf8)).toEqual({
encoding: "base64",
data: Buffer.from(invalidUtf8).toString("base64"),
contentType: "application/octet-stream",
});
});
it("decodes valid UTF-8 entries to text", () => {
const bytes = new TextEncoder().encode("# Notes\n\ncafé ✅\n");
expect(bytesToPortableFileEntry("tasks/pap-1/TASK.md", bytes)).toBe("# Notes\n\ncafé ✅\n");
});
});
describe("readZipArchive", () => {
it("round-trips blob and invalid UTF-8 entries byte-exactly", async () => {
const blobBytes = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 0x00, 0xff]);
const invalidUtf8 = new Uint8Array([0x68, 0x69, 0xff, 0xfe, 0xc0]);
const archive = createStoredZipArchive(
{
"COMPANY.md": "# Company\n",
"blobs/4f2d1c9a": blobBytes,
"notes/raw": invalidUtf8,
},
"paperclip-demo",
);
await expect(readZipArchive(archive)).resolves.toEqual({
rootPath: "paperclip-demo",
files: {
"COMPANY.md": "# Company\n",
"blobs/4f2d1c9a": {
encoding: "base64",
data: Buffer.from(blobBytes).toString("base64"),
contentType: "application/octet-stream",
},
"notes/raw": {
encoding: "base64",
data: Buffer.from(invalidUtf8).toString("base64"),
contentType: "application/octet-stream",
},
},
});
});
});
cli/src/commands/client/cloud-store.ts
-177
View File
@@ -1,177 +0,0 @@
import fs from "node:fs";
import path from "node:path";
import { resolvePaperclipInstanceRoot } from "../../config/home.js";
export interface CloudConnectionTokenRecord {
id: string;
companyStackId: string;
targetOrigin: string;
sourceInstanceId: string;
sourceInstanceFingerprint: string;
scopes: string[];
expiresAt: string;
[key: string]: unknown;
}
export interface CloudConnection {
id: string;
remoteUrl: string;
targetOrigin: string;
targetHost: string;
stackId: string;
stackSlug?: string | null;
stackDisplayName?: string | null;
targetCompanyId: string;
accessToken: string;
token: CloudConnectionTokenRecord;
privateKeyPem: string;
sourcePublicKey: string;
sourceInstanceId: string;
sourceInstanceFingerprint: string;
scopes: string[];
createdAt: string;
updatedAt: string;
}
interface CloudConnectionStore {
version: 1;
connections: Record<string, CloudConnection>;
currentConnectionId?: string;
}
function defaultStore(): CloudConnectionStore {
return {
version: 1,
connections: {},
};
}
export function resolveCloudConnectionStorePath(): string {
return path.resolve(resolvePaperclipInstanceRoot(), "secrets", "cloud-upstream-connections.json");
}
export function readCloudConnectionStore(storePath = resolveCloudConnectionStorePath()): CloudConnectionStore {
if (!fs.existsSync(storePath)) return defaultStore();
const raw = JSON.parse(fs.readFileSync(storePath, "utf8")) as Partial<CloudConnectionStore> | null;
const connections: Record<string, CloudConnection> = {};
if (raw?.connections && typeof raw.connections === "object") {
for (const [id, value] of Object.entries(raw.connections)) {
const normalized = normalizeConnection(value);
if (normalized) connections[id] = normalized;
}
}
const currentConnectionId =
typeof raw?.currentConnectionId === "string" && connections[raw.currentConnectionId]
? raw.currentConnectionId
: Object.values(connections).sort((left, right) => right.updatedAt.localeCompare(left.updatedAt))[0]?.id;
return {
version: 1,
connections,
currentConnectionId,
};
}
export function writeCloudConnectionStore(
store: CloudConnectionStore,
storePath = resolveCloudConnectionStorePath(),
): void {
fs.mkdirSync(path.dirname(storePath), { recursive: true });
fs.writeFileSync(storePath, `${JSON.stringify(store, null, 2)}\n`, { mode: 0o600 });
}
export function upsertCloudConnection(
connection: CloudConnection,
storePath = resolveCloudConnectionStorePath(),
): CloudConnection {
const store = readCloudConnectionStore(storePath);
const existing = store.connections[connection.id];
const now = new Date().toISOString();
const next = {
...connection,
createdAt: existing?.createdAt ?? connection.createdAt ?? now,
updatedAt: now,
};
store.connections[next.id] = next;
store.currentConnectionId = next.id;
writeCloudConnectionStore(store, storePath);
return next;
}
export function getCloudConnection(
remoteUrlOrOrigin?: string,
storePath = resolveCloudConnectionStorePath(),
): CloudConnection | null {
const store = readCloudConnectionStore(storePath);
if (remoteUrlOrOrigin?.trim()) {
const needle = normalizeRemoteLookup(remoteUrlOrOrigin);
return Object.values(store.connections).find((connection) =>
normalizeRemoteLookup(connection.remoteUrl) === needle ||
normalizeRemoteLookup(connection.targetOrigin) === needle
) ?? null;
}
return store.currentConnectionId ? store.connections[store.currentConnectionId] ?? null : null;
}
function normalizeRemoteLookup(value: string): string {
try {
const url = new URL(value);
return url.origin.replace(/\/+$/u, "");
} catch {
return value.trim().replace(/\/+$/u, "");
}
}
function normalizeConnection(value: unknown): CloudConnection | null {
if (typeof value !== "object" || value === null || Array.isArray(value)) return null;
const record = value as Record<string, unknown>;
const id = stringValue(record.id);
const remoteUrl = stringValue(record.remoteUrl);
const targetOrigin = stringValue(record.targetOrigin);
const targetHost = stringValue(record.targetHost);
const stackId = stringValue(record.stackId);
const targetCompanyId = stringValue(record.targetCompanyId);
const accessToken = stringValue(record.accessToken);
const token = typeof record.token === "object" && record.token !== null && !Array.isArray(record.token)
? record.token as CloudConnectionTokenRecord
: null;
const privateKeyPem = stringValue(record.privateKeyPem);
const sourcePublicKey = stringValue(record.sourcePublicKey);
const sourceInstanceId = stringValue(record.sourceInstanceId);
const sourceInstanceFingerprint = stringValue(record.sourceInstanceFingerprint);
const createdAt = stringValue(record.createdAt);
const updatedAt = stringValue(record.updatedAt);
if (
!id || !remoteUrl || !targetOrigin || !targetHost || !stackId || !targetCompanyId ||
!accessToken || !token || !privateKeyPem || !sourcePublicKey || !sourceInstanceId ||
!sourceInstanceFingerprint || !createdAt || !updatedAt
) {
return null;
}
return {
id,
remoteUrl,
targetOrigin,
targetHost,
stackId,
stackSlug: stringValue(record.stackSlug),
stackDisplayName: stringValue(record.stackDisplayName),
targetCompanyId,
accessToken,
token,
privateKeyPem,
sourcePublicKey,
sourceInstanceId,
sourceInstanceFingerprint,
scopes: stringArray(record.scopes),
createdAt,
updatedAt,
};
}
function stringValue(value: unknown): string | null {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
function stringArray(value: unknown): string[] {
return Array.isArray(value) ? value.filter((entry): entry is string => typeof entry === "string") : [];
}
cli/src/commands/client/cloud-transfer.ts
-297
View File
@@ -1,297 +0,0 @@
import { createHash } from "node:crypto";
export const upstreamTransferSchema = {
family: "paperclip-upstream-transfer",
version: "1.0.0",
major: 1,
minor: 0,
} as const;
export type NormalizedSha256 = `sha256:${string}`;
export interface SourceEntityKey {
sourceInstanceId: string;
sourceCompanyId: string;
sourceEntityType: string;
sourceEntityId: string;
sourceNaturalKey?: string;
}
export interface UpstreamTransferWarning {
code: string;
severity: "info" | "warning" | "blocker";
message: string;
entity?: SourceEntityKey;
}
export interface UpstreamTransferEntityRecord {
key: SourceEntityKey;
contentHash: NormalizedSha256;
dependencies: SourceEntityKey[];
warnings: UpstreamTransferWarning[];
}
export interface UpstreamTransferManifestSource {
sourceInstanceId: string;
sourceCompanyId: string;
sourceInstanceKeyFingerprint: string;
exporterVersion: string;
sourceSchemaVersion: string;
}
export interface UpstreamTransferManifestTarget {
targetStackId: string;
targetCompanyId: string;
targetOrigin: string;
supportedSchemaMajor: number;
}
export interface UpstreamTransferChunk {
chunkIndex: number;
totalChunks: number;
byteLength: number;
sha256: NormalizedSha256;
manifestHash: NormalizedSha256;
}
export interface UpstreamTransferManifest {
schema: typeof upstreamTransferSchema;
source: UpstreamTransferManifestSource;
target: UpstreamTransferManifestTarget;
runId: string;
idempotencyKey: string;
generatedAt: string;
entityCount: number;
entities: UpstreamTransferEntityRecord[];
chunks: UpstreamTransferChunk[];
warnings: UpstreamTransferWarning[];
featureFlags: string[];
manifestHash: NormalizedSha256;
}
export interface LocalUpstreamExportEntityInput {
key: SourceEntityKey;
body: Record<string, unknown>;
dependencies?: SourceEntityKey[];
warnings?: UpstreamTransferWarning[];
conflictKeys?: string[];
}
export interface LocalUpstreamExportEntity {
record: UpstreamTransferEntityRecord;
body: Record<string, unknown>;
conflictKeys?: string[];
}
export interface LocalUpstreamExportChunk {
chunkIndex: number;
totalChunks: number;
byteLength: number;
sha256: NormalizedSha256;
payload: {
entityKeys: SourceEntityKey[];
};
}
export interface LocalUpstreamExportBundle {
manifest: UpstreamTransferManifest;
entities: LocalUpstreamExportEntity[];
chunks: LocalUpstreamExportChunk[];
}
export interface BuildLocalUpstreamExportBundleInput {
source: UpstreamTransferManifestSource;
target: UpstreamTransferManifestTarget;
runId: string;
idempotencyKey: string;
entities: LocalUpstreamExportEntityInput[];
warnings?: UpstreamTransferWarning[];
featureFlags?: string[];
maxEntitiesPerChunk?: number;
}
export interface LocalUpstreamPushCoordinatorOptions {
targetOrigin: string;
paperclipCompanyId: string;
fetch?: typeof fetch;
headers?: (input: { method: string; path: string }) => HeadersInit | Promise<HeadersInit>;
}
export class UpstreamImportRequestError extends Error {
readonly status: number;
readonly body: unknown;
constructor(status: number, message: string, body: unknown) {
super(message);
this.status = status;
this.body = body;
}
}
export class LocalUpstreamPushCoordinator {
readonly #targetOrigin: string;
readonly #paperclipCompanyId: string;
readonly #fetch: typeof fetch;
readonly #headers: NonNullable<LocalUpstreamPushCoordinatorOptions["headers"]>;
constructor(options: LocalUpstreamPushCoordinatorOptions) {
this.#targetOrigin = options.targetOrigin.replace(/\/+$/u, "");
this.#paperclipCompanyId = options.paperclipCompanyId;
this.#fetch = options.fetch ?? fetch;
this.#headers = options.headers ?? (() => ({}));
}
async preview(bundle: LocalUpstreamExportBundle): Promise<unknown> {
return this.post(`/api/companies/${encodeURIComponent(this.#paperclipCompanyId)}/upstream-imports/preview`, {
manifest: bundle.manifest,
entities: bundle.entities,
});
}
async apply(bundle: LocalUpstreamExportBundle): Promise<unknown> {
const run = await this.post(`/api/companies/${encodeURIComponent(this.#paperclipCompanyId)}/upstream-imports/runs`, {
mode: "apply",
manifest: bundle.manifest,
entities: bundle.entities,
}) as { run?: { id?: unknown } };
const runId = typeof run.run?.id === "string" ? run.run.id : undefined;
if (!runId) {
throw new Error("Remote upstream importer did not return a run id");
}
for (const chunk of bundle.chunks) {
await this.post(`/api/upstream-import-runs/${encodeURIComponent(runId)}/chunks`, chunk);
}
return this.post(`/api/upstream-import-runs/${encodeURIComponent(runId)}/apply`, {});
}
async events(runId: string): Promise<unknown> {
return this.get(`/api/upstream-import-runs/${encodeURIComponent(runId)}/events`);
}
private async get(path: string): Promise<unknown> {
const response = await this.#fetch(`${this.#targetOrigin}${path}`, {
method: "GET",
headers: await this.#headers({ method: "GET", path }),
});
return parseCoordinatorResponse(response);
}
private async post(path: string, body: unknown): Promise<unknown> {
const response = await this.#fetch(`${this.#targetOrigin}${path}`, {
method: "POST",
headers: {
"Content-Type": "application/json",
...(await this.#headers({ method: "POST", path })),
},
body: JSON.stringify(body),
});
return parseCoordinatorResponse(response);
}
}
export function buildLocalUpstreamExportBundle(
input: BuildLocalUpstreamExportBundleInput,
): LocalUpstreamExportBundle {
const entities = input.entities.map<LocalUpstreamExportEntity>((entity) => ({
record: {
key: entity.key,
contentHash: normalizedContentHash(entity.body),
dependencies: entity.dependencies ?? [],
warnings: entity.warnings ?? [],
},
body: entity.body,
conflictKeys: entity.conflictKeys,
}));
const chunks = buildLocalChunks(entities, input.maxEntitiesPerChunk ?? 100);
const manifestWithoutHash = {
schema: upstreamTransferSchema,
source: input.source,
target: input.target,
runId: input.runId,
idempotencyKey: input.idempotencyKey,
generatedAt: new Date(0).toISOString(),
entityCount: entities.length,
entities: entities.map((entity) => entity.record),
chunks: chunks.map(({ payload: _payload, ...chunk }) => chunk),
warnings: input.warnings ?? [],
featureFlags: (input.featureFlags ?? ["cloud_sync"]).slice().sort(),
};
const manifestHash = normalizedContentHash(manifestWithoutHash);
return {
manifest: {
...manifestWithoutHash,
chunks: manifestWithoutHash.chunks.map((chunk) => ({ ...chunk, manifestHash })),
manifestHash,
},
entities,
chunks,
};
}
export function normalizedContentHash(value: unknown): NormalizedSha256 {
return `sha256:${createHash("sha256").update(canonicalJson(value)).digest("hex")}`;
}
export function canonicalJson(value: unknown): string {
return JSON.stringify(sortJson(value));
}
function buildLocalChunks(
entities: LocalUpstreamExportEntity[],
maxEntitiesPerChunk: number,
): LocalUpstreamExportChunk[] {
if (!Number.isInteger(maxEntitiesPerChunk) || maxEntitiesPerChunk < 1) {
throw new Error("maxEntitiesPerChunk must be a positive integer");
}
if (entities.length === 0) return [];
const groups: LocalUpstreamExportEntity[][] = [];
for (let index = 0; index < entities.length; index += maxEntitiesPerChunk) {
groups.push(entities.slice(index, index + maxEntitiesPerChunk));
}
return groups.map((group, index) => {
const payload = {
entityKeys: group.map((entity) => entity.record.key),
};
return {
chunkIndex: index,
totalChunks: groups.length,
byteLength: Buffer.byteLength(canonicalJson(payload)),
sha256: normalizedContentHash(payload),
payload,
};
});
}
function sortJson(value: unknown): unknown {
if (Array.isArray(value)) return value.map(sortJson);
if (typeof value !== "object" || value === null) return value;
return Object.fromEntries(
Object.entries(value as Record<string, unknown>)
.sort(([left], [right]) => left.localeCompare(right))
.map(([key, entry]) => [key, sortJson(entry)]),
);
}
async function parseCoordinatorResponse(response: Response): Promise<unknown> {
const text = await response.text();
const parsed = text.trim() ? safeParseJson(text) : {};
if (!response.ok) {
const message = typeof parsed === "object" && parsed !== null && "error" in parsed
? String((parsed as { error: unknown }).error)
: `Upstream importer request failed with ${response.status}`;
throw new UpstreamImportRequestError(response.status, message, parsed);
}
return parsed;
}
function safeParseJson(text: string): unknown {
try {
return JSON.parse(text);
} catch {
return text;
}
}
cli/src/commands/client/cloud.ts
-722
View File
@@ -1,722 +0,0 @@
import { createHash, generateKeyPairSync, randomBytes, randomUUID, sign } from "node:crypto";
import { createServer, type Server } from "node:http";
import { URL } from "node:url";
import { Command } from "commander";
import pc from "picocolors";
import type {
CompanyPortabilityExportResult,
CompanyPortabilityFileEntry,
InstanceExperimentalSettings,
} from "@paperclipai/shared";
import { openUrl } from "../../client/board-auth.js";
import { resolvePaperclipInstanceId } from "../../config/home.js";
import {
addCommonClientOptions,
apiPath,
handleCommandError,
printOutput,
resolveCommandContext,
type BaseClientOptions,
} from "./common.js";
import {
buildLocalUpstreamExportBundle,
LocalUpstreamPushCoordinator,
normalizedContentHash,
upstreamTransferSchema,
UpstreamImportRequestError,
type LocalUpstreamExportBundle,
type LocalUpstreamExportEntityInput,
type SourceEntityKey,
type UpstreamTransferManifestSource,
type UpstreamTransferManifestTarget,
type UpstreamTransferWarning,
} from "./cloud-transfer.js";
import {
getCloudConnection,
upsertCloudConnection,
type CloudConnection,
type CloudConnectionTokenRecord,
} from "./cloud-store.js";
const CLOUD_SYNC_CONFLICT_EXIT_CODE = 2;
const CLOUD_SYNC_SCHEMA_MISMATCH_EXIT_CODE = 3;
const CLOUD_SYNC_SCOPES = ["upstream_import:preview", "upstream_import:write", "upstream_import:read"];
const DEVICE_CODE_FALLBACK_EXPIRES_MS = 15 * 60_000;
interface CloudConnectOptions extends BaseClientOptions {
noBrowser?: boolean;
}
interface CloudPushOptions extends BaseClientOptions {
company?: string;
remoteUrl?: string;
dryRun?: boolean;
maxEntitiesPerChunk?: number;
}
interface UpstreamDiscovery {
schema: string;
stack: {
id: string;
slug?: string;
displayName?: string;
companyId: string;
origin: string;
};
auth: {
pkce?: {
authorizeUrl: string;
tokenUrl: string;
codeChallengeMethod: string;
};
deviceCode?: {
deviceCodeUrl: string;
verificationUrl: string;
tokenUrl: string;
};
scopes?: string[];
};
transfer: {
supportedSchemaMajor: number;
featureFlags?: string[];
};
}
interface TokenResponse {
accessToken: string;
token: CloudConnectionTokenRecord;
scopes?: string[];
expiresAt?: string;
}
class CloudAuthRequestError extends Error {
readonly status: number;
readonly body: unknown;
constructor(status: number, message: string, body: unknown) {
super(message);
this.status = status;
this.body = body;
}
}
export function registerCloudCommands(program: Command): void {
const cloud = program.command("cloud").description("Paperclip Cloud upstream sync commands");
addCommonClientOptions(
cloud
.command("connect")
.description("Authorize this local instance to push into a Paperclip Cloud stack")
.argument("<remote-url>", "Paperclip Cloud stack URL")
.option("--no-browser", "Use the device-code flow instead of opening a browser", false)
.action(async (remoteUrl: string, opts: CloudConnectOptions) => {
try {
await connectCloud(remoteUrl, opts);
} catch (err) {
handleCommandError(err);
}
}),
);
addCommonClientOptions(
cloud
.command("push")
.description("Preview or apply a local company push into the connected Paperclip Cloud stack")
.requiredOption("--company <local-company-id>", "Local company ID to export")
.option("--remote-url <remote-url>", "Use a specific stored cloud connection")
.option("--dry-run", "Preview without applying", false)
.option("--max-entities-per-chunk <count>", "Chunk size for upstream uploads", (value) => Number(value), 100)
.action(async (opts: CloudPushOptions) => {
try {
await pushCloud(opts);
} catch (err) {
if (isSchemaMismatchError(err)) {
console.error(pc.red(err instanceof Error ? err.message : String(err)));
process.exitCode = CLOUD_SYNC_SCHEMA_MISMATCH_EXIT_CODE;
return;
}
handleCommandError(err);
}
}),
);
}
export async function connectCloud(remoteUrl: string, opts: CloudConnectOptions = {}): Promise<CloudConnection> {
const ctx = resolveCommandContext(opts);
const discovery = await discoverUpstream(remoteUrl);
assertDiscoveryCompatible(discovery);
const source = createSourceIdentity();
const token = await authorizeConnection(discovery, source, {
noBrowser: Boolean(opts.noBrowser),
});
const targetOrigin = discovery.stack.origin.replace(/\/+$/u, "");
const targetHost = new URL(targetOrigin).host;
const now = new Date().toISOString();
const connection = upsertCloudConnection({
id: connectionId(targetOrigin),
remoteUrl,
targetOrigin,
targetHost,
stackId: discovery.stack.id,
stackSlug: discovery.stack.slug ?? null,
stackDisplayName: discovery.stack.displayName ?? null,
targetCompanyId: discovery.stack.companyId,
accessToken: token.accessToken,
token: token.token,
privateKeyPem: source.privateKeyPem,
sourcePublicKey: source.sourcePublicKey,
sourceInstanceId: source.sourceInstanceId,
sourceInstanceFingerprint: source.sourceInstanceFingerprint,
scopes: token.scopes ?? token.token.scopes ?? CLOUD_SYNC_SCOPES,
createdAt: now,
updatedAt: now,
});
if (ctx.json) {
printOutput(redactConnection(connection), { json: true });
} else {
console.log(pc.bold("Connected to Paperclip Cloud"));
console.log(`stack=${connection.stackDisplayName ?? connection.stackSlug ?? connection.stackId}`);
console.log(`origin=${connection.targetOrigin}`);
console.log(`company=${connection.targetCompanyId}`);
}
return connection;
}
export async function pushCloud(opts: CloudPushOptions): Promise<unknown> {
const ctx = resolveCommandContext(opts, { requireCompany: false });
const localCompanyId = requiredString(opts.company, "--company");
await assertCloudSyncEnabled(ctx.api.get<InstanceExperimentalSettings>("/api/instance/settings/experimental"));
const connection = getCloudConnection(opts.remoteUrl);
if (!connection) {
throw new Error("No cloud connection found. Run `paperclipai cloud connect <remote-url>` first.");
}
const discovery = await discoverUpstream(connection.targetOrigin);
assertDiscoveryCompatible(discovery);
const bundle = await buildBundleFromLocalCompany({
localCompanyId,
connection,
discovery,
localApi: ctx.api,
maxEntitiesPerChunk: opts.maxEntitiesPerChunk,
mode: opts.dryRun ? "preview" : "apply",
});
const coordinator = new LocalUpstreamPushCoordinator({
targetOrigin: connection.targetOrigin,
paperclipCompanyId: connection.targetCompanyId,
headers: ({ method, path }) => cloudProofHeaders(connection, method, path),
});
const result = opts.dryRun ? await coordinator.preview(bundle) : await coordinator.apply(bundle);
const runId = getRunId(result);
const events = !opts.dryRun && runId ? await coordinator.events(runId).catch(() => null) : null;
const summary = summarizeResult(result);
const conflictCount = summary.conflict + summary.staleMapping;
if (ctx.json) {
printOutput({ result, events }, { json: true });
} else {
console.log(pc.bold(opts.dryRun ? "Cloud Push Preview" : "Cloud Push Applied"));
console.log(`run=${runId ?? "-"}`);
console.log(`manifest=${bundle.manifest.manifestHash}`);
console.log(
`create=${summary.create} update=${summary.update} adopt=${summary.adopt} ` +
`skip=${summary.skip} conflict=${summary.conflict} staleMapping=${summary.staleMapping}`,
);
printWarnings(result);
printConflicts(result);
printEvents(events);
}
if (conflictCount > 0) {
process.exitCode = CLOUD_SYNC_CONFLICT_EXIT_CODE;
}
return result;
}
export async function discoverUpstream(remoteUrl: string): Promise<UpstreamDiscovery> {
const base = new URL(remoteUrl);
const discoveryUrl = new URL("/.well-known/paperclip-upstream", base);
return requestCloudJson<UpstreamDiscovery>(discoveryUrl.toString(), { method: "GET" });
}
export function assertDiscoveryCompatible(discovery: UpstreamDiscovery): void {
if (discovery.schema !== "paperclip-upstream-discovery-v1") {
throw new Error("Remote URL is not a Paperclip Cloud upstream target.");
}
if (discovery.transfer.supportedSchemaMajor !== upstreamTransferSchema.major) {
throw new Error(
`Cloud upstream schema mismatch: local major ${upstreamTransferSchema.major}, remote supports ${discovery.transfer.supportedSchemaMajor}.`,
);
}
if (!discovery.transfer.featureFlags?.includes("cloud_sync")) {
throw new Error("Remote Paperclip Cloud stack does not advertise the cloud_sync transfer flag.");
}
}
export function resolveDeviceCodeExpiresAt(expiresAt: string | undefined, nowMs = Date.now()): number {
const parsed = typeof expiresAt === "string" ? Date.parse(expiresAt) : NaN;
return Number.isFinite(parsed) ? parsed : nowMs + DEVICE_CODE_FALLBACK_EXPIRES_MS;
}
export async function buildBundleFromLocalCompany(input: {
localCompanyId: string;
connection: CloudConnection;
discovery: UpstreamDiscovery;
localApi: {
post<T>(path: string, body?: unknown): Promise<T | null>;
};
maxEntitiesPerChunk?: number;
mode: "preview" | "apply";
}): Promise<LocalUpstreamExportBundle> {
const exported = await input.localApi.post<CompanyPortabilityExportResult>(
apiPath`/api/companies/${input.localCompanyId}/export`,
{
include: {
company: true,
agents: true,
projects: true,
issues: true,
skills: true,
},
expandReferencedSkills: true,
},
);
if (!exported) throw new Error("Local company export returned no data.");
const sourceHash = normalizedContentHash({
manifest: exported.manifest,
files: exported.files,
});
const source: UpstreamTransferManifestSource = {
sourceInstanceId: input.connection.sourceInstanceId,
sourceCompanyId: input.localCompanyId,
sourceInstanceKeyFingerprint: input.connection.sourceInstanceFingerprint,
exporterVersion: "paperclipai-cli-cloud-v1",
sourceSchemaVersion: "paperclip-local-portability-v1",
};
const target: UpstreamTransferManifestTarget = {
targetStackId: input.discovery.stack.id,
targetCompanyId: input.discovery.stack.companyId,
targetOrigin: input.discovery.stack.origin,
supportedSchemaMajor: input.discovery.transfer.supportedSchemaMajor,
};
const entities = buildEntitiesFromPortableExport(input.localCompanyId, input.connection.sourceInstanceId, exported);
const idempotencyKey = [
input.mode,
input.connection.sourceInstanceId,
input.localCompanyId,
input.discovery.stack.id,
sourceHash,
].join(":");
return buildLocalUpstreamExportBundle({
source,
target,
runId: `local-${input.mode}-${shortHash(idempotencyKey)}`,
idempotencyKey,
entities,
warnings: exported.warnings.map((message): UpstreamTransferWarning => ({
code: "local_company_export_warning",
severity: "warning",
message,
})),
featureFlags: ["cloud_sync"],
maxEntitiesPerChunk: input.maxEntitiesPerChunk,
});
}
async function authorizeConnection(
discovery: UpstreamDiscovery,
source: ReturnType<typeof createSourceIdentity>,
opts: { noBrowser: boolean },
): Promise<TokenResponse> {
if (!opts.noBrowser && canOpenBrowser() && discovery.auth.pkce) {
try {
return await authorizeWithBrowser(discovery, source);
} catch (error) {
console.error(pc.yellow(`Browser authorization failed; falling back to device-code flow. ${errorMessage(error)}`));
}
}
if (!discovery.auth.deviceCode) {
throw new Error("Remote Paperclip Cloud stack does not support device-code authorization.");
}
return authorizeWithDeviceCode(discovery, source, { openBrowser: !opts.noBrowser && canOpenBrowser() });
}
async function authorizeWithBrowser(
discovery: UpstreamDiscovery,
source: ReturnType<typeof createSourceIdentity>,
): Promise<TokenResponse> {
const pkce = discovery.auth.pkce;
if (!pkce) throw new Error("Remote did not advertise PKCE authorization.");
const callback = await startPkceCallbackServer();
const verifier = randomBytes(32).toString("base64url");
const challenge = createHash("sha256").update(verifier).digest("base64url");
const state = randomUUID();
const authorizeUrl = new URL(pkce.authorizeUrl);
authorizeUrl.searchParams.set("redirectUri", callback.redirectUri);
authorizeUrl.searchParams.set("state", state);
authorizeUrl.searchParams.set("codeChallenge", challenge);
authorizeUrl.searchParams.set("codeChallengeMethod", "S256");
authorizeUrl.searchParams.set("sourceInstanceId", source.sourceInstanceId);
authorizeUrl.searchParams.set("sourceInstanceFingerprint", source.sourceInstanceFingerprint);
authorizeUrl.searchParams.set("sourcePublicKey", source.sourcePublicKey);
authorizeUrl.searchParams.set("scopes", CLOUD_SYNC_SCOPES.join(" "));
try {
console.error(`Open this URL to approve cloud sync:\n${authorizeUrl.toString()}`);
if (!(await openUrl(authorizeUrl.toString()))) {
throw new Error("Could not open a browser.");
}
const code = await callback.waitForCode(state);
return requestCloudJson<TokenResponse>(pkce.tokenUrl, {
method: "POST",
body: JSON.stringify({
grantType: "authorization_code",
code,
redirectUri: callback.redirectUri,
codeVerifier: verifier,
}),
});
} finally {
await callback.close();
}
}
async function authorizeWithDeviceCode(
discovery: UpstreamDiscovery,
source: ReturnType<typeof createSourceIdentity>,
opts: { openBrowser: boolean },
): Promise<TokenResponse> {
const device = discovery.auth.deviceCode;
if (!device) throw new Error("Remote did not advertise device-code authorization.");
const response = await requestCloudJson<{
deviceCode: string;
userCode: string;
verificationUri: string;
expiresAt?: string;
intervalSeconds?: number;
}>(device.deviceCodeUrl, {
method: "POST",
body: JSON.stringify({
stackId: discovery.stack.id,
sourceInstanceId: source.sourceInstanceId,
sourceInstanceFingerprint: source.sourceInstanceFingerprint,
sourcePublicKey: source.sourcePublicKey,
scopes: CLOUD_SYNC_SCOPES,
}),
});
console.error(pc.bold("Cloud device authorization required"));
console.error(`Open: ${response.verificationUri}`);
console.error(`Code: ${response.userCode}`);
if (opts.openBrowser) await openUrl(response.verificationUri);
const expiresAt = resolveDeviceCodeExpiresAt(response.expiresAt);
const intervalMs = Math.max(500, (response.intervalSeconds ?? 5) * 1000);
while (Date.now() < expiresAt) {
await sleep(intervalMs);
try {
return await requestCloudJson<TokenResponse>(device.tokenUrl, {
method: "POST",
body: JSON.stringify({
grantType: "device_code",
deviceCode: response.deviceCode,
}),
});
} catch (error) {
if (error instanceof CloudAuthRequestError && error.body && typeof error.body === "object") {
const code = (error.body as { error?: unknown }).error;
if (code === "authorization_pending") continue;
}
throw error;
}
}
throw new Error("Device-code authorization expired before it was approved.");
}
function buildEntitiesFromPortableExport(
localCompanyId: string,
sourceInstanceId: string,
exported: CompanyPortabilityExportResult,
): LocalUpstreamExportEntityInput[] {
const companyKey: SourceEntityKey = {
sourceInstanceId,
sourceCompanyId: localCompanyId,
sourceEntityType: "company",
sourceEntityId: localCompanyId,
sourceNaturalKey: exported.manifest.company?.name ?? localCompanyId,
};
const entities: LocalUpstreamExportEntityInput[] = [
{
key: companyKey,
body: {
kind: "paperclip_company_portability_manifest",
manifest: exported.manifest,
rootPath: exported.rootPath,
paperclipExtensionPath: exported.paperclipExtensionPath,
fileCount: Object.keys(exported.files).length,
},
conflictKeys: [`company:${companyKey.sourceNaturalKey ?? localCompanyId}`],
},
];
for (const [filePath, entry] of Object.entries(exported.files).sort(([left], [right]) => left.localeCompare(right))) {
entities.push({
key: {
sourceInstanceId,
sourceCompanyId: localCompanyId,
sourceEntityType: "company_setting",
sourceEntityId: shortHash(filePath),
sourceNaturalKey: filePath,
},
body: {
kind: "paperclip_portable_file",
path: filePath,
entry: normalizePortableFileEntry(entry),
},
dependencies: [companyKey],
conflictKeys: [`portable_file:${filePath}`],
});
}
return entities;
}
function normalizePortableFileEntry(entry: CompanyPortabilityFileEntry): Record<string, unknown> {
if (typeof entry === "string") {
return { encoding: "utf8", data: entry };
}
return { ...entry };
}
async function assertCloudSyncEnabled(settingsPromise: Promise<InstanceExperimentalSettings | null>): Promise<void> {
const settings = await settingsPromise;
if (settings?.enableCloudSync !== true) {
throw new Error(
"Cloud sync is disabled. Enable the cloud sync experimental setting before running `paperclipai cloud push`.",
);
}
}
function cloudProofHeaders(connection: CloudConnection, method: string, pathAndSearch: string): Record<string, string> {
const timestamp = new Date().toISOString();
const nonce = randomUUID();
const payload = [
method,
connection.targetHost.toLowerCase(),
pathAndSearch,
connection.token.id,
connection.sourceInstanceId,
timestamp,
nonce,
].join("\n");
return {
Authorization: `Bearer ${connection.accessToken}`,
"X-Paperclip-Upstream-Source-Instance-Id": connection.sourceInstanceId,
"X-Paperclip-Upstream-Proof-Timestamp": timestamp,
"X-Paperclip-Upstream-Proof-Nonce": nonce,
"X-Paperclip-Upstream-Proof-Signature": sign(
null,
Buffer.from(payload, "utf8"),
connection.privateKeyPem,
).toString("base64url"),
};
}
async function requestCloudJson<T>(url: string, init: RequestInit): Promise<T> {
const headers = new Headers(init.headers);
headers.set("accept", "application/json");
if (init.body !== undefined && !headers.has("content-type")) {
headers.set("content-type", "application/json");
}
const response = await fetch(url, { ...init, headers });
const text = await response.text();
const parsed = text.trim() ? JSON.parse(text) as unknown : {};
if (!response.ok) {
const message = typeof parsed === "object" && parsed !== null && "error" in parsed
? String((parsed as { error: unknown }).error)
: `Cloud request failed with ${response.status}`;
throw new CloudAuthRequestError(response.status, message, parsed);
}
return parsed as T;
}
function createSourceIdentity() {
const { publicKey, privateKey } = generateKeyPairSync("ed25519");
const sourcePublicKey = publicKey.export({ type: "spki", format: "pem" }).toString();
const sourceInstanceFingerprint = `sha256:${createHash("sha256")
.update(publicKey.export({ type: "spki", format: "der" }))
.digest("hex")}`;
return {
sourceInstanceId: `paperclip-local-${resolvePaperclipInstanceId()}`,
sourceInstanceFingerprint,
sourcePublicKey,
privateKeyPem: privateKey.export({ type: "pkcs8", format: "pem" }).toString(),
};
}
async function startPkceCallbackServer(): Promise<{
redirectUri: string;
waitForCode: (state: string) => Promise<string>;
close: () => Promise<void>;
}> {
let resolveCode: ((code: string) => void) | null = null;
let rejectCode: ((error: Error) => void) | null = null;
let expectedState = "";
const codePromise = new Promise<string>((resolve, reject) => {
resolveCode = resolve;
rejectCode = reject;
});
const server = createServer((req, res) => {
const url = new URL(req.url ?? "/", "http://127.0.0.1");
const code = url.searchParams.get("code");
const state = url.searchParams.get("state");
if (!code || state !== expectedState) {
res.writeHead(400, { "Content-Type": "text/plain" });
res.end("Paperclip Cloud authorization failed. You can close this tab.");
rejectCode?.(new Error("Authorization callback was missing a valid code or state."));
return;
}
res.writeHead(200, { "Content-Type": "text/plain" });
res.end("Paperclip Cloud authorization complete. You can close this tab.");
resolveCode?.(code);
});
await listenOnLoopback(server);
const address = server.address();
if (typeof address !== "object" || !address?.port) {
throw new Error("Failed to start local authorization callback server.");
}
return {
redirectUri: `http://127.0.0.1:${address.port}/cloud/callback`,
waitForCode: (state: string) => {
expectedState = state;
return codePromise;
},
close: () => closeServer(server),
};
}
function listenOnLoopback(server: Server): Promise<void> {
return new Promise((resolve, reject) => {
server.once("error", reject);
server.listen(0, "127.0.0.1", () => {
server.off("error", reject);
resolve();
});
});
}
function closeServer(server: Server): Promise<void> {
return new Promise((resolve, reject) => {
server.close((error) => error ? reject(error) : resolve());
});
}
function canOpenBrowser(): boolean {
if (process.platform === "darwin" || process.platform === "win32") return true;
return Boolean(process.env.DISPLAY || process.env.WAYLAND_DISPLAY);
}
function summarizeResult(result: unknown): {
create: number;
update: number;
adopt: number;
skip: number;
conflict: number;
staleMapping: number;
} {
const summary = asRecord(asRecord(result)?.summary);
return {
create: numberValue(summary?.create),
update: numberValue(summary?.update),
adopt: numberValue(summary?.adopt),
skip: numberValue(summary?.skip),
conflict: numberValue(summary?.conflict),
staleMapping: numberValue(summary?.staleMapping),
};
}
function printWarnings(result: unknown): void {
const warnings = Array.isArray(asRecord(result)?.warnings) ? asRecord(result)?.warnings as unknown[] : [];
for (const warning of warnings) {
const record = asRecord(warning);
console.log(pc.yellow(`warning=${record?.code ?? "warning"} ${record?.message ?? ""}`.trim()));
}
}
function printConflicts(result: unknown): void {
const conflicts = Array.isArray(asRecord(result)?.conflicts) ? asRecord(result)?.conflicts as unknown[] : [];
for (const conflict of conflicts.slice(0, 10)) {
const record = asRecord(conflict);
console.log(pc.red(`conflict=${record?.conflictKind ?? "target_conflict"} target=${record?.targetEntityId ?? "-"}`));
}
if (conflicts.length > 10) console.log(pc.red(`conflicts_truncated=${conflicts.length - 10}`));
}
function printEvents(events: unknown): void {
const rows = Array.isArray(asRecord(events)?.events) ? asRecord(events)?.events as unknown[] : [];
for (const row of rows.slice(-10)) {
const event = asRecord(row);
console.log(pc.dim(`event=${event?.action ?? "-"} target=${event?.targetEntityId ?? "-"}`));
}
}
function getRunId(result: unknown): string | null {
const run = asRecord(asRecord(result)?.run);
return typeof run?.id === "string" ? run.id : null;
}
function redactConnection(connection: CloudConnection): Record<string, unknown> {
return {
id: connection.id,
remoteUrl: connection.remoteUrl,
targetOrigin: connection.targetOrigin,
stackId: connection.stackId,
targetCompanyId: connection.targetCompanyId,
scopes: connection.scopes,
expiresAt: connection.token.expiresAt,
};
}
function connectionId(targetOrigin: string): string {
return `cloud-${shortHash(targetOrigin)}`;
}
function shortHash(value: string): string {
return createHash("sha256").update(value).digest("hex").slice(0, 16);
}
function requiredString(value: unknown, label: string): string {
if (typeof value === "string" && value.trim()) return value.trim();
throw new Error(`${label} is required.`);
}
function numberValue(value: unknown): number {
return typeof value === "number" && Number.isFinite(value) ? value : 0;
}
function asRecord(value: unknown): Record<string, unknown> | null {
return typeof value === "object" && value !== null && !Array.isArray(value)
? value as Record<string, unknown>
: null;
}
function isSchemaMismatchError(error: unknown): boolean {
if (error instanceof UpstreamImportRequestError) {
return JSON.stringify(error.body).toLowerCase().includes("schema");
}
return error instanceof Error && error.message.toLowerCase().includes("schema mismatch");
}
function errorMessage(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms));
}
export const cloudCommandExitCodes = {
conflict: CLOUD_SYNC_CONFLICT_EXIT_CODE,
schemaMismatch: CLOUD_SYNC_SCHEMA_MISMATCH_EXIT_CODE,
} as const;
cli/src/commands/client/company.ts
+3 -13
View File
@@ -15,7 +15,7 @@ import type {
import { getTelemetryClient, trackCompanyImported } from "../../telemetry.js";
import { ApiRequestError } from "../../client/http.js";
import { openUrl } from "../../client/board-auth.js";
import { binaryContentTypeByExtension, readZipArchive } from "./zip.js";
import { binaryContentTypeByExtension, bytesToPortableFileEntry, isBlobStorePath, readZipArchive } from "./zip.js";
import {
addCommonClientOptions,
apiPath,
@@ -140,16 +140,6 @@ type ImportSelectionState = {
skills: Set<string>;
};
function readPortableFileEntry(filePath: string, contents: Buffer): CompanyPortabilityFileEntry {
const contentType = binaryContentTypeByExtension[path.extname(filePath).toLowerCase()];
if (!contentType) return contents.toString("utf8");
return {
encoding: "base64",
data: contents.toString("base64"),
contentType,
};
}
function portableFileEntryToWriteValue(entry: CompanyPortabilityFileEntry): string | Uint8Array {
if (typeof entry === "string") return entry;
return Buffer.from(entry.data, "base64");
@@ -213,7 +203,7 @@ function shouldIncludePortableFile(filePath: string): boolean {
const isMarkdown = baseName.endsWith(".md");
const isPaperclipYaml = baseName === ".paperclip.yaml" || baseName === ".paperclip.yml";
const contentType = binaryContentTypeByExtension[path.extname(baseName).toLowerCase()];
return isMarkdown || isPaperclipYaml || Boolean(contentType);
return isMarkdown || isPaperclipYaml || Boolean(contentType) || isBlobStorePath(filePath);
}
function findPortableExtensionPath(files: Record<string, CompanyPortabilityFileEntry>): string | null {
@@ -932,7 +922,7 @@ async function collectPackageFiles(
if (!entry.isFile()) continue;
const relativePath = path.relative(root, absolutePath).replace(/\\/g, "/");
if (!shouldIncludePortableFile(relativePath)) continue;
files[relativePath] = readPortableFileEntry(relativePath, await readFile(absolutePath));
files[relativePath] = bytesToPortableFileEntry(relativePath, await readFile(absolutePath));
}
}
cli/src/commands/client/zip.ts
+10 -129
View File
@@ -1,129 +1,10 @@
import { inflateRawSync } from "node:zlib";
import path from "node:path";
import type { CompanyPortabilityFileEntry } from "@paperclipai/shared";
const textDecoder = new TextDecoder();
export const binaryContentTypeByExtension: Record<string, string> = {
".gif": "image/gif",
".jpeg": "image/jpeg",
".jpg": "image/jpeg",
".png": "image/png",
".svg": "image/svg+xml",
".webp": "image/webp",
};
function normalizeArchivePath(pathValue: string) {
return pathValue
.replace(/\\/g, "/")
.split("/")
.filter(Boolean)
.join("/");
}
function readUint16(source: Uint8Array, offset: number) {
return source[offset]! | (source[offset + 1]! << 8);
}
function readUint32(source: Uint8Array, offset: number) {
return (
source[offset]! |
(source[offset + 1]! << 8) |
(source[offset + 2]! << 16) |
(source[offset + 3]! << 24)
) >>> 0;
}
function sharedArchiveRoot(paths: string[]) {
if (paths.length === 0) return null;
const firstSegments = paths
.map((entry) => normalizeArchivePath(entry).split("/").filter(Boolean))
.filter((parts) => parts.length > 0);
if (firstSegments.length === 0) return null;
const candidate = firstSegments[0]![0]!;
return firstSegments.every((parts) => parts.length > 1 && parts[0] === candidate)
? candidate
: null;
}
function bytesToPortableFileEntry(pathValue: string, bytes: Uint8Array): CompanyPortabilityFileEntry {
const contentType = binaryContentTypeByExtension[path.extname(pathValue).toLowerCase()];
if (!contentType) return textDecoder.decode(bytes);
return {
encoding: "base64",
data: Buffer.from(bytes).toString("base64"),
contentType,
};
}
async function inflateZipEntry(compressionMethod: number, bytes: Uint8Array) {
if (compressionMethod === 0) return bytes;
if (compressionMethod !== 8) {
throw new Error("Unsupported zip archive: only STORE and DEFLATE entries are supported.");
}
return new Uint8Array(inflateRawSync(bytes));
}
export async function readZipArchive(source: ArrayBuffer | Uint8Array): Promise<{
rootPath: string | null;
files: Record<string, CompanyPortabilityFileEntry>;
}> {
const bytes = source instanceof Uint8Array ? source : new Uint8Array(source);
const entries: Array<{ path: string; body: CompanyPortabilityFileEntry }> = [];
let offset = 0;
while (offset + 4 <= bytes.length) {
const signature = readUint32(bytes, offset);
if (signature === 0x02014b50 || signature === 0x06054b50) break;
if (signature !== 0x04034b50) {
throw new Error("Invalid zip archive: unsupported local file header.");
}
if (offset + 30 > bytes.length) {
throw new Error("Invalid zip archive: truncated local file header.");
}
const generalPurposeFlag = readUint16(bytes, offset + 6);
const compressionMethod = readUint16(bytes, offset + 8);
const compressedSize = readUint32(bytes, offset + 18);
const fileNameLength = readUint16(bytes, offset + 26);
const extraFieldLength = readUint16(bytes, offset + 28);
if ((generalPurposeFlag & 0x0008) !== 0) {
throw new Error("Unsupported zip archive: data descriptors are not supported.");
}
const nameOffset = offset + 30;
const bodyOffset = nameOffset + fileNameLength + extraFieldLength;
const bodyEnd = bodyOffset + compressedSize;
if (bodyEnd > bytes.length) {
throw new Error("Invalid zip archive: truncated file contents.");
}
const rawArchivePath = textDecoder.decode(bytes.slice(nameOffset, nameOffset + fileNameLength));
const archivePath = normalizeArchivePath(rawArchivePath);
const isDirectoryEntry = /\/$/.test(rawArchivePath.replace(/\\/g, "/"));
if (archivePath && !isDirectoryEntry) {
const entryBytes = await inflateZipEntry(compressionMethod, bytes.slice(bodyOffset, bodyEnd));
entries.push({
path: archivePath,
body: bytesToPortableFileEntry(archivePath, entryBytes),
});
}
offset = bodyEnd;
}
const rootPath = sharedArchiveRoot(entries.map((entry) => entry.path));
const files: Record<string, CompanyPortabilityFileEntry> = {};
for (const entry of entries) {
const normalizedPath =
rootPath && entry.path.startsWith(`${rootPath}/`)
? entry.path.slice(rootPath.length + 1)
: entry.path;
if (!normalizedPath) continue;
files[normalizedPath] = entry.body;
}
return { rootPath, files };
}
// The node-side portability zip reader lives in @paperclipai/shared so the
// server can consume the same codec (a raw uploaded zip is unzipped into the
// exact `{ rootPath, files }` bundle the inline import source carries). This
// module re-exports it to keep the CLI's existing import paths stable.
export {
binaryContentTypeByExtension,
bytesToPortableFileEntry,
isBlobStorePath,
readZipArchive,
} from "@paperclipai/shared/portability-zip";
cli/src/commands/worktree-lib.ts
+2 -1
View File
@@ -197,7 +197,7 @@ export function buildWorktreeConfig(input: {
embeddedPostgresDataDir: paths.embeddedPostgresDataDir,
embeddedPostgresPort: databasePort,
backup: {
enabled: source?.database.backup.enabled ?? true,
enabled: false,
intervalMinutes: source?.database.backup.intervalMinutes ?? 60,
retentionDays: source?.database.backup.retentionDays ?? 30,
dir: paths.backupDir,
@@ -258,6 +258,7 @@ export function buildWorktreeEnvEntries(
PAPERCLIP_CONFIG: paths.configPath,
PAPERCLIP_CONTEXT: paths.contextPath,
PAPERCLIP_IN_WORKTREE: "true",
PAPERCLIP_DB_BACKUP_ENABLED: "false",
...(branding?.name ? { PAPERCLIP_WORKTREE_NAME: branding.name } : {}),
...(branding?.color ? { PAPERCLIP_WORKTREE_COLOR: branding.color } : {}),
};
cli/src/index.ts
-2
View File
@@ -22,7 +22,6 @@ import { registerRoutineCommands } from "./commands/routines.js";
import { registerPipelineCommands } from "./commands/pipelines.js";
import { registerFeedbackCommands } from "./commands/client/feedback.js";
import { registerSecretCommands } from "./commands/client/secrets.js";
import { registerCloudCommands } from "./commands/client/cloud.js";
import { registerSkillsCommands } from "./commands/client/skills.js";
import { registerTeamCommands } from "./commands/client/teams.js";
import { applyDataDirOverride, type DataDirOptionLike } from "./config/data-dir.js";
@@ -221,7 +220,6 @@ registerRoutineCommands(program);
registerPipelineCommands(program);
registerFeedbackCommands(program);
registerSecretCommands(program);
registerCloudCommands(program);
registerSkillsCommands(program);
registerTeamCommands(program);
registerWorktreeCommands(program);
doc/DEVELOPING.md
+7
View File
@@ -355,6 +355,7 @@ This command:
- creates an isolated instance under `~/.paperclip-worktrees/instances/<worktree-id>/`
- when run inside a linked git worktree, mirrors the effective git hooks into that worktree's private git dir
- picks a free app port and embedded PostgreSQL port
- disables automatic database backups for the isolated instance
- by default seeds the isolated DB in `minimal` mode from the current effective Paperclip instance/config (repo-local worktree config when present, otherwise the default instance) via a logical SQL snapshot
Seed modes:
@@ -374,6 +375,7 @@ Provisioned git worktrees also pause seeded routines that still have enabled sch
That repo-local env also sets:
- `PAPERCLIP_IN_WORKTREE=true`
- `PAPERCLIP_DB_BACKUP_ENABLED=false`
- `PAPERCLIP_WORKTREE_NAME=<worktree-name>`
- `PAPERCLIP_WORKTREE_COLOR=<hex-color>`
@@ -655,6 +657,11 @@ schemas. Defaults:
- retain 30 days
- backup dir: `~/.paperclip/instances/default/data/backups`
Automatic backups are disabled for isolated worktree instances created with
`paperclipai worktree init` or `paperclipai worktree:make`. Existing worktree
configs are migrated to the disabled setting when their server next starts. The
main/default instance keeps the normal enabled-by-default behavior.
Configure these in:
```sh
doc/LOW-TRUST-PRESETS.md
+11
View File
@@ -41,6 +41,17 @@ Low-trust agents cannot read or mutate agent configuration, instruction bundles,
or company skill configuration through direct grants. Configuration changes from
low-trust work must go through higher-trust review and promotion paths instead.
## Child→Parent Reporting Under Containment
The direct-parent report comment (`doc/execution-semantics.md` §6, "Child→Parent
Reporting") is **off by default** for `low_trust_review`: a contained run reads
untrusted input, so a free-prose comment into the higher-trust parent thread is
a prompt-injection promotion path. Contained reviewers report by completing
their own review issue (`done` — the verdict is the deliverable; the
`issue_blockers_resolved` wake carries it upward) and by the platform's
system-attributed stop-only relay when they enter `blocked` or `cancelled`.
Never instruct a contained delegate to comment on its parent issue.
## Runtime Containment
Managed `low_trust_review` runs fail closed unless Paperclip can enforce the
doc/SPEC-implementation.md
+17
View File
@@ -625,6 +625,21 @@ Within the watched subtree, a watchdog run may perform only mutations that resto
Every watchdog-triggered mutation must write activity with the watchdog id, source issue id, watchdog issue id when present, run id, and stop fingerprint. Mutations still use the normal status-transition, blocker, assignment, budget, and company-boundary guards.
### Atomic recovery batch
A watchdog run may submit an atomic recovery batch of at most 3 mutations drawn from the allowed-mutation list above, validated against the stop fingerprint that run observed. The server applies the batch all-or-nothing: if the subtree's stop fingerprint changed between observation and application — the subtree went live concurrently — the entire remainder of the batch is aborted and the staleness is recorded as evidence on the reusable watchdog issue. The batch is single-shot per watchdog run. This replaces the exactly-one-fresh-write model: the stale-guard's purpose (never mutate a subtree that concurrently went live) is preserved by fingerprint validation on the whole batch rather than by capping the run at one write, so a restoration that needs both a state-restoring `PATCH` and an explanatory comment cannot forfeit the restoration by ordering the comment first.
### Restoration verification and escalation
Reviewed-fingerprint suppression is disposition-aware. A watchdog disposition of "stopped state is legitimate" suppresses re-fire for that fingerprint as today. A disposition of "live path restored" arms a bounded verification instead:
- if a later scan observes the subtree stopped with a fingerprint equal to the one the restoration claimed to fix, the watchdog re-fires with an incremented attempt count for that fingerprint lineage
- restoration-attempt lineage (attempt number, claimed-fixed fingerprint, restoration actions) is persisted durable watchdog state
- the stop fingerprint (or the lineage check) must account for intermediate-node durable updates so a restoration that changes no stopped leaf is classified as a failed restoration, not as a reviewed stop
- after N attempts (N = 2–3, configuration-bounded) on the same fingerprint lineage, the platform stops re-firing and escalates to a human — the watchdog owner or a board notification — with the attempt history attached
Escalation is terminal for the automatic loop: no further watchdog wakes fire for that lineage until a human or a new durable subtree change produces a different fingerprint.
### Disallowed watchdog mutations
A task watchdog must not:
@@ -668,6 +683,8 @@ Implementation, security, UI, and QA work for task watchdogs must prove these co
- plan-confirmation tests cover stale document revisions, missing purpose markers, outside-subtree targets, governed actions, newer user comments, and explicit human/CTO/Security reservations
- scheduler tests prove live runs, queued wakes, and scheduled retries suppress watchdog wakeups, while terminal, cancelled, blocked, and review leaves are still verified when the subtree has no live path
- tests prove `task_watchdog` origin issues and descendants are excluded from scans so watchdogs do not trigger themselves
- recovery-batch tests prove batches are capped at 3 allowed mutations, applied all-or-nothing, and aborted with recorded evidence when the observed stop fingerprint went stale mid-batch
- restoration-verification tests prove a "live path restored" disposition re-fires on an unchanged fingerprint with an incremented attempt count, a failed intermediate-node restoration is not treated as a reviewed stop, and the N-attempt bound escalates to a human with attempt history instead of firing forever
- regression tests prove watchdog capability discovery comes from wake metadata/denials and denied probes do not create visible issues
- UI copy and badges distinguish task watchdogs from active-run output watchdogs, monitors, reviewers, approvers, and liveness recovery
- prompt/context tests prove custom instructions are appended after non-overridable safety constraints and cannot expand authority
doc/connections/CONNECTOR-PLAYBOOK.md
+2
View File
@@ -4,6 +4,8 @@ This playbook is the repeatable template for adding a vendor to the Apps catalog
Use it when Paperclip acts on an external system through a governed connection: a stored credential, a capability catalog, access profiles and policy rules, and audit. Inbound integrations, such as an external client acting on Paperclip, use gateway or webhook guidance instead.
Every connector built with this playbook is a **plane P2** connection — a resource token in the instance vault, acquired via the connect broker, never a sign-in authenticator. Before writing a connector, read [Identity vs. connections](./README.md#identity-vs-connections) for the P1/P2/P3 boundary and the D7 standing rule (sign-in tokens are never reused as resource tokens; id.paperclip.ing never stores resource tokens; no connections hub on the ID service).
## Output
A complete connector proposal produces:
doc/connections/FIRST-30-MATRIX.md
+5
View File
@@ -9,6 +9,11 @@ review gates for a provider before writing a connector ticket or playbook entry.
Source: harvested from [PAP-2432](/PAP/issues/PAP-2432) and made canonical for
the [PAP-13211](/PAP/issues/PAP-13211) Apps v2 unification program.
> Every provider here is a **plane P2** connection: a resource token in the
> instance vault, acquired via the connect broker — never a sign-in method. See
> [Identity vs. connections](./README.md#identity-vs-connections) for the P1/P2/P3
> boundary and the D7 standing rule.
## Batch Recommendation
First implementation batch after the proof providers:
doc/connections/README.md
+55
View File
@@ -69,6 +69,59 @@ The agent should not hold a durable provider credential. It should hold a
Paperclip run/session token; the server or broker resolves the connection,
checks governance, invokes the provider, and writes audit.
## Identity vs. connections
Signing a user *in* and connecting a *resource* are different planes with
different owners, different token profiles, and different homes. Do not merge
them. This section is the public, connections-side statement of the identity
model so connector implementers inherit the rule without depending on private
identity-service documentation or re-deriving it.
| Plane | Question | Lives where | Token profile |
| --- | --- | --- | --- |
| **P1. Sign-in methods** | *Who are you?* | `paperclip-id` (id.paperclip.ing → Account) | Minimal-scope provider tokens (`openid email profile`), used once to authenticate, encrypted at rest, never exported |
| **P2. Connections (Apps)** | *What may your agents touch?* | Paperclip App instances (`tool_connections`), acquired via the **connect broker** for hosted + self-hosted | Rich-scope, long-lived resource tokens in the **instance's** encrypted vault; per-agent grants; ask-first on writes |
| **P3. Login with Paperclip** | *Who may authenticate against us?* | `paperclip-id` OIDC provider + DB-backed client registry | Our ES256 ID/access tokens issued *by* us to registered RPs (instances, the broker, future third parties) |
Everything in `doc/connections/` — the [First-30 matrix](./FIRST-30-MATRIX.md),
the [connector playbook](./CONNECTOR-PLAYBOOK.md), and the connect-broker work —
lives on **plane P2**. It never acquires, stores, or brokers a P1 sign-in token.
### The standing rule (D7)
Adopted as a standing rule (decision D7) with the identity-model plan. State it
verbatim in any P2 design so the app-store work cannot drift into merging the
planes:
> Sign-in tokens are never reused as resource tokens; id.paperclip.ing never
> stores resource tokens; no connections hub on the ID service.
P2 tokens flow broker → instance vault as pass-through only; the id.paperclip.ing
Account page therefore must **not** grow a "Connections" hub. The reasons to
hold the planes apart (from the plan §3):
- **Scope discipline.** Sign-in wants the narrowest grant; connections want
deliberately broad ones. One button that does both is how you grant repo
access just to log in.
- **Blast radius.** id.paperclip.ing holding every customer's Vercel/Slack/GitHub
resource tokens would make it the single juiciest target in the fleet; the
broker is intentionally pass-through.
- **Self-hosted symmetry.** Instances own their vaults, so self-hosters don't
depend on our uptime to *use* their own connections.
- **Legibility.** Sign-in and connections answer different user questions, and
every product we benchmarked (Vercel, Railway, GitHub, Google) keeps them on
separate pages with separate names.
### Naming alignment
Use the surface-correct name for each plane; they intentionally differ:
| Surface | Plane | Name to use |
| --- | --- | --- |
| Paperclip App instances | P2 | **"Connections"** |
| id.paperclip.ing Account | P1 | **"Ways to sign in"** |
| id.paperclip.ing admin | P3 | **"OIDC clients"** (until the app store productizes it) |
## Packaging Rule
Default to a **catalog entry** when an integration can be described as metadata:
@@ -85,6 +138,8 @@ not own durable tokens.
## Canonical Docs
- [Glossary](./GLOSSARY.md) defines product and internal terms.
- [Identity vs. connections](#identity-vs-connections) is the public statement
of the P1/P2/P3 boundary and the D7 standing rule for connections work.
- [Security threat model](./SECURITY-THREAT-MODEL.md) harvests the keeper from
[PAP-2359](/PAP/issues/PAP-2359) and maps it onto Apps v2.
- [First-30 matrix](./FIRST-30-MATRIX.md) harvests the keeper from
doc/execution-semantics.md
+56 -1
View File
@@ -1,7 +1,7 @@
# Execution Semantics
Status: Current implementation guide
Date: 2026-06-10
Date: 2026-07-23
Audience: Product and engineering
This document explains how Paperclip interprets issue assignment, issue status, execution runs, wakeups, parent/sub-issue structure, and blocker relationships.
@@ -70,6 +70,18 @@ This is the right state for:
- waiting on an external dependency or system when Paperclip does not own a scheduled re-check
- work that automatic recovery could not safely continue
Entering `blocked` requires a routable waiting path. An issue may transition into `blocked` only with at least one of:
- first-class blockers (`blockedByIssueIds`)
- a pending issue-thread interaction or linked approval that names the responder
- a structured unblock descriptor naming `{owner, action}`, where `owner` is an agent id, user id, or the board, and `action` is the concrete step that unblocks the issue
When a structured unblock descriptor is the waiting path, Paperclip immediately notifies the named owner: an agent owner gets a wake, a user or board owner gets an inbox notification. Prose-only blocked — free-text that names an owner or action in a comment without any of the paths above — routes to nobody. It is rejected at the API or auto-classified as `needs_attention` with a board notification, never silently accepted as a healthy waiting state.
A permission denial is not, by itself, a blocker. If an instructed step is denied at an authorization boundary but the issue's own deliverable is complete, the right disposition is `done`, not `blocked` (see the review-delegation rules in §6).
This requirement is prospective-only on rollout: it applies to transitions into `blocked` made after the feature ships, gated on the blocked-transition timestamp against the rollout marker, not on issue `createdAt`. Issues already blocked at upgrade time are untouched — no backfilled notifications, no retroactive validation, no `needs_attention` storm on deploy. Triage of pre-existing prose-blocked issues is a one-time opt-in digest, not a default.
### `in_review`
Execution work is paused because the next move belongs to a reviewer or approver, not the current executor.
@@ -173,6 +185,24 @@ Blocked issues should stay idle while blockers remain unresolved. Paperclip shou
If a parent is truly waiting on a child, model that with blockers. Do not rely on the parent/child relationship alone.
### Child→Parent Reporting
Run-scoped write authorization is subtree-scoped: a run may mutate its checked-out issue and that issue's descendants. Delegated child work still has to report upward, so the platform provides exactly three canonical report channels. Nothing else crosses the boundary.
1. **Completion signal (always on).** When a child that blocks its parent reaches `done`, the `issue_blockers_resolved` wake engages the parent's assignee, who can read the child thread. Completion needs no report comment: the child's own thread is the deliverable record, and relaying `done` as prose would duplicate the first-class wake.
2. **Direct-parent report comment (trust-gated).** The write boundary widens exactly one hop upward: a run checked out on a child issue may POST comments on the child's **direct parent** — comments only (no status, field, assignment, or document writes), the direct parent only (never grandparents or siblings, never lateral). This is gated per trust preset: **on** for `standard`, **off by default** for `low_trust_review` and other review-contained presets, whose input is untrusted content (diffs, external tickets) and whose report comment would be a prompt-injection carrier into higher-trust context.
3. **Stop-only relay (fallback where the report comment is off).** For presets with direct-parent commenting disabled, the platform delivers a system-attributed relay comment to the direct parent when the child transitions into `blocked` or `cancelled` — never on `done` or `in_review`. Stopping is the event the parent must hear about; completion already has the first-class signal above. A relay is a comment, not a disposition transition, so it can never trigger another relay (depth-1 by construction), and relays dedupe per (child, target status) so status flapping cannot spam the parent.
### Review Delegation
Review tasks — security reviews, code reviews, QA verdicts — must instruct the delegate to post findings on **their own review issue** and mark it `done`. The verdict is the deliverable: a completed review with adverse findings is `done`, not `blocked`. The parent's owner is engaged by `issue_blockers_resolved` (plus the direct-parent report comment where the reviewer's preset allows it) and owns any follow-up fixes.
Never instruct a low-trust delegate to comment on the parent issue. That instruction is guaranteed to be denied at the authorization boundary, and a reviewer that converts the denial into a `blocked` disposition with a prose-only owner strands the whole tree indefinitely.
### The Courier Pattern (Lateral Coordination)
The direct-parent report comment intentionally does not open lateral comment access: no writes into sibling subtrees or other agents' boundaries. The sanctioned lateral channel is the **courier pattern**: create a new issue assigned to the target agent that carries the complete instructions and context in its description (company-scoped issue-CREATE is permitted from any run). The courier issue wakes the target agent through normal assignment, keeps the coordination auditable, and avoids widening comment access into another agent's boundary. Because the target agent's run may not be able to read your issues, the courier description must be self-contained — do not rely on links back into your own subtree for essential instructions.
## 7. Accepted-Plan Decomposition
An accepted plan confirmation is permission to decompose one specific accepted plan revision into child issues.
@@ -489,6 +519,11 @@ Recovery rule:
This is a dispatch recovery, not a continuation recovery.
Recovery hand-back is covered by the same liveness guarantee:
- an `issue_recovery_action_restored` wake requested while the resolving recovery run is still active is persisted as a follow-up and dispatched only after that run exits, so it cannot be coalesced into the run that requested it
- if that follow-up is nevertheless lost, the stranded-work backstop treats an assigned `todo` issue with a resolved `handed_back` recovery action from during or after its latest successful run as stranded and queues the bounded assignment recovery wake; the successful resolving run is not, by itself, evidence that the handed-back source work is live
### 9.2 Stranded assigned `in_progress`
Example:
@@ -596,6 +631,12 @@ The watchdog should verify stopped leaves against comments, documents, work prod
When work should continue, the watchdog restores a live path inside the watched subtree: reopen or reassign stuck work, create follow-up issues, repair blockers, set a monitor, or resolve an eligible plan confirmation. When the stopped state is legitimate, the watchdog records why and leaves the subtree with a valid terminal, waiting, blocked, review, or explicit recovery path.
### Atomic recovery batch
Restoration is often more than one write — reopen the dead-end leaf **and** explain it on the parent. A watchdog run may submit a small atomic recovery batch: at most 3 mutations from the allowed-mutation list, validated against the stop fingerprint the run observed. The server applies the batch all-or-nothing and aborts the remainder if the subtree fingerprint changed mid-batch because work went live concurrently.
This preserves the stale-guard's purpose — never keep mutating a subtree that just went live under the watchdog's feet — while removing the failure mode where spending the only permitted write on an informational comment forfeits the state-restoring mutation the recovery actually needed.
### Eligible interaction decisions
A task watchdog may resolve only eligible `request_confirmation` plan confirmations. Eligibility is defined in `doc/SPEC-implementation.md` and must be checked by the server at decision time. The critical constraints are:
@@ -618,6 +659,20 @@ The watchdog's reviewed fingerprint should update only after the watchdog issue
If the watchdog moved work forward, Paperclip should not mark the old fingerprint as permanently acceptable just because the watchdog issue completed. The next scan should observe the changed subtree state and either suppress because work is live or compute a new stopped fingerprint later.
### Restoration verification and escalation
A reviewed fingerprint suppresses re-fire only when the watchdog's recorded disposition was "this stopped state is legitimate." A **"live path restored"** disposition does not earn permanent suppression; it arms a bounded verification instead.
The stop fingerprint must be computed from durable subtree state such that a restoration which changes nothing observable is detectable as a failed restoration. In particular, activity on intermediate (non-leaf) nodes — comments, wakes delivered to an intermediate issue's assignee, runs that end without mutating any stopped leaf — must feed the fingerprint or the attempt lineage; a fingerprint derived from stopped leaves alone makes a failed intermediate-node restoration byte-identical to a reviewed-and-acceptable stop, which silences the watchdog forever.
Verification and escalation semantics:
- if, after a "live path restored" disposition, the subtree is observed stopped again with a fingerprint equal to the one the watchdog claimed to have fixed, the restoration failed and the watchdog re-fires with an incremented attempt count
- attempt lineage is durable watchdog state: attempt number, the fingerprint each attempt claimed to fix, and the restoration actions taken
- after N attempts (N = 2–3) on the same fingerprint lineage, the platform stops re-firing and escalates to a human — the watchdog owner or a board notification — carrying the attempt history
This bounds both failure modes: a failed restoration retries instead of going silent (liveness), and the attempt bound plus human escalation prevents an infinite fire loop (no runaway watchdog). Legitimate stops still suppress exactly as before.
Task watchdogs must not silently mark source work done from prose comments, must not duplicate child trees for the same accepted plan revision, and must not create another task-watchdog issue for the same source issue.
## 12. Silent Active-Run Watchdog
docker/agent-runtime/README.md
+8
View File
@@ -21,6 +21,14 @@ Container images for running coding-agent harnesses in sandboxed environments (f
- tini (PID-1 init, ensures signal propagation)
- Non-root user `paperclip` (uid/gid 1000)
The NodeSource install puts `node` on the default `PATH`. The agent shim in this
image runs the harness directly with that `PATH`. The shim does not source a
login profile, and the runtime never writes a profile or an rc file. Some
sandbox providers instead wrap each command in a login shell. That shell sources
`/etc/profile` and the user profile files to read an owner-supplied `PATH`. No
exec path sources `nvm`. For the full exec-path contract, see
`packages/plugins/sandbox-providers/SANDBOX-REQUIREMENTS.md`.
**Paperclip Binaries:**
- `/usr/local/bin/paperclip-agent-shim`: Go binary compiled from `tools/agent-shim/`. Reads `/run/paperclip/runtime-command.json` and `syscall.Exec`s the harness CLI.
docs/api/issues.md
+1 -1
View File
@@ -62,7 +62,7 @@ Headers: X-Paperclip-Run-Id: {runId}
}
```
The optional `comment` field adds a comment in the same call.
The optional `comment` field adds a comment in the same call. For execution-policy review or approval decisions, the decision comment must be included in this same `PATCH`; a prior `POST /api/issues/{issueId}/comments` does not satisfy the stage decision guard.
Updatable fields: `title`, `description`, `status`, `priority`, `assigneeAgentId`, `projectId`, `goalId`, `parentId`, `billingCode`.
docs/guides/agent-developer/comments-and-communication.md
+2
View File
@@ -19,6 +19,8 @@ PATCH /api/issues/{issueId}
{ "status": "done", "comment": "Implemented login endpoint with JWT auth." }
```
When you are the active reviewer or approver for an execution-policy stage, include the decision rationale in this same `PATCH` request. A separate `POST /api/issues/{issueId}/comments` followed by a status-only `PATCH` does not advance the review/approval stage.
## Comment Style
Use concise markdown with:
docs/guides/board-operator/importing-and-exporting.md
+13
View File
@@ -29,6 +29,14 @@ my-company/
- **SKILL.md** files are compatible with the Agent Skills ecosystem.
- **.paperclip.yaml** holds Paperclip-specific config (adapter types, env inputs, budgets) as an optional sidecar.
## Export & Import in the App
Both flows are also available in the web UI as company settings pages: **Export** and **Import** appear in the company settings navigation.
The **Export** page lets you pick exactly which files go into the bundle before downloading it. Above the file tree it shows a **"Not included in this export"** panel — the export fidelity report — listing data the bundle will not carry (for example attachments, approvals, cost history, or activity log entries), with blocking issues highlighted.
The **Import** page previews the package, lets you resolve name collisions and adapter assignments, and applies the import. A **"Start imported agents and routines paused"** checkbox (on by default) makes imported agents and routines land paused instead of live. After the import finishes, an **"Activate imported agents and routines"** panel lists everything that was imported paused so you can resume the agents and activate the routines you select — nothing starts running until you say so.
## Exporting a Company
Export a company into a portable folder:
@@ -142,6 +150,8 @@ The preview shows:
Imported agents always land with timer heartbeats disabled. Assignment/on-demand wake behavior from the package is preserved, but scheduled runs stay off until a board operator re-enables them.
Imports can additionally request `pauseAutomations` (the default in the app's Import page) so imported agents and routines land fully paused. Use the post-import activation panel — or resume the agents and activate the routines individually — when you are ready for them to run.
### Common Workflows
**Clone a company template from GitHub:**
@@ -184,11 +194,14 @@ The CLI commands use these API endpoints under the hood:
| Action | Endpoint |
|--------|----------|
| Export company | `POST /api/companies/{companyId}/export` |
| Export fidelity report | `GET /api/companies/{companyId}/export/fidelity` |
| Preview import (existing company) | `POST /api/companies/{companyId}/imports/preview` |
| Apply import (existing company) | `POST /api/companies/{companyId}/imports/apply` |
| Preview import (new company) | `POST /api/companies/import/preview` |
| Apply import (new company) | `POST /api/companies/import` |
Import apply requests accept `pauseAutomations: true` to create imported agents and routines in a paused state.
CEO agents can also use the safe import routes (`/imports/preview` and `/imports/apply`) which enforce non-destructive rules: `replace` is rejected, collisions resolve with `rename` or `skip`, and issues are always created as new.
## GitHub Sources
docs/guides/execution-policy.md
+17
View File
@@ -173,9 +173,26 @@ This prevents silent completions where an agent finishes work but leaves no trac
- Only the **active reviewer/approver** (the `currentParticipant` in execution state) can advance or reject the current stage.
- Non-participants who attempt to transition the issue receive a `422 Unprocessable Entity` error.
- Both approvals and change requests **require a comment** — empty or whitespace-only comments are rejected.
- The decision comment must be included in the same `PATCH /api/issues/{issueId}` request that changes the status. A prior `POST /api/issues/{issueId}/comments` entry remains normal discussion and does not satisfy the review/approval decision guard.
## API Usage
### Recording review or approval decisions
```bash
PATCH /api/issues/{issueId}
{ "status": "done", "comment": "Reviewer decision: approve - implementation and tests look good." }
```
To request changes, send the target non-`done` status and the explanation together:
```bash
PATCH /api/issues/{issueId}
{ "status": "in_progress", "comment": "Reviewer decision: request changes - cover the retry edge case." }
```
Do not split a decision into `POST /comments` followed by a status-only `PATCH`; the runtime only records the decision from the `comment` field on the same status update.
### Setting an execution policy on issue creation
```bash
packages/adapter-utils/package.json
+1 -1
View File
@@ -47,7 +47,7 @@
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"typescript": "^5.7.3"
}
}
packages/adapter-utils/src/acpx-engine/execute.test.ts
+674 -2
View File
@@ -3,7 +3,7 @@ import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import type { AcpRuntimeOptions } from "acpx/runtime";
import type { AdapterRuntimeMcpAccess } from "@paperclipai/adapter-utils";
import type { AdapterExecutionContext, AdapterRuntimeMcpAccess } from "@paperclipai/adapter-utils";
import {
DEFAULT_REMOTE_SANDBOX_ADAPTER_TIMEOUT_SEC,
prepareAdapterExecutionTargetRuntime,
@@ -46,7 +46,18 @@ async function makeTempRoot() {
}
afterEach(async () => {
await Promise.all(tempRoots.splice(0).map((root) => fs.rm(root, { recursive: true, force: true })));
// A remote run stages a process-session bridge whose detached event writer can
// still be flushing a trailing event file into `.../process-sessions/<id>/events`
// when the run's own best-effort `client.remove(sessionDir)` (which production
// catch-wraps) has already returned. Under CI load that trailing write can land
// between this recursive delete's directory snapshot and its `rmdir`, surfacing as
// `ENOTEMPTY`. `maxRetries`/`retryDelay` make the cleanup ride out that window the
// same way production tolerates it, instead of failing the just-passed test.
await Promise.all(
tempRoots.splice(0).map((root) =>
fs.rm(root, { recursive: true, force: true, maxRetries: 5, retryDelay: 50 }),
),
);
});
async function pathExists(candidate: string): Promise<boolean> {
@@ -80,7 +91,16 @@ function createLocalSandboxRunner(
}) => void,
) {
let counter = 0;
// Synthetic provider-duration accumulators so per-step payload assertions can
// verify the `providerExecMs`/`providerGetMs` threading end-to-end (the real
// sandbox runner sources these from the Daytona plugin's result metadata; this
// double stands in for that with a fixed per-exec cost).
let providerExecMs = 0;
let providerGetMs = 0;
return {
execCount: () => counter,
providerExecMs: () => providerExecMs,
providerGetMs: () => providerGetMs,
execute: async (input: {
command: string;
args?: string[];
@@ -92,6 +112,8 @@ function createLocalSandboxRunner(
onSpawn?: (meta: { pid: number; startedAt: string }) => Promise<void>;
}) => {
counter += 1;
providerExecMs += 600;
providerGetMs += 15;
onExecute?.(input);
const command = input.command === "bash" ? "/bin/bash" : input.command;
return await runChildProcess(`acpx-sandbox-run-${counter}`, command, input.args ?? [], {
@@ -145,6 +167,7 @@ async function runExecutor(
executionTarget?: Record<string, unknown>;
runtimeMcp?: AdapterRuntimeMcpAccess;
prepareRemoteManagedHome?: AcpxEngineExecutorOptions["prepareRemoteManagedHome"];
startupTraceContext?: AdapterExecutionContext["startupTraceContext"];
} = {},
) {
const runtimeOptions: Record<string, unknown>[] = [];
@@ -179,6 +202,7 @@ async function runExecutor(
authToken: options.authToken,
executionTarget: options.executionTarget,
runtimeMcp: options.runtimeMcp,
startupTraceContext: options.startupTraceContext,
onLog: async (stream: "stdout" | "stderr", text: string) => {
logs.push({ stream, text });
},
@@ -194,6 +218,77 @@ async function runExecutor(
return { logs, meta, events, runtimeOptions, configOptions, sessionInputs, result };
}
// A recording span, used only in tests. It captures the span name, the parent
// span (resolved from the explicit parent-context token), the attribute map,
// the terminal status, and whether the span ended. The engine treats it purely
// through the structural `StartupSpan` contract.
interface RecordingSpan {
name: string;
attributes: Record<string, string | number | boolean>;
parent: RecordingSpan | null;
status: { code: number } | null;
ended: boolean;
setAttribute(key: string, value: string | number | boolean): void;
setStatus(status: { code: number; message?: string }): void;
end(): void;
}
// Build an in-memory startup trace context that records every span. It models
// the real OTel parenting contract: `startSpan(name, options, context)` reads
// the parent from the explicit `context` token that `contextWithSpan` produced,
// so a test asserts the exact parent of each child without an OTel package or
// ambient async-context propagation.
function createRecordingStartupTrace() {
const spans: RecordingSpan[] = [];
const traceContext = {
tracer: {
startSpan(
name: string,
options?: { attributes?: Record<string, string | number | boolean> },
context?: unknown,
) {
const parent =
context && typeof context === "object" && "span" in context
? ((context as { span: RecordingSpan }).span ?? null)
: null;
const span: RecordingSpan = {
name,
attributes: { ...(options?.attributes ?? {}) },
parent,
status: null,
ended: false,
setAttribute(key: string, value: string | number | boolean) {
span.attributes[key] = value;
},
setStatus(status: { code: number }) {
span.status = { code: status.code };
},
end() {
span.ended = true;
},
};
spans.push(span);
return span;
},
},
contextWithSpan(span: unknown) {
return { span };
},
} satisfies AdapterExecutionContext["startupTraceContext"];
return { traceContext, spans };
}
// The closed span-attribute allowlist for a sandbox-start span (Phase 2 + 3).
// A test asserts every recorded attribute key is in this set, so a command,
// path, id, or error-text key can never ride a span.
const ALLOWED_STARTUP_SPAN_ATTRIBUTE_KEYS = new Set([
"step",
"provider",
"roundTrips",
"providerExecMs",
"providerGetMs",
]);
describe("shared ACPX engine runtime behavior", () => {
it("sets Codex model, effort, and fast mode through CODEX_CONFIG without session config calls", async () => {
const { configOptions, meta } = await runExecutor({
@@ -776,6 +871,155 @@ describe("shared ACPX engine runtime behavior", () => {
expect(fp(rotatedKey)).not.toBe(fp(withKey));
});
it("busts the session fingerprint when the referenced-project set or pinned checkout changes", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const cwd = path.join(root, "workspace");
const baseConfig = { agentCommand: "node ./fake-acp.js", stateDir };
// A compatible resume reuses the already-staged referenced-project trees, so
// the fingerprint must fold in the referenced-project set. Otherwise a resume
// reuses a stale tree after the set or a project's pinned checkout changes.
const withReferenced = (additional: unknown[]) => ({
context: {
taskId: "issue-1",
wakeReason: "issue_assigned",
paperclipWorkspace: { cwd, realization: { additional } },
},
});
const projectA = {
path: "/host/project-a",
projectId: "a",
projectWorkspaceId: "ws-a",
repoUrl: "https://example.test/a.git",
repoRef: "ref-a-1",
};
const projectB = {
path: "/host/project-b",
projectId: "b",
projectWorkspaceId: "ws-b",
repoUrl: "https://example.test/b.git",
repoRef: "ref-b-1",
};
const one = await runExecutor(baseConfig, withReferenced([projectA]));
const two = await runExecutor(baseConfig, withReferenced([projectA, projectB]));
const reordered = await runExecutor(baseConfig, withReferenced([projectB, projectA]));
const repointed = await runExecutor(
baseConfig,
withReferenced([projectA, { ...projectB, repoRef: "ref-b-2" }]),
);
const fp = (r: { result: { sessionParams?: unknown } }) =>
(r.result.sessionParams as { configFingerprint?: string } | undefined)?.configFingerprint;
// Adding a referenced project changes the set identity, so the fingerprint
// busts and the next launch stages the current set.
expect(fp(one)).toBeDefined();
expect(fp(two)).not.toBe(fp(one));
// The identity depends on the set, not the order the records arrive in.
expect(fp(reordered)).toBe(fp(two));
// Re-pinning one project's checkout ref busts the fingerprint, so the resume
// re-stages instead of reusing a stale tree.
expect(fp(repointed)).not.toBe(fp(two));
});
it("busts the session fingerprint when referenced-project files change at the same host path", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const cwd = path.join(root, "workspace");
const projectDir = path.join(root, "project-a");
await fs.mkdir(path.join(projectDir, "src"), { recursive: true });
await fs.writeFile(path.join(projectDir, "src", "index.ts"), "export const value = 1;\n");
const baseConfig = { agentCommand: "node ./fake-acp.js", stateDir };
// Same host path and same metadata across runs. Only the file bytes change,
// which mirrors a branch that moved or a re-checkout in place. The metadata
// identity stays constant, so without the content signature a compatible
// resume would reuse a stale staged tree.
const referencedProject = {
path: projectDir,
projectId: "a",
projectWorkspaceId: "ws-a",
repoUrl: "https://example.test/a.git",
repoRef: "main",
};
const withProject = () => ({
context: {
taskId: "issue-1",
wakeReason: "issue_assigned",
paperclipWorkspace: { cwd, realization: { additional: [referencedProject] } },
},
});
const fp = (r: { result: { sessionParams?: unknown } }) =>
(r.result.sessionParams as { configFingerprint?: string } | undefined)?.configFingerprint;
const before = await runExecutor(baseConfig, withProject());
// Change the tree content and its size while the host path and metadata stay
// identical.
await fs.writeFile(path.join(projectDir, "src", "index.ts"), "export const value = 2; // changed\n");
const afterEdit = await runExecutor(baseConfig, withProject());
// Add a new file to the tree.
await fs.writeFile(path.join(projectDir, "src", "extra.ts"), "export const extra = true;\n");
const afterAdd = await runExecutor(baseConfig, withProject());
expect(fp(before)).toBeDefined();
// A content-only edit busts the fingerprint, so the resume re-stages.
expect(fp(afterEdit)).not.toBe(fp(before));
// A new file in the tree busts the fingerprint too.
expect(fp(afterAdd)).not.toBe(fp(afterEdit));
});
it("busts the session fingerprint when referenced-project bytes change but size and mtime stay equal", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const cwd = path.join(root, "workspace");
const projectDir = path.join(root, "project-a");
const filePath = path.join(projectDir, "src", "index.ts");
await fs.mkdir(path.join(projectDir, "src"), { recursive: true });
// The two contents have the same byte length, so a stat-only signature (size
// and modification time) would collide once the modification time also matches.
const firstContent = "export const value = 1;\n";
const secondContent = "export const value = 2;\n";
expect(secondContent.length).toBe(firstContent.length);
await fs.writeFile(filePath, firstContent);
// A re-checkout can restore the same modification time, so pin it explicitly
// and reapply it after the edit. Only the file bytes change between runs.
const pinnedTime = new Date("2026-01-01T00:00:00.000Z");
await fs.utimes(filePath, pinnedTime, pinnedTime);
const referencedProject = {
path: projectDir,
projectId: "a",
projectWorkspaceId: "ws-a",
repoUrl: "https://example.test/a.git",
repoRef: "main",
};
const withProject = () => ({
context: {
taskId: "issue-1",
wakeReason: "issue_assigned",
paperclipWorkspace: { cwd, realization: { additional: [referencedProject] } },
},
});
const fp = (r: { result: { sessionParams?: unknown } }) =>
(r.result.sessionParams as { configFingerprint?: string } | undefined)?.configFingerprint;
const baseConfig = { agentCommand: "node ./fake-acp.js", stateDir };
const before = await runExecutor(baseConfig, withProject());
// Replace the bytes with an equal-length string and restore the same
// modification time, so path, size, and mtime all stay identical across the two runs.
await fs.writeFile(filePath, secondContent);
await fs.utimes(filePath, pinnedTime, pinnedTime);
const after = await runExecutor(baseConfig, withProject());
expect(fp(before)).toBeDefined();
// The content signature reads bytes, so an equal-size, equal-mtime edit busts
// the fingerprint and the resume re-stages instead of reusing a stale tree.
expect(fp(after)).not.toBe(fp(before));
});
it("shapes ACPX session env for remote execution identities", async () => {
const root = await makeTempRoot();
const localCwd = path.join(root, "local");
@@ -1829,6 +2073,85 @@ describe("ACPX engine remote sandbox staging seam (PR 1: workspace + cwd)", () =
expect(sessionInputs[0]?.cwd).toBe(remoteCwd);
});
it("hands the merged paperclip env to the process-session launch when the setups overlap", async () => {
const { stateDir, localCwd, remoteCwd, executionTarget } = await setupRemoteSandbox();
// Decode the process-session LAUNCH payload (the base64 command blob) — the
// in-sandbox process env is carried there, NOT in the exec's own `env`.
let launchPayload: Record<string, unknown> | null = null;
(executionTarget as { runner: unknown }).runner = createLocalSandboxRunner((input) => {
if (input.env?.PAPERCLIP_SANDBOX_EXEC_CHANNEL === "bridge") {
const script = input.args?.[1] ?? "";
const match = script.match(/PAPERCLIP_PROCESS_SESSION_COMMAND_B64='([^']+)'/);
if (match) {
launchPayload = JSON.parse(Buffer.from(match[1]!, "base64").toString("utf8")) as Record<
string,
unknown
>;
}
}
});
await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{ authToken: "real-run-jwt", executionTarget },
);
// The process-session bridge receives its launch env as a DEFERRED thunk —
// the seam that lets its env-independent setup overlap the paperclip bridge
// start instead of running strictly after it.
const processArgs = vi.mocked(startAdapterExecutionTargetProcessSessionBridge).mock.calls[0]![0];
expect(typeof processArgs.env).toBe("function");
// ...and despite the overlap the launch still observes the MERGED paperclip
// env: the paperclip-`env` → process-session-launch hand-off stays sequenced
// under concurrency (bridge base URL + minted bridge token both present, and
// the token is NOT the host run JWT).
const payloadEnv = ((launchPayload as Record<string, unknown> | null)?.env ?? {}) as Record<
string,
unknown
>;
expect(payloadEnv).toMatchObject({ PAPERCLIP_API_BRIDGE_MODE: "queue_v1" });
expect(String(payloadEnv.PAPERCLIP_API_URL ?? "")).toMatch(/^http:\/\/127\.0\.0\.1:\d+$/);
expect(payloadEnv.PAPERCLIP_API_KEY).toBeTruthy();
expect(payloadEnv.PAPERCLIP_API_KEY).not.toBe("real-run-jwt");
});
it("stops the process-session bridge when the paperclip bridge fails under concurrency", async () => {
const { stateDir, localCwd, executionTarget } = await setupRemoteSandbox();
// The paperclip bridge fails; the process-session bridge — started CONCURRENTLY
// with it — still resolves a live handle. The abandon path must stop that
// handle so no started bridge leaks on partial failure.
const stop = vi.fn(async () => {});
vi.mocked(startAdapterExecutionTargetPaperclipBridge).mockImplementationOnce(async () => {
throw new Error("paperclip bridge boom");
});
vi.mocked(startAdapterExecutionTargetProcessSessionBridge).mockImplementationOnce(
async () => ({ agentCommand: null, stop }) as never,
);
const execute = createAcpxEngineExecutor({
createRuntime: () => buildRuntime() as never,
});
await expect(
execute({
runId: "run-bridge-fail",
agent: { id: "agent-1", companyId: "company-1" },
runtime: {},
config: { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
context: {},
authToken: "real-run-jwt",
executionTarget,
onLog: async () => {},
onMeta: async () => {},
onEvent: async () => {},
} as never),
).rejects.toThrow("paperclip bridge boom");
// The concurrently-started process-session bridge was stopped exactly once.
expect(stop).toHaveBeenCalledTimes(1);
});
it("test_remote_session_new_uses_in_sandbox_cwd", async () => {
const { stateDir, localCwd, remoteCwd, executionTarget } = await setupRemoteSandbox();
const { sessionInputs, runtimeOptions } = await runExecutor(
@@ -2679,6 +3002,226 @@ describe("ACPX engine remote session-lifecycle re-staging (PR 3: stage once / re
});
});
describe("ACPX engine sandbox-start spans (opt-in root + child parenting)", () => {
beforeEach(() => {
vi.clearAllMocks();
});
async function remoteSandboxTarget(root: string) {
const remoteCwd = path.join(root, "remote-workspace");
await fs.mkdir(remoteCwd, { recursive: true });
return {
kind: "remote",
transport: "sandbox",
// A plugin-backed key (not a built-in family). It must never ride a span
// as a raw attribute.
providerKey: "fake-plugin",
remoteCwd,
runner: createLocalSandboxRunner(),
};
}
it("emits one root span with a child span per bring-up boundary, each parented to the root", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
const codexHome = path.join(root, "codex-home");
await fs.mkdir(localCwd, { recursive: true });
await fs.mkdir(codexHome, { recursive: true });
const executionTarget = await remoteSandboxTarget(root);
const { traceContext, spans } = createRecordingStartupTrace();
await runExecutor(
{
agent: "codex",
agentCommand: "node ./fake-acp.js",
stateDir,
cwd: localCwd,
env: { CODEX_HOME: codexHome },
},
{ authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext },
);
// Exactly one root span, and it is the bring-up root.
const roots = spans.filter((span) => span.parent === null);
expect(roots).toHaveLength(1);
const rootSpan = roots[0]!;
expect(rootSpan.name).toBe("sandbox.startup");
expect(rootSpan.ended).toBe(true);
// A codex bring-up over the remote sandbox lane crosses all 7 boundaries.
const childNames = spans.filter((span) => span !== rootSpan).map((span) => span.name).sort();
expect(childNames).toEqual(
[
"acp.handshake",
"bridge.paperclip",
"bridge.process-session",
"codex-home.seed",
"skills.reconcile",
"stage.sync",
"workspace.resolve",
],
);
// Every child parents to the one root and ends.
for (const span of spans) {
if (span === rootSpan) continue;
expect(span.parent, `span "${span.name}" must parent to the root`).toBe(rootSpan);
expect(span.ended, `span "${span.name}" must end`).toBe(true);
}
});
it("parents both concurrent bridge spans to the root (neither orphans)", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
await fs.mkdir(localCwd, { recursive: true });
const executionTarget = await remoteSandboxTarget(root);
const { traceContext, spans } = createRecordingStartupTrace();
await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{ authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext },
);
const rootSpan = spans.find((span) => span.name === "sandbox.startup" && span.parent === null);
expect(rootSpan).toBeTruthy();
const paperclip = spans.find((span) => span.name === "bridge.paperclip");
const processSession = spans.find((span) => span.name === "bridge.process-session");
expect(paperclip?.parent).toBe(rootSpan);
expect(processSession?.parent).toBe(rootSpan);
});
it("keeps every span attribute inside the closed allowlist (no command/path/id keys)", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
await fs.mkdir(localCwd, { recursive: true });
const executionTarget = await remoteSandboxTarget(root);
const { traceContext, spans } = createRecordingStartupTrace();
await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{ authToken: "real-run-jwt", executionTarget, startupTraceContext: traceContext },
);
expect(spans.length).toBeGreaterThan(0);
for (const span of spans) {
for (const [key, value] of Object.entries(span.attributes)) {
expect(
ALLOWED_STARTUP_SPAN_ATTRIBUTE_KEYS.has(key),
`span "${span.name}" set a non-allowlisted attribute "${key}"`,
).toBe(true);
// No non-finite numeric attribute (no NaN, no Infinity).
if (typeof value === "number") {
expect(Number.isFinite(value), `attribute "${key}" must be finite`).toBe(true);
}
}
}
});
it("closes the root span with error status when the bring-up handshake fails", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
await fs.mkdir(localCwd, { recursive: true });
const executionTarget = await remoteSandboxTarget(root);
const { traceContext, spans } = createRecordingStartupTrace();
const execute = createAcpxEngineExecutor({
createRuntime: () =>
({
ensureSession: async () => {
throw new Error("handshake boom");
},
startTurn: () => ({
events: (async function* () {})(),
result: Promise.resolve({ status: "failed" }),
cancel: async () => {},
}),
setConfigOption: async () => {},
close: async () => {},
}) as never,
});
const result = await execute({
runId: "run-handshake-fail",
agent: { id: "agent-1", companyId: "company-1" },
runtime: {},
config: { agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
context: {},
authToken: "real-run-jwt",
executionTarget,
startupTraceContext: traceContext,
onLog: async () => {},
onMeta: async () => {},
onEvent: async () => {},
} as never);
expect(result.exitCode).toBe(1);
const rootSpan = spans.find((span) => span.name === "sandbox.startup" && span.parent === null);
expect(rootSpan).toBeTruthy();
expect(rootSpan!.ended).toBe(true);
// `2` is `SpanStatusCode.ERROR`.
expect(rootSpan!.status?.code).toBe(2);
});
it("opens no span and does not throw when no trace context is injected", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
await fs.mkdir(localCwd, { recursive: true });
const executionTarget = await remoteSandboxTarget(root);
// runExecutor asserts exitCode 0 internally; the run must complete with no
// injected trace context (the default no-op path).
const { result } = await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{ authToken: "real-run-jwt", executionTarget },
);
expect(result.exitCode).toBe(0);
});
it("opens no span for a local run, even when a trace context is injected", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
await fs.mkdir(localCwd, { recursive: true });
const { traceContext, spans } = createRecordingStartupTrace();
// A local run has no execution target. The `sandbox.startup` span names a
// sandbox bring-up, so a local run must stay out of sandbox telemetry.
const { result } = await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{ authToken: "real-run-jwt", startupTraceContext: traceContext },
);
expect(result.exitCode).toBe(0);
expect(spans).toHaveLength(0);
});
it("opens no span for an SSH run, even when a trace context is injected", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
const remoteCwd = path.join(root, "remote-workspace");
await fs.mkdir(localCwd, { recursive: true });
const { traceContext, spans } = createRecordingStartupTrace();
// An SSH run is remote but is not a sandbox, so it also stays out of sandbox
// telemetry.
const { result } = await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{
authToken: "real-run-jwt",
executionTarget: { kind: "remote", transport: "ssh", remoteCwd },
startupTraceContext: traceContext,
},
);
expect(result.exitCode).toBe(0);
expect(spans).toHaveLength(0);
});
});
describe("ACPX engine per-step startup timing (run.startup.step events)", () => {
beforeEach(() => {
vi.clearAllMocks();
@@ -2773,6 +3316,135 @@ describe("ACPX engine per-step startup timing (run.startup.step events)", () =>
}
});
it("carries roundTrips + provider durations for sequential startup steps and keeps concurrent bridge steps duration-only", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
const remoteCwd = path.join(root, "remote-workspace");
await fs.mkdir(localCwd, { recursive: true });
await fs.mkdir(remoteCwd, { recursive: true });
const executionTarget = {
kind: "remote",
transport: "sandbox",
providerKey: "fake-plugin",
remoteCwd,
runner: createLocalSandboxRunner(),
};
const { events } = await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{ authToken: "real-run-jwt", executionTarget },
);
const steps = stepEvents(events);
const seen = new Map(steps.map((event) => [String(event.payload?.step), event]));
// Every timed boundary still records duration.
for (const event of steps) {
expect(typeof event.payload?.durationMs).toBe("number");
}
// Sequential boundaries retain runner-counter attribution.
for (const step of ["workspace.resolve", "stage.sync", "acp.handshake"]) {
expect(typeof seen.get(step)?.payload?.roundTrips).toBe("number");
}
// workspace.resolve is host-only → zero host→sandbox execs.
expect(seen.get("workspace.resolve")?.payload?.roundTrips).toBe(0);
// stage.sync ships the workspace over the exec seam → at least one round-trip,
// and the accumulated provider durations scale with it.
const stageSync = seen.get("stage.sync");
expect(stageSync?.payload?.roundTrips as number).toBeGreaterThan(0);
expect(stageSync?.payload?.providerExecMs).toBe(
(stageSync?.payload?.roundTrips as number) * 600,
);
expect(stageSync?.payload?.providerGetMs).toBe(
(stageSync?.payload?.roundTrips as number) * 15,
);
// Concurrent bridge steps are duration-only so they do not double-count
// shared runner counters while their lifecycles overlap.
for (const step of ["bridge.paperclip", "bridge.process-session"]) {
expect(seen.get(step)?.payload?.roundTrips).toBeUndefined();
expect(seen.get(step)?.payload?.providerExecMs).toBeUndefined();
expect(seen.get(step)?.payload?.providerGetMs).toBeUndefined();
}
// The external ACP client crosses no host exec seam.
expect(seen.get("acp.handshake")?.payload?.roundTrips).toBe(0);
});
it("splits acp.handshake into createRuntimeMs and ensureSessionMs sub-phases", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const localCwd = path.join(root, "worktree");
const remoteCwd = path.join(root, "remote-workspace");
await fs.mkdir(localCwd, { recursive: true });
await fs.mkdir(remoteCwd, { recursive: true });
const executionTarget = {
kind: "remote",
transport: "sandbox",
providerKey: "fake-plugin",
remoteCwd,
runner: createLocalSandboxRunner(),
};
const { events } = await runExecutor(
{ agent: "custom", agentCommand: "node ./fake-acp.js", stateDir, cwd: localCwd },
{ authToken: "real-run-jwt", executionTarget },
);
const handshake = stepEvents(events).find((event) => event.payload?.step === "acp.handshake");
expect(handshake).toBeTruthy();
expect(typeof handshake!.payload?.createRuntimeMs).toBe("number");
expect(handshake!.payload?.createRuntimeMs as number).toBeGreaterThanOrEqual(0);
expect(typeof handshake!.payload?.ensureSessionMs).toBe("number");
expect(handshake!.payload?.ensureSessionMs as number).toBeGreaterThanOrEqual(0);
});
it("emits no acp.handshake event when a warm-handle hit skips the handshake", async () => {
const root = await makeTempRoot();
const stateDir = path.join(root, "state");
const warmHandles = new Map();
const secondEvents: Array<{ eventType: string; payload?: Record<string, unknown> }> = [];
const execute = createAcpxEngineExecutor({
warmHandles,
createRuntime: () => buildRuntime() as never,
});
const config = {
agent: "custom",
agentCommand: "node ./fake-acp.js",
stateDir,
mode: "persistent",
warmHandleIdleMs: 60_000,
};
const first = await execute({
runId: "run-warm-1",
agent: { id: "agent-1", companyId: "company-1" },
runtime: {},
config,
context: {},
onLog: async () => {},
onMeta: async () => {},
onEvent: async () => {},
} as never);
// The second run reuses the warm handle, so the whole handshake block is
// skipped — it must emit NO acp.handshake event (not a zero-duration one).
await execute({
runId: "run-warm-2",
agent: { id: "agent-1", companyId: "company-1" },
runtime: { sessionParams: (first as { sessionParams?: unknown }).sessionParams },
config,
context: {},
onLog: async () => {},
onMeta: async () => {},
onEvent: async (event: { eventType: string; payload?: Record<string, unknown> }) => {
secondEvents.push(event);
},
} as never);
const handshakeEmitted = secondEvents.some(
(event) => event.eventType === "run.startup.step" && event.payload?.step === "acp.handshake",
);
expect(handshakeEmitted).toBe(false);
});
it("does not emit startup-step events on a local (non-sandbox) run except workspace.resolve", async () => {
const root = await makeTempRoot();
const localCwd = path.join(root, "worktree");
packages/adapter-utils/src/acpx-engine/execute.ts
+448 -40
View File
@@ -28,6 +28,7 @@ import {
type AdapterExecutionTargetTimeoutResolution,
type AdapterManagedRuntimeAsset,
type PreparedAdapterExecutionTargetRuntime,
type SandboxAdditionalSource,
} from "@paperclipai/adapter-utils/execution-target";
import {
DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE,
@@ -82,7 +83,16 @@ import {
DEFAULT_ACP_ENGINE_TIMEOUT_SEC,
DEFAULT_ACP_ENGINE_WARM_HANDLE_IDLE_MS,
} from "./constants.js";
import { measureStartupStep } from "./startup-timing.js";
import {
measureStartupStep,
NOOP_STARTUP_SPAN,
NOOP_STARTUP_TRACE_CONTEXT,
type StartupSpan,
type StartupSpanContext,
type StartupStepMeasureOptions,
type StartupTraceContext,
} from "./startup-timing.js";
import type { CommandManagedRuntimeRunner } from "../command-managed-runtime.js";
const defaultModuleDir = path.dirname(fileURLToPath(import.meta.url));
const PAPERCLIP_MANAGED_CODEX_SKILLS_MANIFEST = ".paperclip-managed-skills.json";
@@ -381,6 +391,12 @@ interface AcpxPreparedRuntime {
paperclipClaudeSettings: PaperclipClaudeSettingsResult | null;
mcpServers: NonNullable<AcpRuntimeOptions["mcpServers"]>;
mcpIdentity: Array<{ name: string; url: string; connectionId: string }>;
// Per-step round-trip / provider-duration readers sourced from the sandbox
// runner's counters (Open Q1). Empty for local runs and the runner-less
// fallback, where no host→sandbox exec seam exists. Threaded into the
// `acp.handshake` `measureStartupStep` call in the executor (the other six
// boundaries live inside `buildRuntime` and read it directly).
stepMetrics: StartupStepMeasureOptions;
}
const defaultWarmHandles = new Map<string, RuntimeCacheEntry>();
@@ -411,6 +427,83 @@ function shortHash(value: unknown): string {
return createHash("sha256").update(stableJson(value)).digest("hex").slice(0, 16);
}
// Directory names the staging path never ships for a referenced project (heavy
// build/cache output and git history). The content signature skips them so it
// reflects only the staged tree and never reads their bytes. Keep this set equal
// to the staging excludes in the sandbox and remote runtimes.
const REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS = new Set([
"node_modules",
"vendor",
"dist",
"build",
"out",
"coverage",
".next",
".turbo",
".cache",
".git",
]);
/**
* Content signature of a referenced-project host tree for the session fingerprint.
*
* The staged-runtime cache reuses an already-staged referenced-project tree on a
* compatible resume and does not re-sync it. Referenced-project metadata (id, host
* path, workspace id, repo url, pinned ref) can stay identical while the files at
* that host path change: a branch moved to a new commit, a re-checkout in place, or
* a dirty worktree. So the metadata identity alone lets a resume serve a stale tree.
* This signature folds the tree's own content state into the identity.
*
* The walk reads each file's relative path and bytes and folds them into the hash.
* It reads bytes, not only file stats. A stat-only signature (size and modification
* time) collides when an edit keeps the byte length and the modification time — a
* re-checkout that restores the same size and timestamp. The byte hash busts on any
* content change, so the fingerprint busts and the next launch stages the current
* tree. The walk skips the heavy build, cache, and git directories the staging path
* never ships, and records a symlink by its target text without following it. On a
* read error the function returns a stable marker, so the fingerprint does not churn
* while staging surfaces the real error. The walk runs only when the run carries
* referenced projects (the multi-project sync path).
*/
async function referencedSourceContentSignature(localPath: string): Promise<string> {
const hash = createHash("sha256");
const walk = async (relative: string): Promise<void> => {
const current = relative ? path.join(localPath, relative) : localPath;
const dirents = await fs.readdir(current, { withFileTypes: true });
dirents.sort((left, right) => (left.name < right.name ? -1 : left.name > right.name ? 1 : 0));
for (const dirent of dirents) {
const next = relative ? path.posix.join(relative, dirent.name) : dirent.name;
if (dirent.isDirectory()) {
if (REFERENCED_SOURCE_SIGNATURE_SKIP_DIRS.has(dirent.name)) {
continue;
}
await walk(next);
continue;
}
const absolute = path.join(localPath, next);
const stats = await fs.lstat(absolute);
if (stats.isSymbolicLink()) {
const target = await fs.readlink(absolute);
hash.update(`symlink:${next}:${target}\n`);
continue;
}
if (!stats.isFile()) {
hash.update(`other:${next}:${stats.mode}\n`);
continue;
}
hash.update(`file:${next}:${stats.size}\n`);
hash.update(await fs.readFile(absolute));
hash.update("\n");
}
};
try {
await walk("");
} catch (error) {
return `unreadable:${String(error)}`;
}
return hash.digest("hex").slice(0, 16);
}
function defaultPaperclipInstanceDir(): string {
const home = process.env.PAPERCLIP_HOME?.trim() || path.join(os.homedir(), ".paperclip");
const instanceId = process.env.PAPERCLIP_INSTANCE_ID?.trim() || "default";
@@ -838,6 +931,11 @@ async function prepareCodexSkillRuntime(input: {
// `onEvent`), so the codex skill prep behaves identically when unmeasured.
onEvent?: AdapterExecutionContext["onEvent"];
now?: () => number;
// Round-trip / provider-duration readers for the nested `skills.reconcile`
// boundary (Open Q1). Threaded from `buildRuntime` so the step reports the
// same host→sandbox counters as its siblings (0 here — skill prep is
// host-only — which is itself the answer to "does this step exec?").
stepMetrics?: StartupStepMeasureOptions;
}): Promise<{ identity: Record<string, unknown>; commandNotes: string[] }> {
const now = input.now ?? (() => Date.now());
const envConfig = parseObject(input.config.env);
@@ -870,6 +968,7 @@ async function prepareCodexSkillRuntime(input: {
selectedSkills,
onLog: input.onLog,
}),
input.stepMetrics ?? {},
);
for (const entry of selectedSkills) {
@@ -1196,6 +1295,11 @@ async function stageAcpRemoteRuntime(input: {
workspaceRemoteDir?: string;
timeoutSec: number;
assets?: AdapterManagedRuntimeAsset[];
// Referenced (additional) projects to stage into the sandbox as plain,
// read-only trees alongside the anchor workspace. Empty unless run prep
// resolved referenced projects (gated upstream), so the anchor-only path is
// unchanged.
additionalSources?: SandboxAdditionalSource[];
onLog: AdapterExecutionContext["onLog"];
onRuntimeProgress: AdapterExecutionContext["onRuntimeProgress"];
}): Promise<PreparedAdapterExecutionTargetRuntime> {
@@ -1211,15 +1315,38 @@ async function stageAcpRemoteRuntime(input: {
workspaceLocalDir: input.workspaceLocalDir,
...(input.workspaceRemoteDir ? { workspaceRemoteDir: input.workspaceRemoteDir } : {}),
...(input.assets && input.assets.length > 0 ? { assets: input.assets } : {}),
...(input.additionalSources && input.additionalSources.length > 0
? { additionalSources: input.additionalSources }
: {}),
onProgress: (line) => input.onLog("stdout", line),
onRuntimeProgress: input.onRuntimeProgress,
});
}
// Bind a startup-step round-trip/provider-duration reader set to a runner's
// cumulative counters (Open Q1). Only the sandbox runner instruments the exec
// seam, so a runner without `execCount` (SSH, or none) yields an empty option
// set and the affected steps omit the fields entirely. Reader closures are
// passed — not the runner — so `measureStartupStep` stays runner-agnostic.
function buildStartupStepMetrics(
runner: CommandManagedRuntimeRunner | undefined,
): StartupStepMeasureOptions {
if (!runner) return {};
return {
...(runner.execCount ? { roundTrips: () => runner.execCount!() } : {}),
...(runner.providerExecMs ? { providerExecMs: () => runner.providerExecMs!() } : {}),
...(runner.providerGetMs ? { providerGetMs: () => runner.providerGetMs!() } : {}),
};
}
async function buildRuntime(input: {
ctx: AdapterExecutionContext;
engine: AcpxEngineSettings;
deps: AcpxEngineExecutorOptions;
// The injected tracer plus the root-span parent-context token. Merged into
// every startup-step option set, so each boundary span parents to the one
// root span (`sandbox.startup`) that the executor opens.
spanParent: Pick<StartupStepMeasureOptions, "tracer" | "parentContext">;
}): Promise<AcpxPreparedRuntime> {
const { runId, agent, config, context, authToken } = input.ctx;
// Injectable monotonic clock for per-step startup timing. Hoisted above the
@@ -1242,6 +1369,43 @@ async function buildRuntime(input: {
const useConfiguredInsteadOfAgentHome = workspaceSource === "agent_home" && configuredCwd.length > 0;
const effectiveWorkspaceCwd = useConfiguredInsteadOfAgentHome ? "" : workspaceCwd;
const cwd = effectiveWorkspaceCwd || configuredCwd || process.cwd();
// Referenced (additional) projects to stage into the sandbox alongside the
// anchor workspace, read from the workspace realization record. The list is
// empty unless run prep resolved referenced projects — gated upstream by the
// multi-project workspace-sync kill-switch — so the anchor-only path is
// unchanged.
const realizationContext = parseObject(workspaceContext.realization);
const additionalSourceRecords = (
Array.isArray(realizationContext.additional) ? realizationContext.additional : []
).map((entry) => parseObject(entry));
const additionalSources: SandboxAdditionalSource[] = additionalSourceRecords
.map((entry) => ({ localPath: asString(entry.path, ""), projectId: asString(entry.projectId, "") }))
.filter((entry) => entry.localPath.length > 0 && entry.projectId.length > 0);
// Stable identity of the referenced-project set for the session fingerprint.
// The staged-runtime cache reuses already-staged referenced-project trees on a
// compatible resume, so the fingerprint must change when the set OR a project's
// pinned checkout changes. Without this, a resume reuses a stale staged tree.
// Fold in each project's id, host path, workspace id, and pinned ref; sort by
// projectId so the identity depends on the set, not the record order.
const additionalSourcesIdentityBase = additionalSourceRecords
.map((entry) => ({
projectId: asString(entry.projectId, ""),
localPath: asString(entry.path, ""),
projectWorkspaceId: asString(entry.projectWorkspaceId, ""),
repoUrl: asString(entry.repoUrl, ""),
repoRef: asString(entry.repoRef, ""),
}))
.filter((entry) => entry.localPath.length > 0 && entry.projectId.length > 0)
.sort((a, b) => (a.projectId < b.projectId ? -1 : a.projectId > b.projectId ? 1 : 0));
// Metadata alone does not change on a content-only checkout change (same host
// path and pinned ref, new file bytes). Fold in each tree's content signature so
// a file add, remove, or edit busts the fingerprint and the resume re-stages.
const additionalSourcesIdentity = await Promise.all(
additionalSourcesIdentityBase.map(async (entry) => ({
...entry,
contentSignature: await referencedSourceContentSignature(entry.localPath),
})),
);
const executionTarget = readAdapterExecutionTarget({
executionTarget: input.ctx.executionTarget,
legacyRemoteExecution: input.ctx.executionTransport?.remoteExecution,
@@ -1252,6 +1416,29 @@ async function buildRuntime(input: {
? remoteExecutionIdentity.remoteCwd
: cwd;
const executionTargetIsRemote = remoteExecutionIdentity !== null;
// Round-trip / provider-duration readers for per-step attribution (Open Q1),
// sourced from the sandbox runner's cumulative counters. `measureStartupStep`
// reads each as a `() => number` closure (never the runner itself, Risk R1)
// and emits the per-step delta. Empty when there is no runner (local runs,
// the runner-less ACP→CLI fallback, or an SSH runner that does not
// instrument the seam), so those steps simply omit the fields.
// Merge the injected tracer + root parent-context into every step option set,
// so each boundary span parents to the root span. With no injected trace
// context both fields are no-ops and the span path stays inert.
const stepMetrics: StartupStepMeasureOptions = {
...buildStartupStepMetrics(
executionTarget?.kind === "remote" && executionTarget.transport === "sandbox"
? executionTarget.runner
: undefined,
),
...input.spanParent,
};
// The two bridge-start steps intentionally overlap, so their runner counters
// would double-count each other if we sampled them here. Keep the shared
// counter attribution on the sequential startup phases only; the concurrent
// bridge steps still emit duration telemetry (and a span), just not
// misleading per-step round-trip/provider deltas.
const concurrentBridgeStepMetrics: StartupStepMeasureOptions = { ...input.spanParent };
const shapedWorkspaceEnv = shapePaperclipWorkspaceEnvForExecution({
workspaceCwd: effectiveWorkspaceCwd,
workspaceWorktreePath,
@@ -1262,6 +1449,7 @@ async function buildRuntime(input: {
// here on the awaited directory materialization.
await measureStartupStep(input.ctx, nowMs, "workspace.resolve", () =>
ensureAbsoluteDirectory(cwd, { createIfMissing: true }),
stepMetrics,
);
const acpxAgent = normalizeAgent(config);
@@ -1422,7 +1610,9 @@ async function buildRuntime(input: {
onLog: input.ctx.onLog,
onEvent: input.ctx.onEvent,
now: nowMs,
stepMetrics,
}),
stepMetrics,
);
skillsIdentity = preparedSkills.identity;
skillCommandNotes.push(...preparedSkills.commandNotes);
@@ -1504,6 +1694,11 @@ async function buildRuntime(input: {
requestedThinkingEffort,
fastMode,
remoteExecutionIdentity,
// Referenced-project set + pinned-checkout identity. A change here (a project
// added, removed, or re-pinned) invalidates a warm/resumable session so the
// next launch stages the current referenced-project trees instead of reusing
// a stale staged tree.
additionalSourcesIdentity,
skillsIdentity,
skillPromptInstructions,
paperclipClaudeSettings: paperclipClaudeSettings
@@ -1632,6 +1827,7 @@ async function buildRuntime(input: {
workspaceRemoteDir: sessionCwd,
timeoutSec,
assets,
additionalSources,
onLog: input.ctx.onLog,
onRuntimeProgress: input.ctx.onRuntimeProgress,
});
@@ -1675,7 +1871,7 @@ async function buildRuntime(input: {
};
}
return { stagedRuntime: await stage([]), teardown: null, dispose: null };
});
}, stepMetrics);
const delta: Record<string, string> = {};
for (const [key, value] of Object.entries(env)) {
if (envBeforeStage[key] !== value) delta[key] = value;
@@ -1704,8 +1900,21 @@ async function buildRuntime(input: {
let runtimeEnv: Record<string, string> = {};
try {
if (useRemoteProcessSession) {
// Step 5 — bridge.paperclip: start the sandbox ACP API callback bridge.
paperclipBridge = await measureStartupStep(input.ctx, nowMs, "bridge.paperclip", () =>
// Steps 5 + 6 — bring up BOTH host-side sandbox bridges concurrently. Their
// remote subtrees are disjoint (`…/paperclip-bridge/…` vs
// `…/process-sessions/…`), so the env-INDEPENDENT setup of each overlaps,
// trending wall time from serial (~bridge.paperclip + ~bridge.process-session)
// toward ~max(the two). The ONE real dependency — the paperclip bridge's
// returned `env` must reach the process-session LAUNCH — is sequenced by
// `finalizeLaunchEnv`: the process-session bridge runs its env-independent
// dir/script setup first, then awaits that thunk right before its launch, so
// the launch always observes the merged paperclip env.
//
// Measurement caveat: both starts share ONE runner counter, so their
// overlapping `providerExecMs`/`roundTrips` deltas are approximate (the same
// caveat as `acp.handshake`). Both `run.startup.step` events still emit —
// `measureStartupStep` records them in a `finally`, even on a start failure.
const paperclipStart = measureStartupStep(input.ctx, nowMs, "bridge.paperclip", () =>
startAdapterExecutionTargetPaperclipBridge({
runId,
target: { ...executionTarget, streamRunLogs: false },
@@ -1715,36 +1924,59 @@ async function buildRuntime(input: {
hostApiToken: env.PAPERCLIP_API_KEY,
onLog: input.ctx.onLog,
}),
concurrentBridgeStepMetrics,
);
if (paperclipBridge) {
Object.assign(env, paperclipBridge.env);
await input.ctx.onLog("stdout", "[paperclip] Sandbox ACP API callback bridge enabled for this run.\n");
}
// The single sequencing point (paperclip `env` → process-session launch).
// Memoized so the merge + log + `runtimeEnv` build run EXACTLY once whether
// the process-session bridge consumes it at launch or we finalize it below.
let launchEnvPromise: Promise<Record<string, string>> | null = null;
const finalizeLaunchEnv = (): Promise<Record<string, string>> =>
(launchEnvPromise ??= (async () => {
const paperclip = await paperclipStart;
if (paperclip) {
Object.assign(env, paperclip.env);
await input.ctx.onLog("stdout", "[paperclip] Sandbox ACP API callback bridge enabled for this run.\n");
}
return (runtimeEnv = resolveRuntimeEnv(env));
})());
const processSessionStart = measureStartupStep(input.ctx, nowMs, "bridge.process-session", () =>
startAdapterExecutionTargetProcessSessionBridge({
runId,
target: executionTarget,
runtimeRootDir: stagedRuntime?.runtimeRootDir ?? null,
adapterKey: input.engine.adapterType,
command: "sh",
args: ["-lc", `exec ${agentCommandShell}`],
cwd: sessionCwd,
// Deferred: the process-session bridge runs its env-independent setup,
// then calls this to get the launch env AFTER the paperclip env merge.
env: finalizeLaunchEnv,
timeoutSec,
onLog: input.ctx.onLog,
}),
concurrentBridgeStepMetrics,
);
// Settle BOTH starts (mirrors `cleanupRemoteBridges`' `Promise.allSettled`):
// collect whichever handles started plus the first failure. Both handles
// stay individually declared so the catch below can stop whichever started.
const started = await settleRemoteBridgeStarts(paperclipStart, processSessionStart);
paperclipBridge = started.paperclipBridge;
processSessionBridge = started.processSessionBridge;
if (started.failure) throw started.failure;
// Guarantee the paperclip env merge ran even if the process-session bridge
// returned without consuming the launch env (memoized ⇒ a no-op if it did).
await finalizeLaunchEnv();
} else {
// Local / runner-less lanes never start a bridge, but the returned prepared
// runtime and the log builder still read `runtimeEnv`.
runtimeEnv = resolveRuntimeEnv(env);
}
runtimeEnv = Object.fromEntries(
Object.entries(ensurePathInEnv({ ...process.env, ...env })).filter(
(entry): entry is [string, string] => typeof entry[1] === "string",
),
);
// Step 6 — bridge.process-session: start the in-sandbox process session.
processSessionBridge = useRemoteProcessSession
? await measureStartupStep(input.ctx, nowMs, "bridge.process-session", () =>
startAdapterExecutionTargetProcessSessionBridge({
runId,
target: executionTarget,
runtimeRootDir: stagedRuntime?.runtimeRootDir ?? null,
adapterKey: input.engine.adapterType,
command: "sh",
args: ["-lc", `exec ${agentCommandShell}`],
cwd: sessionCwd,
env: runtimeEnv,
timeoutSec,
onLog: input.ctx.onLog,
}),
)
: null;
} catch (err) {
await paperclipBridge?.stop().catch(() => {});
// On a partial concurrent bring-up failure, ONE bridge may have started while
// the other threw; `Promise.allSettled` stops whichever started so no live
// bridge leaks (mirrors `cleanupRemoteBridges`). Both handles are individually
// declared above, so either may be non-null here.
await Promise.allSettled([paperclipBridge?.stop(), processSessionBridge?.stop()]);
// The staged home / copy-back teardown must run even if a bridge fails to
// start after the workspace + managed home were already staged into the
// sandbox, so a refreshed credential is copied back on this error path too.
@@ -1813,6 +2045,7 @@ async function buildRuntime(input: {
paperclipClaudeSettings,
mcpServers,
mcpIdentity,
stepMetrics,
};
}
@@ -1870,6 +2103,54 @@ async function applySessionConfigOptions(input: {
}
}
/**
* Build the process-session launch env: the host env overlaid with the run's
* `env` (so the merged paperclip bridge vars win) and a guaranteed `PATH`,
* narrowed to string values. Shared by the remote concurrent bring-up and the
* local / runner-less lane so both resolve the runtime env identically.
*/
function resolveRuntimeEnv(env: Record<string, string>): Record<string, string> {
return Object.fromEntries(
Object.entries(ensurePathInEnv({ ...process.env, ...env })).filter(
(entry): entry is [string, string] => typeof entry[1] === "string",
),
);
}
/**
* Bring up the two host-side sandbox bridges concurrently and settle both.
*
* Mirrors `cleanupRemoteBridges`' `Promise.allSettled` idiom (settle, not
* `Promise.all`): running BOTH starts to completion is what lets the caller STOP
* a bridge that DID start when its sibling threw — so a partial failure never
* leaks a live bridge. Returns whichever handles started plus the first failure
* (paperclip before process-session) for the caller to rethrow through the
* shared abandon path.
*/
async function settleRemoteBridgeStarts(
paperclipStart: Promise<AdapterExecutionTargetPaperclipBridgeHandle | null>,
processSessionStart: Promise<AdapterExecutionTargetProcessSessionBridgeHandle | null>,
): Promise<{
paperclipBridge: AdapterExecutionTargetPaperclipBridgeHandle | null;
processSessionBridge: AdapterExecutionTargetProcessSessionBridgeHandle | null;
failure: unknown;
}> {
const [paperclip, processSession] = await Promise.allSettled([
paperclipStart,
processSessionStart,
]);
return {
paperclipBridge: paperclip.status === "fulfilled" ? paperclip.value : null,
processSessionBridge: processSession.status === "fulfilled" ? processSession.value : null,
failure:
paperclip.status === "rejected"
? paperclip.reason
: processSession.status === "rejected"
? processSession.reason
: null,
};
}
async function cleanupRemoteBridges(prepared: AcpxPreparedRuntime): Promise<void> {
await Promise.allSettled([
prepared.processSessionBridge?.stop(),
@@ -2524,6 +2805,54 @@ function warmHandleMatches(
return entry !== undefined && entry.runtime === runtime && entry.handle === handle;
}
/** The stable name of the one root span for a sandbox bring-up. It is a fixed
* low-cardinality constant, never derived from run/user data. */
const STARTUP_ROOT_SPAN_NAME = "sandbox.startup";
/**
* Open the one root span for a sandbox bring-up and return its parent-context
* token plus a guarded `end`. The span parents every startup boundary span:
* the engine forwards `parentContext` to each `measureStartupStep` call. The
* `end` closure runs at most once (bring-up complete OR a bring-up failure) and
* swallows every tracer error, so observability never changes startup control
* flow. With no injected trace context, the tracer is a no-op and the span is
* a no-op.
*/
function openStartupRootSpan(tracing: StartupTraceContext): {
parentContext: StartupSpanContext;
end: (failed: boolean) => void;
} {
let span: StartupSpan;
try {
span = tracing.tracer.startSpan(STARTUP_ROOT_SPAN_NAME);
} catch {
span = NOOP_STARTUP_SPAN;
}
let parentContext: StartupSpanContext;
try {
parentContext = tracing.contextWithSpan(span);
} catch {
parentContext = undefined;
}
let ended = false;
return {
parentContext,
end: (failed: boolean) => {
if (ended) return;
ended = true;
try {
// `2` is `SpanStatusCode.ERROR`. `adapter-utils` stays OTel-free, so it
// uses the numeric value that a real injected span reads as the error
// status.
if (failed) span.setStatus({ code: 2 });
span.end();
} catch {
// Observability must not change startup control flow.
}
},
};
}
export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
const createRuntime = deps.createRuntime ?? createAcpRuntime;
const now = deps.now ?? (() => Date.now());
@@ -2554,7 +2883,39 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
now,
idleMs: warmIdleMs,
});
const prepared = await buildRuntime({ ctx, engine, deps });
// The `sandbox.startup` span names a sandbox bring-up. It must not cover a
// local or SSH run: those runs have no sandbox, so they stay out of sandbox
// telemetry. Open the real root span only when the target is a remote
// sandbox; every other target forces the no-op trace context, so the whole
// startup span path stays inert regardless of the injected context.
const startupExecutionTarget = readAdapterExecutionTarget({
executionTarget: ctx.executionTarget,
legacyRemoteExecution: ctx.executionTransport?.remoteExecution,
});
const targetsRemoteSandbox =
startupExecutionTarget?.kind === "remote" && startupExecutionTarget.transport === "sandbox";
// Open the one root span for this bring-up. It spans `buildRuntime` through
// `acp.handshake`, so every startup boundary span parents to it. `spanParent`
// carries the injected tracer + the root parent-context token into each
// `measureStartupStep` call. With no injected trace context the whole path
// is a no-op. `endRootSpan` runs exactly once — at bring-up completion or on
// a bring-up failure.
const tracing =
targetsRemoteSandbox && ctx.startupTraceContext
? ctx.startupTraceContext
: NOOP_STARTUP_TRACE_CONTEXT;
const rootSpan = openStartupRootSpan(tracing);
const spanParent: Pick<StartupStepMeasureOptions, "tracer" | "parentContext"> = {
tracer: tracing.tracer,
parentContext: rootSpan.parentContext,
};
let prepared: AcpxPreparedRuntime;
try {
prepared = await buildRuntime({ ctx, engine, deps, spanParent });
} catch (err) {
rootSpan.end(true);
throw err;
}
// State the effective wall-clock timeout and its source up front so a
// later timeout is diagnosable from the run log alone. Goes to stderr:
// the acpx stdout log stream carries JSON acpx.* event payloads and must
@@ -2594,7 +2955,22 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
? (chunk) => routeChildStderr(childStderrState, chunk)
: undefined,
};
const runtime = cached?.runtime ?? createRuntime(runtimeOptions);
// Open Q2: split the ~7s `acp.handshake` into the two in-repo-observable
// sub-phases — the ACP runtime construction (`createRuntime`) vs the session
// establishment envelope (`ensureSession`). The finer spawn/`initialize`/
// `session/new` split lives inside external `acpx` and is gated on an
// upstream lifecycle hook (not bundled here). `createRuntime` runs once and
// only on a cold start; a warm-handle hit reuses `cached.runtime`, so
// `createRuntimeMs` stays undefined and the split reports nothing for it.
let createRuntimeMs: number | undefined;
let runtime: AcpRuntime;
if (cached?.runtime) {
runtime = cached.runtime;
} else {
const createRuntimeStart = now();
runtime = createRuntime(runtimeOptions);
createRuntimeMs = now() - createRuntimeStart;
}
if (cached) clearWarmHandleTimer(cached);
if (!canResume && asString(previousParams.runtimeSessionName, "")) {
await ctx.onLog(
@@ -2612,17 +2988,29 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
try {
// Step 7 — acp.handshake: ACP session establishment (session/new or
// resume). A throwing handshake still reports its duration before the
// resume-retry path below runs.
handle = await measureStartupStep(ctx, now, "acp.handshake", () =>
runtime.ensureSession({
// resume-retry path below runs. `roundTrips` is expected to be 0 (the
// ACP client is external, not the host exec seam); the payload also
// carries the createRuntime/ensureSession sub-split (Open Q2).
let ensureSessionMs: number | undefined;
handle = await measureStartupStep(ctx, now, "acp.handshake", async () => {
const ensureSessionStart = now();
const established = await runtime.ensureSession({
sessionKey: prepared.sessionKey,
agent: prepared.acpxAgent,
mode: prepared.mode,
cwd: prepared.cwd,
resumeSessionId,
sessionOptions: { env: prepared.env },
});
ensureSessionMs = now() - ensureSessionStart;
return established;
}, {
...prepared.stepMetrics,
extra: () => ({
...(createRuntimeMs !== undefined ? { createRuntimeMs } : {}),
...(ensureSessionMs !== undefined ? { ensureSessionMs } : {}),
}),
);
});
} catch (err) {
if (!resumeSessionId || !isResumeFailure(err)) throw err;
clearSession = true;
@@ -2631,18 +3019,32 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
"stdout",
`[paperclip] ACPX resume session "${resumeSessionId}" is unavailable; retrying with a fresh session.\n`,
);
handle = await measureStartupStep(ctx, now, "acp.handshake", () =>
runtime.ensureSession({
// Fresh-session retry: the runtime was already constructed on the
// first attempt (never re-created), so this event reports only its
// own `ensureSessionMs` — no `createRuntimeMs`.
let retryEnsureSessionMs: number | undefined;
handle = await measureStartupStep(ctx, now, "acp.handshake", async () => {
const ensureSessionStart = now();
const established = await runtime.ensureSession({
sessionKey: prepared.sessionKey,
agent: prepared.acpxAgent,
mode: prepared.mode,
cwd: prepared.cwd,
sessionOptions: { env: prepared.env },
});
retryEnsureSessionMs = now() - ensureSessionStart;
return established;
}, {
...prepared.stepMetrics,
extra: () => ({
...(retryEnsureSessionMs !== undefined ? { ensureSessionMs: retryEnsureSessionMs } : {}),
}),
);
});
}
}
} catch (err) {
// Bring-up failed at the handshake — close the root span with error status.
rootSpan.end(true);
const { classified, message } = await emitAcpxFailure({
ctx,
prepared,
@@ -2666,6 +3068,8 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
}
if (!handle) {
// Bring-up produced no session handle — close the root span with error status.
rootSpan.end(true);
await discardStagedRuntime({ handles: stagedRuntimes, prepared });
await cleanupRemoteBridges(prepared);
return {
@@ -2680,6 +3084,10 @@ export function createAcpxEngineExecutor(deps: AcpxEngineExecutorOptions = {}) {
summary: "ACPX did not return a runtime session handle.",
};
}
// Bring-up is complete: the session handle is established. Close the root
// span here, so it covers `buildRuntime` through `acp.handshake` and no
// further. The agent turn runs after and is out of the startup root's scope.
rootSpan.end(false);
const sessionHandle = handle;
try {
await applySessionConfigOptions({
packages/adapter-utils/src/acpx-engine/startup-timing.test.ts
+285 -1
View File
@@ -1,6 +1,49 @@
import { describe, expect, it, vi } from "vitest";
import type { AdapterRuntimeEvent } from "../types.js";
import { measureStartupStep } from "./startup-timing.js";
import type { StartupSpan, StartupTracer } from "./startup-timing.js";
import { measureStartupStep, normalizeProviderFamily } from "./startup-timing.js";
/**
* A recording span for the mock tracer. It captures the attribute set, the
* status, and the end count so a test can assert the emitted span shape.
*/
class MockSpan implements StartupSpan {
readonly attributes: Record<string, string | number | boolean> = {};
status: { code: number; message?: string } | undefined;
endCount = 0;
constructor(
readonly name: string,
initial: Record<string, string | number | boolean> | undefined,
) {
if (initial) Object.assign(this.attributes, initial);
}
setAttribute(key: string, value: string | number | boolean): void {
this.attributes[key] = value;
}
setStatus(status: { code: number; message?: string }): void {
this.status = status;
}
end(): void {
this.endCount += 1;
}
}
/** A recording tracer that keeps every span it opens for assertions. */
function makeMockTracer(): { tracer: StartupTracer; spans: MockSpan[] } {
const spans: MockSpan[] = [];
const tracer: StartupTracer = {
startSpan(name, options) {
const span = new MockSpan(name, options?.attributes);
spans.push(span);
return span;
},
};
return { tracer, spans };
}
describe("measureStartupStep", () => {
it("emits one run.startup.step event with the step name and measured durationMs", async () => {
@@ -28,6 +71,86 @@ describe("measureStartupStep", () => {
expect(events[0]!.message).toBe("startup step: stage.sync (150ms)");
});
it("includes the roundTrips delta in the payload when a round-trip reader is supplied", async () => {
let t = 0;
const now = () => t;
// Cumulative host→sandbox exec counter; the step performs 3 execs.
let execCount = 5;
const events: AdapterRuntimeEvent[] = [];
const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => {
events.push(event);
});
await measureStartupStep({ onEvent }, now, "stage.sync", async () => {
t = 90;
execCount += 3;
return "ok";
}, { roundTrips: () => execCount });
expect(events[0]!.payload).toMatchObject({
step: "stage.sync",
durationMs: 90,
roundTrips: 3,
});
});
it("reports roundTrips: 0 for a step that performs no execs (reader supplied)", async () => {
const now = () => 0;
const events: AdapterRuntimeEvent[] = [];
const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => {
events.push(event);
});
await measureStartupStep({ onEvent }, now, "workspace.resolve", async () => "ok", {
roundTrips: () => 7,
});
expect(events[0]!.payload).toMatchObject({ step: "workspace.resolve", roundTrips: 0 });
});
it("omits roundTrips from the payload when no reader is supplied", async () => {
const now = () => 0;
const events: AdapterRuntimeEvent[] = [];
const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => {
events.push(event);
});
await measureStartupStep({ onEvent }, now, "workspace.resolve", async () => "ok");
expect(events[0]!.payload).not.toHaveProperty("roundTrips");
});
it("accumulates provider exec/get durations and merges extra fields into the payload", async () => {
let t = 0;
const now = () => t;
let execMs = 100;
let getMs = 40;
const events: AdapterRuntimeEvent[] = [];
const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => {
events.push(event);
});
await measureStartupStep({ onEvent }, now, "acp.handshake", async () => {
t = 7000;
execMs += 600; // one provider executeCommand round-trip
getMs += 15; // one client.get re-fetch
return "handle";
}, {
providerExecMs: () => execMs,
providerGetMs: () => getMs,
extra: () => ({ createRuntimeMs: 12, ensureSessionMs: 6988 }),
});
expect(events[0]!.payload).toMatchObject({
step: "acp.handshake",
durationMs: 7000,
providerExecMs: 600,
providerGetMs: 15,
createRuntimeMs: 12,
ensureSessionMs: 6988,
});
});
it("returns the wrapped fn result unchanged", async () => {
const now = () => 0;
const onEvent = vi.fn(async () => {});
@@ -113,4 +236,165 @@ describe("measureStartupStep", () => {
}),
).rejects.toBe(boom);
});
it("opens one span and ends it once for a normal step", async () => {
const { tracer, spans } = makeMockTracer();
const onEvent = vi.fn(async () => {});
await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", {
tracer,
});
expect(spans).toHaveLength(1);
expect(spans[0]!.name).toBe("stage.sync");
expect(spans[0]!.attributes.step).toBe("stage.sync");
expect(spans[0]!.endCount).toBe(1);
});
it("ends the span and sets an error status when fn throws, then re-throws", async () => {
const { tracer, spans } = makeMockTracer();
const onEvent = vi.fn(async () => {});
const boom = new Error("step failed");
await expect(
measureStartupStep({ onEvent }, () => 0, "acp.handshake", async () => {
throw boom;
}, { tracer }),
).rejects.toBe(boom);
expect(spans).toHaveLength(1);
expect(spans[0]!.endCount).toBe(1);
// SpanStatusCode.ERROR === 2 in @opentelemetry/api.
expect(spans[0]!.status?.code).toBe(2);
});
it("sets the same roundTrips / providerExecMs / providerGetMs deltas on the payload and the span", async () => {
let t = 0;
const now = () => t;
let execCount = 5;
let execMs = 100;
let getMs = 40;
const events: AdapterRuntimeEvent[] = [];
const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => {
events.push(event);
});
const { tracer, spans } = makeMockTracer();
await measureStartupStep({ onEvent }, now, "stage.sync", async () => {
t = 90;
execCount += 3;
execMs += 600;
getMs += 15;
return "ok";
}, {
tracer,
roundTrips: () => execCount,
providerExecMs: () => execMs,
providerGetMs: () => getMs,
});
const payload = events[0]!.payload as Record<string, unknown>;
expect(payload.roundTrips).toBe(3);
expect(payload.providerExecMs).toBe(600);
expect(payload.providerGetMs).toBe(15);
// The span carries the identical deltas — one build block feeds both.
expect(spans[0]!.attributes.roundTrips).toBe(3);
expect(spans[0]!.attributes.providerExecMs).toBe(600);
expect(spans[0]!.attributes.providerGetMs).toBe(15);
});
it("sets no span attribute (and no payload field) when a reader returns undefined", async () => {
const events: AdapterRuntimeEvent[] = [];
const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => {
events.push(event);
});
const { tracer, spans } = makeMockTracer();
await measureStartupStep({ onEvent }, () => 0, "workspace.resolve", async () => "ok", {
tracer,
// A reader may return undefined when the counter is unavailable. The guard
// must omit the attribute rather than emit NaN or 0.
roundTrips: () => undefined as unknown as number,
providerExecMs: () => undefined as unknown as number,
});
expect(spans[0]!.attributes).not.toHaveProperty("roundTrips");
expect(spans[0]!.attributes).not.toHaveProperty("providerExecMs");
expect(Object.values(spans[0]!.attributes).some((v) => Number.isNaN(v))).toBe(false);
const payload = events[0]!.payload as Record<string, unknown>;
expect(payload).not.toHaveProperty("roundTrips");
expect(payload).not.toHaveProperty("providerExecMs");
});
it("normalizes a plugin-backed provider key to plugin and keeps a built-in family as-is", async () => {
const onEvent = vi.fn(async () => {});
const custom = makeMockTracer();
await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", {
tracer: custom.tracer,
provider: "acme-cloud-runner",
});
expect(custom.spans[0]!.attributes.provider).toBe("plugin");
const builtIn = makeMockTracer();
await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", {
tracer: builtIn.tracer,
provider: "daytona",
});
expect(builtIn.spans[0]!.attributes.provider).toBe("daytona");
});
it("normalizeProviderFamily maps every non-built-in key to plugin", () => {
for (const key of ["daytona", "kubernetes", "e2b", "cloudflare", "exe-dev", "modal", "novita"]) {
expect(normalizeProviderFamily(key)).toBe(key);
}
for (const key of ["acme", "my-plugin", "", "DAYTONA", undefined]) {
expect(normalizeProviderFamily(key)).toBe("plugin");
}
});
it("emits exactly the allowlisted span-attribute key set and no command / path / ID / error-text key", async () => {
const onEvent = vi.fn(async () => {});
const { tracer, spans } = makeMockTracer();
await measureStartupStep({ onEvent }, () => 0, "acp.handshake", async () => "ok", {
tracer,
provider: "daytona",
roundTrips: () => 3,
providerExecMs: () => 600,
providerGetMs: () => 15,
// extra() carries caller-measured numbers into the EVENT payload only.
// It must never widen the span-attribute set.
extra: () => ({ createRuntimeMs: 12, ensureSessionMs: 6988 }),
});
expect(Object.keys(spans[0]!.attributes).sort()).toEqual(
["provider", "providerExecMs", "providerGetMs", "roundTrips", "step"],
);
// extra() keys stay off the span.
expect(spans[0]!.attributes).not.toHaveProperty("createRuntimeMs");
expect(spans[0]!.attributes).not.toHaveProperty("ensureSessionMs");
// No free-form identifier / command / path key leaks in. The pattern uses
// no `i` flag, so the camelCase `Id` matches `runId` / `userId` but not the
// "id" inside the allowlisted `provider`.
for (const key of Object.keys(spans[0]!.attributes)) {
expect(key).not.toMatch(/command|args|env|stdout|stderr|path|url|repo|ref|branch|Id|_id|error|message/);
}
});
it("uses a no-op tracer by default so a call without a tracer changes nothing", async () => {
const events: AdapterRuntimeEvent[] = [];
const onEvent = vi.fn(async (event: AdapterRuntimeEvent) => {
events.push(event);
});
// No tracer supplied. The helper must still emit the event and return the
// value without throwing.
const result = await measureStartupStep({ onEvent }, () => 0, "stage.sync", async () => "ok", {
roundTrips: () => 3,
});
expect(result).toBe("ok");
expect(events[0]!.payload).toMatchObject({ step: "stage.sync" });
});
});
packages/adapter-utils/src/acpx-engine/startup-timing.ts
+265 -8
View File
@@ -9,37 +9,294 @@ import type { AdapterExecutionContext, AdapterRuntimeEvent } from "../types.js";
*/
export const RUN_STARTUP_STEP_EVENT_TYPE = "run.startup.step";
function buildStepEvent(step: string, durationMs: number): AdapterRuntimeEvent {
/**
* The public built-in sandbox provider families. A key in this set is safe to
* emit as a low-cardinality span attribute. Any other key is operator-defined
* (plugin-backed) and unbounded, so `normalizeProviderFamily` maps it to the
* generic value `plugin`. Keep this list closed and small.
*/
const BUILT_IN_PROVIDER_FAMILIES: ReadonlySet<string> = new Set([
"daytona",
"kubernetes",
"e2b",
"cloudflare",
"exe-dev",
"modal",
"novita",
]);
/** The generic family for any provider key outside the built-in list. */
const PLUGIN_PROVIDER_FAMILY = "plugin";
/**
* Map a raw provider key to a low-cardinality public family. Return the key
* unchanged when it is a built-in family. Return `plugin` for every other
* value, so an operator-defined plugin key never becomes an unbounded span
* attribute. A missing or empty key also maps to `plugin`.
*/
export function normalizeProviderFamily(key: string | undefined): string {
if (key && BUILT_IN_PROVIDER_FAMILIES.has(key)) return key;
return PLUGIN_PROVIDER_FAMILY;
}
/**
* The value of `SpanStatusCode.ERROR` in `@opentelemetry/api`. `adapter-utils`
* stays OTel-free, so the timing helper uses the numeric value directly. A real
* injected OTel span reads it as the error status.
*/
const SPAN_STATUS_CODE_ERROR = 2;
/**
* A minimal, OTel-free span contract. The server injects a real
* `@opentelemetry/api` span, which satisfies this shape structurally. The
* default is a no-op span, so a step with no injected tracer changes nothing.
*/
export interface StartupSpan {
setAttribute(key: string, value: string | number | boolean): void;
setStatus(status: { code: number; message?: string }): void;
end(): void;
}
/**
* An opaque parent-context token. The server builds it from the OTel
* `@opentelemetry/api` `context` / `trace` helpers. `adapter-utils` never reads
* it; it only forwards it to `startSpan`, so this package stays OTel-free. A
* child span opened with this token parents to the span the token carries.
*/
export type StartupSpanContext = unknown;
/**
* A minimal, OTel-free tracer contract. The server injects a real
* `@opentelemetry/api` tracer, which satisfies this shape structurally. The
* `startSpan` signature is a subset of the OTel one, so a real tracer is
* assignable here. The optional third argument is the explicit parent context:
* a real OTel `startSpan(name, options, context)` parents the new span to the
* span that `context` carries. `adapter-utils` passes it through as an opaque
* token, so parenting never depends on ambient async-context propagation.
*/
export interface StartupTracer {
startSpan(
name: string,
options?: { attributes?: Record<string, string | number | boolean> },
context?: StartupSpanContext,
): StartupSpan;
}
/**
* The injected tracer plus the one context helper the engine needs to build a
* parent-context token from the root span. The server binds these to
* `@opentelemetry/api` (`trace.getTracer`, `trace.setSpan` over
* `context.active()`). The default is a no-op, so the whole span path stays a
* no-op until the server injects a real implementation.
*/
export interface StartupTraceContext {
readonly tracer: StartupTracer;
/**
* Return a parent-context token whose active span is `span`. A child span
* opened with this token parents to `span`. The token is opaque to
* `adapter-utils`.
*/
contextWithSpan(span: StartupSpan): StartupSpanContext;
}
/** A shared no-op span. It implements the structural span contract and does
* nothing, so a caller with no injected tracer changes no behavior. */
export const NOOP_STARTUP_SPAN: StartupSpan = {
setAttribute() {},
setStatus() {},
end() {},
};
const NOOP_SPAN = NOOP_STARTUP_SPAN;
/**
* The default tracer. It opens no real span, so `measureStartupStep` behaves
* exactly as before when the caller injects no tracer.
*/
const NOOP_TRACER: StartupTracer = {
startSpan: () => NOOP_SPAN,
};
/**
* The default trace context. Its tracer is a no-op and it produces no parent
* token, so the engine emits no spans until the server injects a real
* implementation.
*/
export const NOOP_STARTUP_TRACE_CONTEXT: StartupTraceContext = {
tracer: NOOP_TRACER,
contextWithSpan: () => undefined,
};
/**
* Set a numeric span attribute only when the value is a finite number. A reader
* that returns `undefined` (the counter is unavailable) yields no attribute,
* never `NaN` and never a misleading `0`. This mirrors the host counter guard
* at `environment-execution-target.ts`.
*/
function setFiniteNumberAttr(
span: StartupSpan,
key: string,
value: number | undefined,
): void {
if (typeof value === "number" && Number.isFinite(value)) {
span.setAttribute(key, value);
}
}
/**
* Compute a counter delta from a reader. Return `undefined` when the reader is
* absent, or when either the start or the end snapshot is not a finite number.
* A `undefined` result yields no payload field and no span attribute.
*/
function finiteDelta(
read: (() => number) | undefined,
start: number | undefined,
): number | undefined {
if (!read) return undefined;
const end = read();
if (typeof end !== "number" || !Number.isFinite(end)) return undefined;
const base = typeof start === "number" && Number.isFinite(start) ? start : 0;
return end - base;
}
/**
* Optional per-step attribution attached to a `run.startup.step` event, all
* additive to the free-form jsonb payload (no schema change). Each reader is a
* plain `() => number` closure so the timing helper stays decoupled from the
* runner/provider it reads (Risk R1): `measureStartupStep` snapshots the reader
* before `fn` and again in `finally`, emitting the delta.
*
* - `roundTrips` — cumulative host→sandbox `runner.execute` count; the delta is
* how many round-trips the step performed (Open Q1, host boundary).
* - `providerExecMs` / `providerGetMs` — cumulative provider-reported wall-time
* (ms) for the `executeCommand` REST call vs the `client.get` sandbox
* re-fetch; the delta attributes the step's round-trip time to its parts
* (Open Q1, finer provider attribution).
* - `extra` — a reader (called once in `finally`, after `fn` settles) returning
* any additional numeric fields to merge into the payload; used by
* `acp.handshake` to carry its `createRuntimeMs` / `ensureSessionMs` sub-split
* (Open Q2), which are measured by the caller rather than read from a counter.
* The `extra` map feeds the EVENT payload only. Its keys never become span
* attributes, so a free-form key cannot widen the closed span allowlist.
* - `tracer` — an injected structural tracer. It defaults to a no-op, so the
* span path changes no runtime behavior until the server injects a real
* tracer. The span carries only the closed attribute allowlist (`step`, the
* normalized `provider`, and the finite counter deltas).
* - `parentContext` — an opaque parent-context token from the root span. When
* set, the step's span parents to that root. `measureStartupStep` forwards it
* to `startSpan` and never inspects it, so parenting stays explicit and does
* not depend on ambient async-context propagation.
* - `provider` — the raw provider key for the step. `measureStartupStep`
* normalizes it through `normalizeProviderFamily` before it sets the
* low-cardinality `provider` span attribute. It never sets the raw key.
*/
export interface StartupStepMeasureOptions {
roundTrips?: () => number;
providerExecMs?: () => number;
providerGetMs?: () => number;
extra?: () => Record<string, number>;
tracer?: StartupTracer;
parentContext?: StartupSpanContext;
provider?: string;
}
function buildStepEvent(payload: Record<string, unknown>): AdapterRuntimeEvent {
const step = String(payload.step);
const durationMs = payload.durationMs as number;
return {
eventType: RUN_STARTUP_STEP_EVENT_TYPE,
stream: "system",
level: "info",
message: `startup step: ${step} (${durationMs}ms)`,
payload: { step, durationMs },
payload,
};
}
/**
* Time `fn` with the injected `now` clock and emit exactly one
* `run.startup.step` event carrying `{ step, durationMs }`. The event fires in a
* `finally`, so a throwing step still reports its duration before the error is
* re-thrown. `now` is injected (never `Date.now()` here) so callers/tests stay
* deterministic, and `ctx.onEvent` is optional — a missing sink is a no-op that
* neither throws nor swallows `fn`'s return value or error.
* `run.startup.step` event carrying `{ step, durationMs }` plus any counters
* supplied via `options`. The event fires in a `finally`, so a throwing step
* still reports its duration before the error is re-thrown. `now` is injected
* (never `Date.now()` here) so callers/tests stay deterministic, and
* `ctx.onEvent` is optional — a missing sink is a no-op that neither throws nor
* swallows `fn`'s return value or error. A step skipped by a warm cache never
* calls this helper, so it emits no event (never a zero).
*
* When `options.tracer` is injected, the helper also opens one span at `start`
* and ends it in the `finally`. The span carries a closed attribute allowlist:
* `step`, the normalized `provider`, and the finite counter deltas
* (`roundTrips` / `providerExecMs` / `providerGetMs`). A throwing `fn` sets the
* span error status before the span ends. The span build reuses the same delta
* values as the event payload, so the two paths never drift. The tracer
* defaults to a no-op, so a caller with no tracer changes nothing. Every span
* call sits inside the same error swallow as the event sink, so a throwing
* tracer never changes startup control flow.
*/
export async function measureStartupStep<T>(
ctx: Pick<AdapterExecutionContext, "onEvent">,
now: () => number,
step: string,
fn: () => Promise<T>,
options: StartupStepMeasureOptions = {},
): Promise<T> {
const start = now();
const roundTripsStart = options.roundTrips?.();
const providerExecStart = options.providerExecMs?.();
const providerGetStart = options.providerGetMs?.();
// Open the span with only the low-cardinality allowlisted attributes known at
// the start: the step name and the normalized provider family.
const tracer = options.tracer ?? NOOP_TRACER;
const startAttributes: Record<string, string> = { step };
if (options.provider !== undefined) {
startAttributes.provider = normalizeProviderFamily(options.provider);
}
let span: StartupSpan;
try {
span = tracer.startSpan(step, { attributes: startAttributes }, options.parentContext);
} catch {
// A throwing tracer must not change startup control flow.
span = NOOP_SPAN;
}
let stepFailed = false;
try {
return await fn();
} catch (err) {
stepFailed = true;
throw err;
} finally {
const durationMs = now() - start;
// One attribute-build block feeds both the event payload and the span, so
// the two paths never drift. `undefined` deltas produce neither a payload
// field nor a span attribute (fail open — never `NaN`, never `0`).
const roundTrips = finiteDelta(options.roundTrips, roundTripsStart);
const providerExecMs = finiteDelta(options.providerExecMs, providerExecStart);
const providerGetMs = finiteDelta(options.providerGetMs, providerGetStart);
const payload: Record<string, unknown> = { step, durationMs };
if (roundTrips !== undefined) payload.roundTrips = roundTrips;
if (providerExecMs !== undefined) payload.providerExecMs = providerExecMs;
if (providerGetMs !== undefined) payload.providerGetMs = providerGetMs;
if (options.extra) {
// `extra` feeds the EVENT payload only. Its keys never become span
// attributes, so it cannot widen the closed span allowlist.
Object.assign(payload, options.extra());
}
try {
await ctx.onEvent?.(buildStepEvent(step, durationMs));
if (stepFailed) span.setStatus({ code: SPAN_STATUS_CODE_ERROR });
setFiniteNumberAttr(span, "roundTrips", roundTrips);
setFiniteNumberAttr(span, "providerExecMs", providerExecMs);
setFiniteNumberAttr(span, "providerGetMs", providerGetMs);
span.end();
} catch {
// Observability must not change startup control flow.
}
try {
await ctx.onEvent?.(buildStepEvent(payload));
} catch {
// Observability must not change startup control flow.
}
packages/adapter-utils/src/command-managed-runtime.test.ts
+483 -5
View File
@@ -10,6 +10,7 @@ import {
prepareCommandManagedRuntime,
type CommandManagedRuntimeRunner,
} from "./command-managed-runtime.js";
import type { SandboxSyncOperation } from "./sandbox-managed-runtime.js";
import type { RunProcessResult } from "./server-utils.js";
const execFile = promisify(execFileCallback);
@@ -31,7 +32,12 @@ function makeSpawnRunner(options: {
supportsSingleStreamStdinProgress: options.supportsSingleStreamStdinProgress,
execute: async (input) =>
await new Promise<RunProcessResult>((resolve) => {
calls.push({ command: input.command, args: input.args, cwd: input.cwd, stdin: input.stdin });
calls.push({
command: input.command,
args: input.args,
cwd: input.cwd,
stdin: input.stdin,
});
const startedAt = new Date().toISOString();
const command =
input.command === "sh" ? "/bin/sh" : input.command === "bash" ? "/bin/bash" : input.command;
@@ -93,6 +99,10 @@ function toArrayBuffer(buffer: Buffer): ArrayBuffer {
return Uint8Array.from(buffer).buffer;
}
function shellQuoteForTest(value: string): string {
return `'${value.replace(/'/g, `'"'"'`)}'`;
}
async function withBase64StringByteLimit<T>(limitBytes: number, fn: () => Promise<T>): Promise<T> {
const originalToString = Buffer.prototype.toString;
Buffer.prototype.toString = function patchedToString(
@@ -293,6 +303,89 @@ describe("command managed runtime", () => {
);
});
it("stages each additional project into an isolated dir on the base64/tar transport, one failure skipped", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-runtime-additional-"));
cleanupDirs.push(rootDir);
const localWorkspaceDir = path.join(rootDir, "local-workspace");
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
await mkdir(localWorkspaceDir, { recursive: true });
await mkdir(remoteWorkspaceDir, { recursive: true });
await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor\n", "utf8");
const goodOne = path.join(rootDir, "src-one");
const goodTwo = path.join(rootDir, "src-two");
await mkdir(goodOne, { recursive: true });
await mkdir(path.join(goodTwo, "nested"), { recursive: true });
await writeFile(path.join(goodOne, "one.txt"), "one body\n", "utf8");
await writeFile(path.join(goodTwo, "nested", "two.txt"), "two body\n", "utf8");
// The `makeSpawnRunner` runner exposes no native syncIn, so staging rides the
// base64/tar fallback. The middle source points at a missing directory, so
// its tar build fails; failure isolation skips only it.
const { runner } = makeSpawnRunner();
const prepared = await prepareCommandManagedRuntime({
runner,
spec: {
remoteCwd: remoteWorkspaceDir,
timeoutMs: 30_000,
},
adapterKey: "claude",
workspaceLocalDir: localWorkspaceDir,
additionalSources: [
{ localPath: goodOne, projectId: "one" },
{ localPath: path.join(rootDir, "missing"), projectId: "broken" },
{ localPath: goodTwo, projectId: "two" },
],
});
const runtimeRootDir = path.posix.join(remoteWorkspaceDir, ".paperclip-runtime", "claude");
expect(Object.keys(prepared.additionalSourceDirs).sort()).toEqual(["one", "two"]);
expect(prepared.additionalSourceDirs.one).toBe(path.posix.join(runtimeRootDir, "project-one"));
expect(prepared.additionalSourceDirs.two).toBe(path.posix.join(runtimeRootDir, "project-two"));
expect(prepared.additionalSourceDirs.broken).toBeUndefined();
// Each healthy project's tree materialized in its OWN dir (nested files kept).
await expect(readFile(path.join(prepared.additionalSourceDirs.one, "one.txt"), "utf8")).resolves.toBe("one body\n");
await expect(readFile(path.join(prepared.additionalSourceDirs.two, "nested", "two.txt"), "utf8")).resolves.toBe(
"two body\n",
);
// The broken project's dir was never created.
await expect(readFile(path.join(runtimeRootDir, "project-broken"), "utf8")).rejects.toMatchObject({
code: "ENOENT",
});
});
it("keeps adapter detection on the profile-backed shell path", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-runtime-detect-"));
cleanupDirs.push(rootDir);
const localWorkspaceDir = path.join(rootDir, "local-workspace");
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
await mkdir(localWorkspaceDir, { recursive: true });
await mkdir(remoteWorkspaceDir, { recursive: true });
const { runner, calls } = makeSpawnRunner();
await prepareCommandManagedRuntime({
runner,
spec: {
remoteCwd: remoteWorkspaceDir,
timeoutMs: 30_000,
},
adapterKey: "claude",
workspaceLocalDir: localWorkspaceDir,
installCommand: "echo install",
detectCommand: "sh",
});
// The detection probe must be the first shell invocation, so a CLI on the
// sandbox default PATH is discoverable before we decide whether to install.
expect(calls[0]?.args?.join(" ")).toContain("command -v 'sh'");
// Detection succeeds here, so the install command must be skipped entirely;
// the remaining calls are workspace staging, never the install command.
expect(calls.some((call) => call.args?.join(" ").includes("echo install"))).toBe(false);
});
it("runs setup commands from a stable root cwd when staging into a nested remote workspace dir", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-runtime-nested-"));
cleanupDirs.push(rootDir);
@@ -344,7 +437,8 @@ describe("command managed runtime", () => {
});
// Exactly one upload process: O(1) round-trips regardless of payload size.
expect(calls.length).toBe(1);
expect(calls.length).toBe(2);
expect(calls[1].args?.join(" ")).toContain("rm -rf");
expect(calls[0].stdin).toBeTypeOf("string");
const written = await readFile(remotePath);
@@ -371,11 +465,49 @@ describe("command managed runtime", () => {
// stage-then-atomic-rename shape (temp .paperclip-upload + `mv -f`).
const script = (calls[0].args ?? []).join(" ");
expect(script).toContain(`${remotePath}.paperclip-upload`);
expect(script).toContain(`trap cleanup EXIT`);
expect(script).toContain(`mv -f`);
expect(script.indexOf(".paperclip-upload")).toBeLessThan(script.indexOf("mv -f"));
expect(await readFile(remotePath, "utf8")).toBe("hello atomic\n");
});
it("cleans up a staged upload when rename fails", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-upload-cleanup-"));
cleanupDirs.push(rootDir);
const remotePath = path.join(rootDir, "nested", "payload.bin");
const payload = Buffer.alloc(3 * 1024 * 1024, 7);
const { runner, calls } = makeSpawnRunner({ supportsSingleStreamStdinProgress: true });
const delegatedExecute = runner.execute.bind(runner);
runner.execute = async (input) => {
const script = (input.args ?? []).join(" ");
if (script.includes("mv -f") && script.includes(".paperclip-upload.")) {
calls.push({ command: input.command, args: input.args, cwd: input.cwd, stdin: input.stdin });
return {
exitCode: 1,
signal: null,
timedOut: false,
stdout: "",
stderr: "rename failed",
pid: null,
startedAt: new Date().toISOString(),
};
}
return await delegatedExecute(input);
};
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
await expect(client.writeFile(remotePath, toArrayBuffer(payload))).rejects.toThrow(/rename failed/);
const uploadCall = calls.find((call) => (call.args ?? []).join(" ").includes(".paperclip-upload."));
expect(uploadCall).toBeDefined();
const stagedPath = (uploadCall?.args ?? []).join(" ").match(/([/A-Za-z0-9_.-]+\.paperclip-upload\.[A-Za-z0-9-]+)/)?.[1];
expect(stagedPath).toBeDefined();
await expect(readFile(stagedPath!, "utf8")).rejects.toMatchObject({ code: "ENOENT" });
expect(calls.some((call) => (call.args ?? []).join(" ").includes(`rm -rf '${stagedPath}'`))).toBe(true);
await expect(readFile(remotePath, "utf8")).rejects.toMatchObject({ code: "ENOENT" });
});
it("stages a single-file write to a temp then renames it on the chunked fallback path too", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-atomic-fallback-"));
cleanupDirs.push(rootDir);
@@ -393,15 +525,23 @@ describe("command managed runtime", () => {
expect((await readFile(remotePath)).equals(payload)).toBe(true);
});
it("leaves the client without syncIn/syncOut unless the runner supports both (fallback preserved)", () => {
it("test_client_syncIn_present_even_without_native_runner_syncIn", () => {
// Phase 2 (PAP-3222): `client.syncIn` is ALWAYS present so the caller can
// delegate unconditionally. `syncOut` stays native-only (no generic outbound
// fallback in this seam).
const base = makeSpawnRunner().runner;
expect(createCommandManagedRuntimeClient({ runner: base, commandCwd: "/", timeoutMs: 1 }).syncIn).toBeUndefined();
const client = createCommandManagedRuntimeClient({ runner: base, commandCwd: "/", timeoutMs: 1 });
expect(client.syncIn).toBeTypeOf("function");
expect(client.syncOut).toBeUndefined();
// A runner advertising only one verb still gets the fallback syncIn; syncOut
// stays undefined (native delegation needs BOTH verbs).
const onlyIn: CommandManagedRuntimeRunner = { ...base, syncIn: async () => ({ operations: [] }) };
const partial = createCommandManagedRuntimeClient({ runner: onlyIn, commandCwd: "/", timeoutMs: 1 });
expect(partial.syncIn).toBeUndefined();
expect(partial.syncIn).toBeTypeOf("function");
expect(partial.syncOut).toBeUndefined();
// With both verbs, syncIn delegates natively and syncOut is exposed.
const both: CommandManagedRuntimeRunner = {
...base,
syncIn: async () => ({ operations: [] }),
@@ -412,6 +552,344 @@ describe("command managed runtime", () => {
expect(native.syncOut).toBeTypeOf("function");
});
it("test_client_syncIn_delegates_to_native_runner_with_zero_execute_calls", async () => {
// With a native runner, `client.syncIn` forwards `files` + `postUploadCommands`
// to the runner and issues ZERO `execute` round-trips (the provider owns the
// transport + command execution).
let executeCalls = 0;
const forwarded: SandboxSyncOperation[][] = [];
const runner: CommandManagedRuntimeRunner = {
execute: async () => {
executeCalls += 1;
return { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "", pid: null, startedAt: "" };
},
syncIn: async (operations) => {
forwarded.push(operations);
return {
operations: operations.map((op) => ({
operationId: op.operationId,
filesTransferred: op.files.length,
bytesTransferred: 0,
})),
};
},
syncOut: async () => ({ operations: [] }),
};
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 1 });
const operations: SandboxSyncOperation[] = [
{
operationId: "op-1",
files: [{ sourcePath: "/host/a", targetPath: "/remote/a", kind: "directory" }],
postUploadCommands: [{ command: "echo done", cwd: "/remote/a" }],
},
];
const result = await client.syncIn!(operations);
expect(executeCalls).toBe(0);
expect(forwarded).toEqual([operations]);
expect(result.operations[0]).toMatchObject({ operationId: "op-1", filesTransferred: 1 });
});
it("fallback syncIn tarballs+uploads a directory then runs post-upload commands in order", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-syncin-fallback-"));
cleanupDirs.push(rootDir);
const sourceDir = path.join(rootDir, "source");
const targetDir = path.join(rootDir, "target");
const markerDir = path.join(rootDir, "markers");
await mkdir(path.join(sourceDir, "nested"), { recursive: true });
await mkdir(markerDir, { recursive: true });
await writeFile(path.join(sourceDir, "file.txt"), "payload\n", "utf8");
await writeFile(path.join(sourceDir, "nested", "deep.txt"), "deep\n", "utf8");
const { runner, calls } = makeSpawnRunner({ supportsSingleStreamStdinProgress: true });
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
await client.syncIn!([
{
operationId: "op-dir",
files: [{ sourcePath: sourceDir, targetPath: targetDir, kind: "directory" }],
postUploadCommands: [
{ command: "touch " + shellQuoteForTest(path.join(markerDir, "1-first")) },
{ command: "touch " + shellQuoteForTest(path.join(markerDir, "2-second")) },
],
},
]);
// Files landed via tar → untar (destroy-then-replace).
expect(await readFile(path.join(targetDir, "file.txt"), "utf8")).toBe("payload\n");
expect(await readFile(path.join(targetDir, "nested", "deep.txt"), "utf8")).toBe("deep\n");
// Both post-upload commands ran (markers exist).
await expect(readFile(path.join(markerDir, "1-first"))).resolves.toBeDefined();
await expect(readFile(path.join(markerDir, "2-second"))).resolves.toBeDefined();
// Ordering: upload → untar → command 1 → command 2. The tarball upload is the
// single stdin-backed call; the untar and the two commands follow it in order.
const scripts = calls.map((call) => (call.args ?? []).join("\n"));
const uploadIdx = scripts.findIndex((s) => s.includes(".paperclip-syncin.tar") && s.includes("base64 -d"));
const untarIdx = scripts.findIndex((s) => s.includes("tar -xf") && s.includes(targetDir));
const cmd1Idx = scripts.findIndex((s) => s.includes("1-first"));
const cmd2Idx = scripts.findIndex((s) => s.includes("2-second"));
expect(uploadIdx).toBeGreaterThanOrEqual(0);
expect(untarIdx).toBeGreaterThan(uploadIdx);
expect(cmd1Idx).toBeGreaterThan(untarIdx);
expect(cmd2Idx).toBeGreaterThan(cmd1Idx);
});
it("fallback syncIn runs a post-upload command under its own timeout, not the sync-client default", async () => {
// The run-specific timeout (`spec.timeoutMs`, stamped onto each delegated
// post-upload command) can differ from the sync client's own default. The
// fallback must honor the per-command `timeoutMs` so the delegated
// extract/cleanup/merge runs under the run limit — not the sync default.
const syncClientTimeoutMs = 30_000;
const runTimeoutMs = 7_000;
const execTimeouts: Array<number | undefined> = [];
const runner: CommandManagedRuntimeRunner = {
execute: async (input) => {
execTimeouts.push(input.timeoutMs);
return { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "", pid: null, startedAt: "" };
},
};
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: syncClientTimeoutMs });
await client.syncIn!([
{
operationId: "op-timeout",
files: [],
postUploadCommands: [
{ command: "echo carries-run-timeout", timeoutMs: runTimeoutMs },
{ command: "echo defaults-to-sync-timeout" },
],
},
]);
// First command carries the run timeout; a command with no explicit timeout
// still falls back to the sync-client default (matched by the stamping in
// prepareSandboxManagedRuntime, which never leaves a delegated command bare).
expect(execTimeouts).toEqual([runTimeoutMs, syncClientTimeoutMs]);
});
it("fallback syncIn stages mode-constrained files before chmod and rename", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-syncin-mode-"));
cleanupDirs.push(rootDir);
const sourceFile = path.join(rootDir, "source.txt");
const targetFile = path.join(rootDir, "target.txt");
await writeFile(sourceFile, "payload\n", "utf8");
const { runner, calls } = makeSpawnRunner({ supportsSingleStreamStdinProgress: true });
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
await client.syncIn!([
{
operationId: "op-mode",
files: [{ sourcePath: sourceFile, targetPath: targetFile, kind: "file", mode: 0o640 }],
},
]);
expect(await readFile(targetFile, "utf8")).toBe("payload\n");
const scripts = calls.map((call) => (call.args ?? []).join(" "));
expect(scripts).toHaveLength(5);
expect(scripts[0]).toContain(targetFile + ".paperclip-syncin.");
expect(scripts[0]).toContain(".paperclip-upload.");
expect(scripts[1]).toContain("rm -rf");
expect(scripts[1]).toContain(".paperclip-upload.");
expect(scripts[2]).toContain("chmod 640");
expect(scripts[2]).toContain(targetFile + ".paperclip-syncin.");
expect(scripts[3]).toContain("mv -f");
expect(scripts[3]).toContain(targetFile + ".paperclip-syncin.");
expect(scripts[3]).toContain(targetFile);
expect(scripts[4]).toContain("rm -rf");
expect(scripts[4]).toContain(targetFile + ".paperclip-syncin.");
});
it("fallback syncIn cleans up a staged file when chmod fails before rename", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-syncin-cleanup-"));
cleanupDirs.push(rootDir);
const sourceFile = path.join(rootDir, "source.txt");
const targetFile = path.join(rootDir, "target.txt");
await writeFile(sourceFile, "payload\n", "utf8");
const { runner, calls } = makeSpawnRunner({ supportsSingleStreamStdinProgress: true });
const delegatedExecute = runner.execute.bind(runner);
runner.execute = async (input) => {
const script = (input.args ?? []).join(" ");
if (script.includes("chmod 600")) {
calls.push({ command: input.command, args: input.args, cwd: input.cwd, stdin: input.stdin });
return {
exitCode: 1,
signal: null,
timedOut: false,
stdout: "",
stderr: "chmod failed",
pid: null,
startedAt: new Date().toISOString(),
};
}
return await delegatedExecute(input);
};
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
await expect(
client.syncIn!([
{
operationId: "op-cleanup",
files: [{ sourcePath: sourceFile, targetPath: targetFile, kind: "file", mode: 0o600 }],
},
]),
).rejects.toThrow(/chmod failed/);
const chmodCall = calls.find((call) => (call.args ?? []).join(" ").includes("chmod 600"));
expect(chmodCall).toBeDefined();
const stagedPath = (chmodCall?.args ?? []).join(" ").match(/chmod 600 '([^']+)'/)?.[1];
expect(stagedPath).toBeDefined();
await expect(readFile(stagedPath!, "utf8")).rejects.toMatchObject({ code: "ENOENT" });
expect(calls.some((call) => (call.args ?? []).join(" ").includes(`rm -rf '${stagedPath}'`))).toBe(true);
await expect(readFile(targetFile, "utf8")).rejects.toMatchObject({ code: "ENOENT" });
});
it("test_post_upload_commands_execute_verbatim_not_rewritten (C1 opaque)", async () => {
// The provider/client treats each command as opaque: it is executed VERBATIM,
// never concatenated with asset keys / paths or otherwise rewritten.
const executed: string[] = [];
const runner: CommandManagedRuntimeRunner = {
execute: async (input) => {
// Only capture the post-upload command executions (single `sh -c <cmd>`).
if ((input.args?.[0] === "-c") && typeof input.args?.[1] === "string") {
executed.push(input.args[1]);
}
return { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "", pid: null, startedAt: "" };
},
};
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 1 });
const verbatim = "my-tool --flag 'quoted value' && echo $HOME";
await client.syncIn!([
{ operationId: "op-verbatim", files: [], postUploadCommands: [{ command: verbatim }] },
]);
// The exact string appears among executed scripts, unmodified.
expect(executed).toContain(verbatim);
});
it("test_post_upload_command_cwd_escaping_target_root_is_rejected (C2)", async () => {
// A `cwd` that escapes the operation's target root — via `..` or an absolute
// path outside the target — is rejected BEFORE any handoff (no execute).
let executeCalls = 0;
const runner: CommandManagedRuntimeRunner = {
execute: async () => {
executeCalls += 1;
return { exitCode: 0, signal: null, timedOut: false, stdout: "", stderr: "", pid: null, startedAt: "" };
},
};
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 1 });
const traversal: SandboxSyncOperation[] = [
{
operationId: "op-traversal",
files: [{ sourcePath: "/host/a", targetPath: "/remote/a", kind: "directory" }],
postUploadCommands: [{ command: "echo x", cwd: "/remote/a/../etc" }],
},
];
await expect(client.syncIn!(traversal)).rejects.toThrow(/confined absolute POSIX path|escapes/);
const absoluteEscape: SandboxSyncOperation[] = [
{
operationId: "op-escape",
files: [{ sourcePath: "/host/a", targetPath: "/remote/a", kind: "directory" }],
postUploadCommands: [{ command: "echo x", cwd: "/etc/passwd" }],
},
];
await expect(client.syncIn!(absoluteEscape)).rejects.toThrow(/escapes the operation's target root/);
// A confined cwd (equal to the target root) passes confinement — it fails
// later at tar time (the source dir does not exist), which is a DIFFERENT
// error than a confinement rejection.
const confined: SandboxSyncOperation[] = [
{
operationId: "op-confined",
files: [{ sourcePath: "/host/a", targetPath: "/remote/a", kind: "directory" }],
postUploadCommands: [{ command: "echo x", cwd: "/remote/a" }],
},
];
let confinementRejected = false;
try {
await client.syncIn!(confined);
} catch (error) {
confinementRejected = /escapes the operation's target root|confined absolute POSIX path/.test(
(error as Error).message,
);
}
expect(confinementRejected).toBe(false);
// Confinement rejected before any exec for the escape cases.
expect(executeCalls).toBe(0);
});
it("test_fallback_syncIn_aborts_and_rejects_on_first_nonzero_exit (C4 fail-fast)", async () => {
// The first non-zero post-upload command aborts the operation: syncIn rejects,
// the remaining commands do NOT run, and there is no silent partial fallback.
const executed: string[] = [];
const runner: CommandManagedRuntimeRunner = {
execute: async (input) => {
const script = input.args?.[1] ?? "";
executed.push(script);
const isFailing = script.includes("FAIL-COMMAND");
return {
exitCode: isFailing ? 3 : 0,
signal: null,
timedOut: false,
stdout: "",
stderr: isFailing ? "boom" : "",
pid: null,
startedAt: "",
};
},
};
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 1 });
await expect(
client.syncIn!([
{
operationId: "op-failfast",
files: [],
postUploadCommands: [
{ command: "echo before" },
{ command: "FAIL-COMMAND" },
{ command: "echo SHOULD-NOT-RUN" },
],
},
]),
).rejects.toThrow(/exit code 3|boom/);
expect(executed.some((s) => s.includes("echo before"))).toBe(true);
expect(executed.some((s) => s.includes("FAIL-COMMAND"))).toBe(true);
expect(executed.some((s) => s.includes("SHOULD-NOT-RUN"))).toBe(false);
});
it("test_single_stream_writeFile_collapses_roundtrips_under_96MiB", async () => {
// Research A1: with single-stream enabled a ≤96 MiB write is ONE round-trip;
// without it, the chunked path is `2 + ceil(bytes / 3 MiB)`. Same payload,
// same client API — only the runner capability flag differs.
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-single-stream-collapse-"));
cleanupDirs.push(rootDir);
const payload = Buffer.alloc(9 * 1024 * 1024, 7); // 9 MiB → chunked = 2 + 3 = 5 execs
const single = makeSpawnRunner({ supportsSingleStreamStdinProgress: true });
const singleClient = createCommandManagedRuntimeClient({ runner: single.runner, commandCwd: "/", timeoutMs: 30_000 });
await singleClient.writeFile(path.join(rootDir, "single.bin"), toArrayBuffer(payload));
expect(single.calls.length).toBe(2);
const chunked = makeSpawnRunner({ supportsSingleStreamStdinProgress: false });
const chunkedClient = createCommandManagedRuntimeClient({ runner: chunked.runner, commandCwd: "/", timeoutMs: 30_000 });
await chunkedClient.writeFile(path.join(rootDir, "chunked.bin"), toArrayBuffer(payload));
// 3 (init temp + final mv + cleanup) + ceil(9MiB / 3MiB) = 6 round-trips.
expect(chunked.calls.length).toBe(3 + Math.ceil(payload.byteLength / (3 * 1024 * 1024)));
expect(chunked.calls.length).toBeGreaterThan(single.calls.length);
expect((await readFile(path.join(rootDir, "single.bin"))).equals(payload)).toBe(true);
expect((await readFile(path.join(rootDir, "chunked.bin"))).equals(payload)).toBe(true);
});
it("falls back to chunked upload progress when the runner cannot report mid-stream stdin progress", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-write-fallback-"));
cleanupDirs.push(rootDir);
packages/adapter-utils/src/command-managed-runtime.ts
+238 -41
View File
@@ -1,7 +1,12 @@
import { promises as fs } from "node:fs";
import { randomUUID } from "node:crypto";
import os from "node:os";
import path from "node:path";
import {
createTarballFromDirectory,
prepareSandboxManagedRuntime,
type PreparedSandboxManagedRuntime,
type SandboxAdditionalSource,
type SandboxManagedRuntimeAsset,
type SandboxManagedRuntimeClient,
type SandboxRemoteExecutionSpec,
@@ -20,6 +25,25 @@ export interface CommandManagedRuntimeRunner {
* and let the caller choose a chunked upload path when progress is requested.
*/
supportsSingleStreamStdinProgress?: boolean;
/**
* Cumulative count of host→sandbox `execute` round-trips this runner has
* performed (Open Q1). Present only on runners that instrument the single
* exec seam (the sandbox runner); the per-step delta is emitted as
* `run.startup.step` `payload.roundTrips`. A `() => number` reader, never the
* runner itself, is threaded into `measureStartupStep` so the timing helper
* stays runner-agnostic.
*/
execCount?(): number;
/**
* Cumulative provider-reported wall-time (ms) for the `executeCommand` REST
* call ({@link providerExecMs}) vs the `client.get` sandbox re-fetch that
* precedes it ({@link providerGetMs}), accumulated across every `execute`
* round-trip (Open Q1, finer attribution). Present only when the provider
* surfaces these durations on its result metadata; the per-step deltas are
* emitted as `payload.providerExecMs` / `payload.providerGetMs`.
*/
providerExecMs?(): number;
providerGetMs?(): number;
execute(input: {
command: string;
args?: string[];
@@ -103,6 +127,73 @@ function requireSuccessfulResult(result: RunProcessResult, action: string): void
throw new Error(`${action} failed with exit code ${result.exitCode ?? "null"}${detail}`);
}
function bufferToArrayBuffer(buffer: Buffer): ArrayBuffer {
// Copy out of the (possibly pooled) Node Buffer so the ArrayBuffer we hand to
// the client transport owns exactly these bytes.
return buffer.buffer.slice(buffer.byteOffset, buffer.byteOffset + buffer.byteLength) as ArrayBuffer;
}
// Named builder (Security Condition C3): extract an uploaded tarball into its
// target directory as a clean destroy-then-replace, then remove the tarball.
// Every path is shell-quoted; the fallback NEVER concatenates untrusted asset
// keys / file names into the shell.
function buildSyncInExtractDirectoryCommand(input: { remoteTarPath: string; targetDir: string }): string {
return (
`rm -rf ${shellQuote(input.targetDir)} && ` +
`mkdir -p ${shellQuote(input.targetDir)} && ` +
`tar -xf ${shellQuote(input.remoteTarPath)} -C ${shellQuote(input.targetDir)} && ` +
`rm -f ${shellQuote(input.remoteTarPath)}`
);
}
// Named builder (C3): apply a POSIX mode to a placed file. Octal literal, quoted
// path; no interpolation of untrusted values.
function buildSyncInChmodCommand(input: { mode: number; targetPath: string }): string {
return `chmod ${(input.mode & 0o7777).toString(8)} ${shellQuote(input.targetPath)}`;
}
function buildSyncInRenameCommand(input: { sourcePath: string; targetPath: string }): string {
return "mv -f " + shellQuote(input.sourcePath) + " " + shellQuote(input.targetPath);
}
function buildUniqueStagingPath(input: { targetPath: string; suffix: string }): string {
return `${input.targetPath}${input.suffix}.${randomUUID()}`;
}
async function bestEffortRemoveRemotePath(client: SandboxManagedRuntimeClient, remotePath: string): Promise<void> {
await client.remove(remotePath).catch(() => undefined);
}
/**
* Host-side confinement guard for a sync operation's post-upload command `cwd`
* (Security Condition C2). Runs BEFORE any handoff — native delegation OR the
* generic fallback — so an out-of-root `cwd` is rejected fail-closed before a
* provider ever sees it. `cwd` (when present) MUST be an absolute POSIX path with
* no `..` segment, confined to (equal to or under) one of the operation's own
* file-mapping target paths. Commands with no `cwd` are unconstrained here and
* default to the runtime's stable command cwd at exec time.
*/
export function assertPostUploadCommandsConfined(operations: readonly SandboxSyncOperation[]): void {
for (const operation of operations) {
const commands = operation.postUploadCommands ?? [];
if (commands.length === 0) continue;
const targetRoots = operation.files.map((mapping) => path.posix.normalize(mapping.targetPath));
for (const command of commands) {
if (command.cwd == null) continue;
const raw = command.cwd;
if (!path.posix.isAbsolute(raw) || raw.split("/").includes("..")) {
throw new Error(`post-upload command cwd is not a confined absolute POSIX path: ${raw}`);
}
const normalized = path.posix.normalize(raw);
const within = targetRoots.some(
(root) => normalized === root || normalized.startsWith(`${root}/`),
);
if (!within) {
throw new Error(`post-upload command cwd escapes the operation's target root: ${raw}`);
}
}
}
}
export function createCommandManagedRuntimeClient(input: {
runner: CommandManagedRuntimeRunner;
commandCwd: string;
@@ -139,47 +230,52 @@ export function createCommandManagedRuntimeClient(input: {
const total = buffer.byteLength;
const encodedLength = base64EncodedLength(total);
const remoteDir = path.posix.dirname(remotePath);
const remoteTempPath = `${remotePath}.paperclip-upload`;
const remoteTempPath = buildUniqueStagingPath({ targetPath: remotePath, suffix: ".paperclip-upload" });
const canUseSingleStreamProgressPath = input.runner.supportsSingleStreamStdinProgress === true;
// Primary path: a single round-trip. Stream the entire base64 body to one
// `base64 -d` process via stdin, decode straight into a temp file, then
// atomically rename into place. This replaces the previous loop that did
// one `printf >> tmpfile` shell round-trip per 32 KB — thousands of serial
// processes for a large workspace — with exactly one process.
if (
encodedLength <= REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES &&
canUseSingleStreamProgressPath
) {
const body = buffer.toString("base64");
await options?.onProgress?.(0, total);
try {
// Primary path: a single round-trip. Stream the entire base64 body to one
// `base64 -d` process via stdin, decode straight into a temp file, then
// atomically rename into place. This replaces the previous loop that did
// one `printf >> tmpfile` shell round-trip per 32 KB — thousands of serial
// processes for a large workspace — with exactly one process.
if (
encodedLength <= REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES &&
canUseSingleStreamProgressPath
) {
const body = buffer.toString("base64");
await options?.onProgress?.(0, total);
await runShell(
`cleanup() { rm -f ${shellQuote(remoteTempPath)}; }; trap cleanup EXIT INT TERM; ` +
`mkdir -p ${shellQuote(remoteDir)} && ` +
`base64 -d > ${shellQuote(remoteTempPath)} && ` +
`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`,
{ stdin: body },
);
await options?.onProgress?.(total, total);
return;
}
// Bounded fallback for payloads too large to hand the runner as one stdin
// string: append the base64 body to a remote temp file in large chunks
// (orders of magnitude fewer round-trips than the old 32 KB loop), decoding
// each self-contained chunk on arrival and emitting progress per write,
// then atomically rename into place.
await runShell(
`mkdir -p ${shellQuote(remoteDir)} && ` +
`base64 -d > ${shellQuote(remoteTempPath)} && ` +
`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`,
{ stdin: body },
`rm -f ${shellQuote(remoteTempPath)} && : > ${shellQuote(remoteTempPath)}`,
);
for (let offset = 0; offset < total; offset += REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE) {
const end = Math.min(total, offset + REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE);
const chunk = buffer.subarray(offset, end).toString("base64");
await runShell(`base64 -d >> ${shellQuote(remoteTempPath)}`, { stdin: chunk });
await options?.onProgress?.(end, total);
}
await runShell(`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`);
await options?.onProgress?.(total, total);
return;
} finally {
await bestEffortRemoveRemotePath(client, remoteTempPath);
}
// Bounded fallback for payloads too large to hand the runner as one stdin
// string: append the base64 body to a remote temp file in large chunks
// (orders of magnitude fewer round-trips than the old 32 KB loop), decoding
// each self-contained chunk on arrival and emitting progress per write,
// then atomically rename into place.
await runShell(
`mkdir -p ${shellQuote(remoteDir)} && ` +
`rm -f ${shellQuote(remoteTempPath)} && : > ${shellQuote(remoteTempPath)}`,
);
for (let offset = 0; offset < total; offset += REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE) {
const end = Math.min(total, offset + REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE);
const chunk = buffer.subarray(offset, end).toString("base64");
await runShell(`base64 -d >> ${shellQuote(remoteTempPath)}`, { stdin: chunk });
await options?.onProgress?.(end, total);
}
await runShell(`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`);
await options?.onProgress?.(total, total);
},
readFile: async (remotePath, options) => {
// Chunked reads intentionally query the remote size first, even without
@@ -253,13 +349,110 @@ export function createCommandManagedRuntimeClient(input: {
},
};
// Expose the native sync capability to the orchestrator only when the runner
// supports BOTH directions; a provider that advertises just one verb (or
// neither) keeps the byte-identical base64 fallback for both.
const { syncIn, syncOut } = input.runner;
if (syncIn && syncOut) {
client.syncIn = (operations) => syncIn(operations);
client.syncOut = (operations) => syncOut(operations);
// Generic base64-tar fallback for `syncIn` on runners without native sync:
// place each operation's files (host-side tarball → `writeFile` → destroy-then-
// replace untar for directories, direct `writeFile` for single files), then run
// the operation's ordered `postUploadCommands` fail-fast. Byte-for-byte
// behavior-equivalent to the caller-inlined tar path it will replace. All exec
// rides the shared `execute` seam so `execCount`/`providerExecMs` still
// attribute (Open Q1).
const fallbackSyncIn = async (operations: SandboxSyncOperation[]): Promise<SandboxSyncResult> => {
const resultOperations: SandboxSyncResult["operations"] = [];
const tempDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-syncin-fallback-"));
try {
for (const operation of operations) {
let filesTransferred = 0;
let bytesTransferred = 0;
for (const [index, mapping] of operation.files.entries()) {
const cleanupPaths: string[] = [];
try {
if (mapping.kind === "directory") {
const archivePath = path.join(tempDir, `syncin-${index}.tar`);
await createTarballFromDirectory({
localDir: mapping.sourcePath,
archivePath,
exclude: mapping.exclude,
followSymlinks: mapping.followSymlinks,
});
const tarBytes = await fs.readFile(archivePath);
const remoteTarPath = buildUniqueStagingPath({
targetPath: mapping.targetPath,
suffix: ".paperclip-syncin.tar",
});
cleanupPaths.push(remoteTarPath);
await client.writeFile(remoteTarPath, bufferToArrayBuffer(tarBytes));
await client.run(
buildSyncInExtractDirectoryCommand({ remoteTarPath, targetDir: mapping.targetPath }),
{ timeoutMs: input.timeoutMs },
);
bytesTransferred += tarBytes.byteLength;
} else {
const fileBytes = await fs.readFile(mapping.sourcePath);
const targetPathForWrite = mapping.mode != null
? buildUniqueStagingPath({ targetPath: mapping.targetPath, suffix: ".paperclip-syncin" })
: mapping.targetPath;
if (mapping.mode != null) cleanupPaths.push(targetPathForWrite);
await client.writeFile(targetPathForWrite, bufferToArrayBuffer(fileBytes));
if (mapping.mode != null) {
await client.run(
buildSyncInChmodCommand({ mode: mapping.mode, targetPath: targetPathForWrite }),
{ timeoutMs: input.timeoutMs },
);
await client.run(
buildSyncInRenameCommand({ sourcePath: targetPathForWrite, targetPath: mapping.targetPath }),
{ timeoutMs: input.timeoutMs },
);
}
bytesTransferred += fileBytes.byteLength;
}
} finally {
for (const cleanupPath of cleanupPaths.reverse()) {
await bestEffortRemoveRemotePath(client, cleanupPath);
}
}
filesTransferred += 1;
}
// Ordered, fail-fast post-upload commands (C1 opaque / C4 fail-loud). Each
// command string is executed VERBATIM — never rewritten, concatenated, or
// appended to. First non-zero exit or timeout throws and stops the rest.
for (const command of operation.postUploadCommands ?? []) {
const result = await input.runner.execute({
command: shellCommand,
args: shellCommandArgs(command.command),
cwd: command.cwd ?? input.commandCwd,
timeoutMs: command.timeoutMs ?? input.timeoutMs,
});
requireSuccessfulResult(result, command.command);
}
resultOperations.push({
operationId: operation.operationId,
filesTransferred,
bytesTransferred,
});
}
} finally {
await fs.rm(tempDir, { recursive: true, force: true }).catch(() => undefined);
}
return { operations: resultOperations };
};
// `client.syncIn` is ALWAYS present: it delegates to the runner's native
// transport when the provider advertises BOTH sync verbs, otherwise it runs the
// generic fallback above. Either way, post-upload command `cwd` confinement (C2)
// is validated on the host BEFORE any handoff. `syncOut` stays native-only —
// there is no generic outbound fallback in this seam.
const nativeSyncIn = input.runner.syncIn;
const nativeSyncOut = input.runner.syncOut;
const hasNativeBoth = Boolean(nativeSyncIn && nativeSyncOut);
client.syncIn = async (operations) => {
assertPostUploadCommandsConfined(operations);
if (hasNativeBoth) {
return await nativeSyncIn!(operations);
}
return await fallbackSyncIn(operations);
};
if (hasNativeBoth) {
client.syncOut = (operations) => nativeSyncOut!(operations);
}
return client;
@@ -275,6 +468,8 @@ export async function prepareCommandManagedRuntime(input: {
workspaceExclude?: string[];
preserveAbsentOnRestore?: string[];
assets?: CommandManagedRuntimeAsset[];
/** Referenced (additional) projects to stage into the sandbox as plain, read-only trees. */
additionalSources?: SandboxAdditionalSource[];
installCommand?: string | null;
/** When provided alongside `installCommand`, skip the install if `command -v <detectCommand>` succeeds. */
detectCommand?: string | null;
@@ -330,6 +525,7 @@ export async function prepareCommandManagedRuntime(input: {
workspaceExclude: mergeRuntimeExcludes(input.workspaceExclude),
preserveAbsentOnRestore: input.preserveAbsentOnRestore,
assets: input.assets,
additionalSources: input.additionalSources,
onProgress: input.onProgress,
onRuntimeProgress: input.onRuntimeProgress,
});
@@ -367,6 +563,7 @@ export async function prepareCommandManagedRuntime(input: {
workspaceExclude: mergeRuntimeExcludes(input.workspaceExclude),
preserveAbsentOnRestore: input.preserveAbsentOnRestore,
assets: input.assets,
additionalSources: input.additionalSources,
onProgress: input.onProgress,
onRuntimeProgress: input.onRuntimeProgress,
});
packages/adapter-utils/src/execution-target-sandbox.test.ts
+54
View File
@@ -194,6 +194,60 @@ describe("sandbox adapter execution targets", () => {
});
});
it("creates the process session directories only in the launch exec, not in upfront makeDir execs", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-process-session-makedir-"));
cleanupDirs.push(rootDir);
const childPath = path.join(rootDir, "noop-acp-child.mjs");
await writeFile(childPath, "process.stdin.on('data', () => {});\n", "utf8");
const delegate = createLocalSandboxRunner();
const execScripts: string[] = [];
const runner = {
execute: vi.fn(async (input: Parameters<typeof delegate.execute>[0]) => {
execScripts.push(input.args?.[1] ?? "");
return delegate.execute(input);
}),
};
const target: AdapterSandboxExecutionTarget = {
kind: "remote",
transport: "sandbox",
providerKey: "local-test",
remoteCwd: rootDir,
timeoutMs: 30_000,
runner,
};
const bridge = await startAdapterExecutionTargetProcessSessionBridge({
runId: "run-process-session-makedir",
target,
runtimeRootDir: path.posix.join(rootDir, ".paperclip-runtime", "acpx"),
adapterKey: "acpx",
command: process.execPath,
args: [childPath],
cwd: rootDir,
env: {},
timeoutSec: 5,
onLog: async () => {},
});
expect(bridge).not.toBeNull();
try {
// No standalone `mkdir -p '<dir>/stdin'` or `.../events` exec runs before launch.
const standaloneSessionDirExecs = execScripts.filter((script) =>
/^mkdir -p '[^']*\/(stdin|events)'\s*$/.test(script),
);
expect(standaloneSessionDirExecs).toEqual([]);
// The launch exec creates both directories in one `mkdir -p` line.
const launchExecs = execScripts.filter(
(script) => script.includes("nohup") && /mkdir -p [^\n]*\/stdin[^\n]*\/events/.test(script),
);
expect(launchExecs.length).toBe(1);
} finally {
await bridge?.stop();
}
});
it("bridges bidirectional sandbox process sessions through a local ACPX-spawnable proxy", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-process-session-"));
cleanupDirs.push(rootDir);
packages/adapter-utils/src/execution-target.ts
+63 -8
View File
@@ -14,6 +14,8 @@ import {
prepareRemoteManagedRuntime,
remoteExecutionSessionMatches,
} from "./remote-managed-runtime.js";
import type { SandboxAdditionalSource } from "./sandbox-managed-runtime.js";
export type { SandboxAdditionalSource } from "./sandbox-managed-runtime.js";
import {
createCommandManagedSandboxCallbackBridgeQueueClient,
createSandboxCallbackBridgeAsset,
@@ -22,6 +24,7 @@ import {
sandboxCallbackBridgeDirectories,
startSandboxCallbackBridgeServer,
startSandboxCallbackBridgeWorker,
syncRemoteTextFileWithHashSkip,
} from "./sandbox-callback-bridge.js";
import {
createSandboxRunLogTailFactory,
@@ -113,6 +116,12 @@ export interface PreparedAdapterExecutionTargetRuntime {
workspaceRemoteDir: string | null;
runtimeRootDir: string | null;
assetDirs: Record<string, string>;
/**
* Remote directory of each additional (referenced) project that staged
* successfully, keyed by `projectId`. Empty for a local target or when no
* additional sources were requested.
*/
additionalSourceDirs: Record<string, string>;
restoreWorkspace(onProgress?: RuntimeProgressSink): Promise<void>;
}
@@ -1106,6 +1115,8 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
workspaceExclude?: string[];
preserveAbsentOnRestore?: string[];
assets?: AdapterManagedRuntimeAsset[];
/** Referenced (additional) projects to stage into the sandbox as plain, read-only trees. */
additionalSources?: SandboxAdditionalSource[];
installCommand?: string | null;
/** When provided alongside `installCommand`, skip the install if the binary is already on PATH. */
detectCommand?: string | null;
@@ -1123,6 +1134,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
workspaceRemoteDir: null,
runtimeRootDir: null,
assetDirs: {},
additionalSourceDirs: {},
restoreWorkspace: async () => {},
};
}
@@ -1136,6 +1148,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
workspaceRemoteDir: input.workspaceRemoteDir,
syncWorkspace: input.syncWorkspace,
assets: input.assets,
additionalSources: input.additionalSources,
onProgress: input.onProgress,
});
return {
@@ -1143,6 +1156,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
workspaceRemoteDir: prepared.workspaceRemoteDir,
runtimeRootDir: prepared.runtimeRootDir,
assetDirs: prepared.assetDirs,
additionalSourceDirs: prepared.additionalSourceDirs,
restoreWorkspace: prepared.restoreWorkspace,
};
}
@@ -1166,6 +1180,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
workspaceExclude: input.workspaceExclude,
preserveAbsentOnRestore: input.preserveAbsentOnRestore,
assets: input.assets,
additionalSources: input.additionalSources,
installCommand: input.installCommand,
detectCommand: input.detectCommand,
onProgress: input.onProgress,
@@ -1176,6 +1191,7 @@ export async function prepareAdapterExecutionTargetRuntime(input: {
workspaceRemoteDir: prepared.workspaceRemoteDir,
runtimeRootDir: prepared.runtimeRootDir,
assetDirs: prepared.assetDirs,
additionalSourceDirs: prepared.additionalSourceDirs,
restoreWorkspace: prepared.restoreWorkspace,
};
}
@@ -1258,11 +1274,33 @@ async function writeProcessSessionProxyScript(dir: string, port: number, token:
return proxyPath;
}
// Content-hash-skip the process-session remote script write, mirroring the
// sandbox callback bridge entrypoint sha256 gate. The script is a static
// Paperclip-authored `.mjs` that only changes when the build changes, so on a
// warm start (same sandbox, script already present) the single sha-gate exec
// skips the ~3-exec base64 upload entirely. `syncRemoteTextFileWithHashSkip`
// fails loud on a check error rather than silently re-uploading.
async function syncProcessSessionRemoteScript(input: {
client: ReturnType<typeof createCommandManagedSandboxCallbackBridgeQueueClient>;
runner: CommandManagedRuntimeRunner;
remoteCwd: string;
remoteScriptDir: string;
remoteScriptPath: string;
}): Promise<void> {
await input.client.writeTextFile(input.remoteScriptPath, getProcessSessionRemoteSource());
timeoutMs?: number | null;
shellCommand?: "bash" | "sh" | null;
}): Promise<{ uploaded: boolean }> {
const { uploaded } = await syncRemoteTextFileWithHashSkip({
runner: input.runner,
remoteCwd: input.remoteCwd,
remoteDir: input.remoteScriptDir,
remotePath: input.remoteScriptPath,
body: getProcessSessionRemoteSource(),
label: "Process session remote script",
action: "sync process session remote script",
lockDir: path.posix.join(input.remoteScriptDir, ".paperclip-process-session-script.lock"),
timeoutMs: input.timeoutMs,
shellCommand: input.shellCommand,
});
return { uploaded };
}
async function readRemoteJsonFiles(input: {
@@ -1303,7 +1341,12 @@ export async function startAdapterExecutionTargetProcessSessionBridge(input: {
command: string;
args: string[];
cwd: string;
env: Record<string, string>;
// The launch env is consumed ONLY when building the base64 `commandPayload`
// below — never during the env-INDEPENDENT dir/script setup. Accepting a
// resolver (in addition to a plain object) lets a caller overlap that setup
// with other work — e.g. starting the paperclip callback bridge — and hand the
// merged env in right before the launch.
env: Record<string, string> | (() => Promise<Record<string, string>>);
timeoutSec?: number | null;
onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
}): Promise<AdapterExecutionTargetProcessSessionBridgeHandle | null> {
@@ -1335,15 +1378,27 @@ export async function startAdapterExecutionTargetProcessSessionBridge(input: {
shellCommand,
});
await client.makeDir(stdinDir);
await client.makeDir(eventsDir);
await syncProcessSessionRemoteScript({ client, remoteScriptPath });
// The launch exec below re-creates stdinDir and eventsDir with one `mkdir -p`,
// and the remote script also creates them on start. No reader touches the two
// directories before the launch exec runs, so upfront makeDir execs are redundant.
await syncProcessSessionRemoteScript({
runner,
remoteCwd: target.remoteCwd,
remoteScriptDir: bridgeRuntimeDir,
remoteScriptPath,
timeoutMs,
shellCommand,
});
// Resolve the launch env AFTER the env-independent setup above, so a caller
// can defer it until an upstream dependency (e.g. the paperclip bridge's env)
// is ready without blocking the dir/script setup.
const launchEnv = typeof input.env === "function" ? await input.env() : input.env;
const commandPayload = Buffer.from(JSON.stringify({
command: input.command,
args: input.args,
cwd: input.cwd || target.remoteCwd,
env: sanitizeRemoteExecutionEnv(input.env),
env: sanitizeRemoteExecutionEnv(launchEnv),
}), "utf8").toString("base64");
await onLog("stdout", `[paperclip] Starting ACP process session bridge in sandbox (${target.providerKey ?? "provider"}).\n`);
packages/adapter-utils/src/remote-managed-runtime.test.ts
+92 -1
View File
@@ -15,7 +15,7 @@ const {
stdout: Buffer.from('{"token":"remote"}\n').toString("base64"),
stderr: "",
})),
syncDirectoryToSsh: vi.fn(async () => undefined),
syncDirectoryToSsh: vi.fn(async (_input: { localDir: string }) => undefined),
}));
vi.mock("./ssh.js", () => ({
@@ -92,4 +92,95 @@ describe("remote managed runtime", () => {
);
expect(restoredAuth).toBe('{"token":"remote"}\n');
});
it("stages each additional project into its own isolated SSH dir, isolating one failure", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-remote-runtime-additional-"));
cleanupDirs.push(rootDir);
const workspaceDir = path.join(rootDir, "workspace");
const firstDir = path.join(rootDir, "referenced-first");
const secondDir = path.join(rootDir, "referenced-second");
const brokenDir = path.join(rootDir, "referenced-broken");
await mkdir(workspaceDir, { recursive: true });
// The transfer rejects only for the broken project's directory.
syncDirectoryToSsh.mockImplementation(async (input: { localDir: string }) => {
if (input.localDir === brokenDir) throw new Error("ssh transfer failed");
return undefined;
});
const prepared = await prepareRemoteManagedRuntime({
spec: {
host: "127.0.0.1",
port: 2222,
username: "fixture",
remoteWorkspacePath: "/app",
remoteCwd: "/app",
privateKey: "PRIVATE KEY",
knownHosts: "KNOWN HOSTS",
strictHostKeyChecking: true,
},
runId: "run-additional",
adapterKey: "codex",
workspaceLocalDir: workspaceDir,
workspaceRemoteDir: "/app",
syncWorkspace: false,
additionalSources: [
{ localPath: firstDir, projectId: "first" },
{ localPath: brokenDir, projectId: "broken" },
{ localPath: secondDir, projectId: "second" },
],
});
// Each healthy project staged into its OWN isolated dir under the runtime
// root; the broken one is skipped, not fatal.
expect(Object.keys(prepared.additionalSourceDirs).sort()).toEqual(["first", "second"]);
expect(prepared.additionalSourceDirs.first).toBe("/app/.paperclip-runtime/codex/project-first");
expect(prepared.additionalSourceDirs.second).toBe("/app/.paperclip-runtime/codex/project-second");
expect(prepared.additionalSourceDirs.broken).toBeUndefined();
expect(syncDirectoryToSsh).toHaveBeenCalledWith(expect.objectContaining({
localDir: firstDir,
remoteDir: "/app/.paperclip-runtime/codex/project-first",
}));
expect(syncDirectoryToSsh).toHaveBeenCalledWith(expect.objectContaining({
localDir: secondDir,
remoteDir: "/app/.paperclip-runtime/codex/project-second",
}));
});
it("skips an additional project whose localPath is not absolute", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-remote-runtime-relative-"));
cleanupDirs.push(rootDir);
const workspaceDir = path.join(rootDir, "workspace");
const healthyDir = path.join(rootDir, "referenced-healthy");
await mkdir(workspaceDir, { recursive: true });
const prepared = await prepareRemoteManagedRuntime({
spec: {
host: "127.0.0.1",
port: 2222,
username: "fixture",
remoteWorkspacePath: "/app",
remoteCwd: "/app",
privateKey: "PRIVATE KEY",
knownHosts: "KNOWN HOSTS",
strictHostKeyChecking: true,
},
runId: "run-relative",
adapterKey: "codex",
workspaceLocalDir: workspaceDir,
workspaceRemoteDir: "/app",
syncWorkspace: false,
additionalSources: [
{ localPath: "relative/referenced", projectId: "relative" },
{ localPath: healthyDir, projectId: "healthy" },
],
});
// The relative-path project never reaches the transfer and is skipped; the
// absolute-path project still stages.
expect(Object.keys(prepared.additionalSourceDirs)).toEqual(["healthy"]);
expect(syncDirectoryToSsh).not.toHaveBeenCalledWith(expect.objectContaining({
localDir: "relative/referenced",
}));
});
});
packages/adapter-utils/src/remote-managed-runtime.ts
+63 -1
View File
@@ -7,10 +7,26 @@ import {
restoreWorkspaceFromSshExecution,
syncDirectoryToSsh,
} from "./ssh.js";
import type { SandboxManagedRuntimeAssetRestoreContext } from "./sandbox-managed-runtime.js";
import type {
SandboxAdditionalSource,
SandboxManagedRuntimeAssetRestoreContext,
} from "./sandbox-managed-runtime.js";
import { captureDirectorySnapshot } from "./workspace-restore-merge.js";
import type { RuntimeProgressSink } from "./runtime-progress.js";
const REMOTE_ADDITIONAL_SOURCE_HEAVY_DIR_EXCLUDES = [
"node_modules",
"vendor",
"dist",
"build",
"out",
"coverage",
".next",
".turbo",
".cache",
".git",
].flatMap((entry) => [entry, `${entry}/*`, `*/${entry}`, `*/${entry}/*`]);
export interface RemoteManagedRuntimeAsset {
key: string;
localDir: string;
@@ -25,6 +41,12 @@ export interface PreparedRemoteManagedRuntime {
workspaceRemoteDir: string;
runtimeRootDir: string;
assetDirs: Record<string, string>;
/**
* Remote directory of each additional (referenced) project that staged
* successfully, keyed by `projectId`. A project whose staging failed is
* absent (per-project failure isolation).
*/
additionalSourceDirs: Record<string, string>;
restoreWorkspace(onProgress?: RuntimeProgressSink): Promise<void>;
}
@@ -86,6 +108,8 @@ export async function prepareRemoteManagedRuntime(input: {
workspaceRemoteDir?: string;
syncWorkspace?: boolean;
assets?: RemoteManagedRuntimeAsset[];
/** Referenced (additional) projects to stage as plain, read-only trees. */
additionalSources?: SandboxAdditionalSource[];
// Upload progress sink. Threaded for the byte-counting transport rewrite; the
// child task wires it into the workspace/asset transfers.
onProgress?: RuntimeProgressSink;
@@ -148,12 +172,50 @@ export async function prepareRemoteManagedRuntime(input: {
throw error;
}
// Stage each referenced (additional) project as a plain, read-only tree in its
// OWN isolated remote directory (`project-<projectId>`). Additional sources
// never get the anchor's git-history/overlay semantics. Per-project failure
// isolation: one project's failure logs a warning and is skipped; the run and
// the other projects continue (no workspace restore, unlike an asset failure).
const additionalSourceDirs: Record<string, string> = {};
for (const source of input.additionalSources ?? []) {
const { localPath, projectId } = source;
try {
if (!path.posix.isAbsolute(localPath)) {
throw new Error(`additional source localPath is not an absolute path: ${localPath}`);
}
if (
projectId.length === 0 ||
projectId.includes("/") ||
projectId.includes("\\") ||
projectId.includes("..")
) {
throw new Error(`additional source projectId is not a simple path segment: ${projectId}`);
}
const remoteDir = path.posix.join(runtimeRootDir, `project-${projectId}`);
await syncDirectoryToSsh({
spec: input.spec,
localDir: localPath,
remoteDir,
exclude: REMOTE_ADDITIONAL_SOURCE_HEAVY_DIR_EXCLUDES,
onProgress: input.onProgress,
progressLabel: `project-${projectId}`,
});
additionalSourceDirs[projectId] = remoteDir;
} catch (error) {
console.warn(
`[paperclip] Failed to stage referenced project ${projectId}; skipping it. ${String(error)}`,
);
}
}
return {
spec: input.spec,
workspaceLocalDir: input.workspaceLocalDir,
workspaceRemoteDir,
runtimeRootDir,
assetDirs,
additionalSourceDirs,
restoreWorkspace: async (onProgress?: RuntimeProgressSink) => {
if (preparedWorkspace && baselineSnapshot) {
await restoreWorkspaceFromSshExecution({
packages/adapter-utils/src/sandbox-callback-bridge.test.ts
+237
View File
@@ -13,6 +13,7 @@ import {
createSandboxCallbackBridgeAsset,
createSandboxCallbackBridgeToken,
sandboxCallbackBridgeDirectories,
syncRemoteTextFileWithHashSkip,
syncSandboxCallbackBridgeEntrypoint,
startSandboxCallbackBridgeServer,
startSandboxCallbackBridgeWorker,
@@ -437,6 +438,7 @@ describe("sandbox callback bridge", () => {
const worker = await startSandboxCallbackBridgeWorker({
client: {
makeDir: async () => {},
makeDirs: async () => {},
listJsonFiles: async () => {
throw new Error(
"list /remote/.paperclip-runtime/gemini/paperclip-bridge/queue/requests failed with exit code 255: kex_exchange_identification: read: Connection reset by peer",
@@ -862,6 +864,135 @@ describe("sandbox callback bridge", () => {
).resolves.toEqual([]);
});
// The process-session remote script is a static, Paperclip-authored `.mjs`
// written into the sandbox on every bridge start. `syncRemoteTextFileWithHashSkip`
// (which now backs that write, mirroring the bridge-entrypoint sha256 gate)
// content-hash-skips it so a warm start where the remote script already matches
// costs ZERO write execs instead of the prior ~3 (prepare/append/finalize base64
// upload).
it("test_process_session_script_skipped_when_remote_hash_matches: warm start with a matching remote hash writes 0 execs", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-hashskip-warm-"));
cleanupDirs.push(rootDir);
const remoteDir = path.join(rootDir, "runtime", "codex", "process-sessions");
const remotePath = path.posix.join(remoteDir, "paperclip-process-session-remote.mjs");
const lockDir = path.posix.join(remoteDir, ".paperclip-process-session-script.lock");
const body = "console.log('process session remote script v1');\n";
let execCount = 0;
const inner = createExecRunner();
const runner = {
execute: async (input: Parameters<typeof inner.execute>[0]) => {
execCount += 1;
return inner.execute(input);
},
};
const args = {
runner,
remoteCwd: rootDir,
remoteDir,
remotePath,
body,
label: "Process session remote script",
action: "sync process session remote script",
lockDir,
timeoutMs: 30_000,
} as const;
// Cold start: the script is uploaded (single sha-gate exec that writes).
const first = await syncRemoteTextFileWithHashSkip(args);
expect(first.uploaded).toBe(true);
await expect(readFile(remotePath, "utf8")).resolves.toBe(body);
// Warm start: the remote hash matches, so the write is skipped entirely.
execCount = 0;
const second = await syncRemoteTextFileWithHashSkip(args);
expect(second.uploaded).toBe(false);
// A single hash-gate round-trip that performed 0 writes (down from ~3 execs).
expect(execCount).toBe(1);
// sha is still returned on the skip path so callers get a well-formed result.
expect(second.sha256).toBe(first.sha256);
// The remote file is unchanged and no upload/partial/lock leftovers remain.
await expect(readFile(remotePath, "utf8")).resolves.toBe(body);
await expect(
readdir(remoteDir).then((entries) =>
entries.filter(
(entry) =>
entry.endsWith(".paperclip-upload.b64") ||
entry.endsWith(".partial") ||
entry === ".paperclip-process-session-script.lock",
),
),
).resolves.toEqual([]);
});
it("test_process_session_script_rewritten_on_hash_mismatch: a mismatched remote hash still rewrites the script", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-hashskip-cold-"));
cleanupDirs.push(rootDir);
const remoteDir = path.join(rootDir, "runtime", "codex", "process-sessions");
const remotePath = path.posix.join(remoteDir, "paperclip-process-session-remote.mjs");
const lockDir = path.posix.join(remoteDir, ".paperclip-process-session-script.lock");
const body = "console.log('process session remote script v2');\n";
// Pre-seed the remote with a DIFFERENT script (a prior/stale build).
await mkdir(remoteDir, { recursive: true });
await writeFile(remotePath, "console.log('stale remote script');\n", "utf8");
const result = await syncRemoteTextFileWithHashSkip({
runner: createExecRunner(),
remoteCwd: rootDir,
remoteDir,
remotePath,
body,
label: "Process session remote script",
action: "sync process session remote script",
lockDir,
timeoutMs: 30_000,
});
expect(result.uploaded).toBe(true);
await expect(readFile(remotePath, "utf8")).resolves.toBe(body);
});
it("fails loud when the hash-skip sync exec errors instead of silently re-uploading", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-hashskip-fail-"));
cleanupDirs.push(rootDir);
const remoteDir = path.join(rootDir, "runtime", "codex", "process-sessions");
const remotePath = path.posix.join(remoteDir, "paperclip-process-session-remote.mjs");
const lockDir = path.posix.join(remoteDir, ".paperclip-process-session-script.lock");
// A runner whose exec fails: the hash-gate cannot be evaluated. The write
// must surface the failure, never swallow it and re-upload behind a green
// return value.
const runner = {
execute: async () => ({
exitCode: 1,
signal: null,
timedOut: false,
stdout: "",
stderr: "hash gate boom",
pid: null,
startedAt: new Date().toISOString(),
}),
};
await expect(
syncRemoteTextFileWithHashSkip({
runner,
remoteCwd: rootDir,
remoteDir,
remotePath,
body: "console.log('never written');\n",
label: "Process session remote script",
action: "sync process session remote script",
lockDir,
timeoutMs: 30_000,
}),
).rejects.toThrow(/sync process session remote script/i);
// Nothing was written to the remote path on the failure path.
await expect(readFile(remotePath, "utf8")).rejects.toThrow();
});
it("permits the documented heartbeat surface and denies unrelated routes", () => {
const allowed: Array<{ method: string; path: string }> = [
{ method: "GET", path: "/api/agents/me" },
@@ -984,4 +1115,110 @@ describe("sandbox callback bridge", () => {
},
}));
});
it("creates the bridge queue directories in one directory-creation exec", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-bridge-makedirs-"));
cleanupDirs.push(rootDir);
const queueDir = path.posix.join(rootDir, "queue");
const directories = sandboxCallbackBridgeDirectories(queueDir);
const makeDir = vi.fn(async () => {});
const makeDirs = vi.fn(async () => {});
const worker = await startSandboxCallbackBridgeWorker({
client: {
makeDir,
makeDirs,
listJsonFiles: async () => [],
readTextFile: async () => {
throw new Error("unexpected readTextFile");
},
writeTextFile: async () => {},
rename: async () => {},
remove: async () => {},
},
queueDir,
authorizeRequest: async () => null,
handleRequest: async () => ({ status: 200, body: "ok" }),
});
await worker.stop();
expect(makeDir).not.toHaveBeenCalled();
expect(makeDirs).toHaveBeenCalledTimes(1);
expect(makeDirs).toHaveBeenCalledWith([
directories.rootDir,
directories.requestsDir,
directories.responsesDir,
directories.logsDir,
]);
});
it("falls back to sequential makeDir when the queue client omits makeDirs", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-bridge-makedir-fallback-"));
cleanupDirs.push(rootDir);
const queueDir = path.posix.join(rootDir, "queue");
const directories = sandboxCallbackBridgeDirectories(queueDir);
const makeDir = vi.fn(async (_remotePath: string) => {});
// A queue client that predates the batched makeDirs method. The worker
// must still create every queue directory through sequential makeDir.
const worker = await startSandboxCallbackBridgeWorker({
client: {
makeDir,
listJsonFiles: async () => [],
readTextFile: async () => {
throw new Error("unexpected readTextFile");
},
writeTextFile: async () => {},
rename: async () => {},
remove: async () => {},
},
queueDir,
authorizeRequest: async () => null,
handleRequest: async () => ({ status: 200, body: "ok" }),
});
await worker.stop();
expect(makeDir.mock.calls.map((call) => call[0])).toEqual([
directories.rootDir,
directories.requestsDir,
directories.responsesDir,
directories.logsDir,
]);
});
it("runs one mkdir -p exec for makeDirs on the command-managed queue client", async () => {
const runner = {
execute: vi.fn(async (_input: { args?: string[] }) => ({
exitCode: 0,
signal: null,
timedOut: false,
stdout: "",
stderr: "",
pid: null,
startedAt: new Date().toISOString(),
})),
};
const client = createCommandManagedSandboxCallbackBridgeQueueClient({
runner,
remoteCwd: "/workspace",
timeoutMs: 30_000,
});
// The command-managed client always provides the batched makeDirs method.
expect(client.makeDirs).toBeDefined();
await client.makeDirs?.(["/workspace/a", "/workspace/b", "/workspace/c"]);
expect(runner.execute).toHaveBeenCalledTimes(1);
const call = runner.execute.mock.calls[0][0];
const script = call.args?.[call.args.length - 1] ?? "";
expect(script).toContain("mkdir -p");
expect(script).toContain("/workspace/a");
expect(script).toContain("/workspace/b");
expect(script).toContain("/workspace/c");
});
});
packages/adapter-utils/src/sandbox-callback-bridge.ts
+99 -25
View File
@@ -149,6 +149,11 @@ export interface SandboxCallbackBridgeDirectories {
export interface SandboxCallbackBridgeQueueClient {
makeDir(remotePath: string): Promise<void>;
// Optional batched directory create. The built-in clients create every
// queue directory in one remote exec. A client that predates this method
// omits it; the worker falls back to sequential `makeDir` calls, so an
// external implementation stays compatible without a change.
makeDirs?(remotePaths: string[]): Promise<void>;
listJsonFiles(remotePath: string): Promise<string[]>;
readTextFile(remotePath: string): Promise<string>;
writeTextFile(remotePath: string, body: string): Promise<void>;
@@ -359,6 +364,11 @@ export function createFileSystemSandboxCallbackBridgeQueueClient(): SandboxCallb
makeDir: async (remotePath) => {
await fs.mkdir(remotePath, { recursive: true });
},
makeDirs: async (remotePaths) => {
for (const remotePath of remotePaths) {
await fs.mkdir(remotePath, { recursive: true });
}
},
listJsonFiles: async (remotePath) => {
const entries = await fs.readdir(remotePath, { withFileTypes: true }).catch(() => []);
return entries
@@ -470,6 +480,13 @@ export function createCommandManagedSandboxCallbackBridgeQueueClient(input: {
makeDir: async (remotePath) => {
await runChecked(`mkdir ${remotePath}`, `mkdir -p ${shellQuote(remotePath)}`);
},
makeDirs: async (remotePaths) => {
if (remotePaths.length === 0) {
return;
}
const quoted = remotePaths.map((remotePath) => shellQuote(remotePath));
await runChecked(`mkdir ${remotePaths.join(" ")}`, `mkdir -p ${quoted.join(" ")}`);
},
listJsonFiles: async (remotePath) => {
const result = await runShell(
input.runner,
@@ -606,10 +623,21 @@ export async function startSandboxCallbackBridgeWorker(input: {
const pollIntervalMs = normalizeTimeoutMs(input.pollIntervalMs, DEFAULT_BRIDGE_POLL_INTERVAL_MS);
const maxBodyBytes = normalizeTimeoutMs(input.maxBodyBytes, DEFAULT_BRIDGE_MAX_BODY_BYTES);
const directories = sandboxCallbackBridgeDirectories(input.queueDir);
await input.client.makeDir(directories.rootDir);
await input.client.makeDir(directories.requestsDir);
await input.client.makeDir(directories.responsesDir);
await input.client.makeDir(directories.logsDir);
const queueDirectories = [
directories.rootDir,
directories.requestsDir,
directories.responsesDir,
directories.logsDir,
];
if (input.client.makeDirs) {
await input.client.makeDirs(queueDirectories);
} else {
// Backward-compatible fallback for a queue client that omits the batched
// makeDirs method. Create each queue directory with a single makeDir.
for (const directory of queueDirectories) {
await input.client.makeDir(directory);
}
}
let stopping = false;
let inFlight = 0;
@@ -777,34 +805,54 @@ export async function startSandboxCallbackBridgeWorker(input: {
};
}
export async function syncSandboxCallbackBridgeEntrypoint(input: {
/**
* Content-hash-skip write of a Paperclip-authored text file into the sandbox, in
* a SINGLE remote exec. The body's sha256 is computed on the host; the one shell
* round-trip skips the write entirely when the remote file already hashes to the
* same value (warm start — 0 write execs), otherwise it uploads (base64 over
* stdin), verifies the decoded bytes, and atomically renames into place. A
* PID-liveness lock serializes concurrent writers to the same path and the
* verify step guards against a torn upload.
*
* Fail loudly: a non-zero remote exit (surfaced by `requireSuccessfulResult`) or
* malformed result JSON throws rather than silently re-uploading and masking a
* failed check. The only intentional degradation is when the remote has neither
* `sha256sum` nor `shasum` — then the skip cannot be proven and we conservatively
* re-upload (and the post-upload verify is best-effort, as noted inline).
*/
export async function syncRemoteTextFileWithHashSkip(input: {
runner: CommandManagedRuntimeRunner;
remoteCwd: string;
assetRemoteDir: string;
bridgeAsset: SandboxCallbackBridgeAsset;
remoteDir: string;
remotePath: string;
body: string;
// Human-readable noun phrase used in fail-loud messages, e.g.
// "Sandbox callback bridge entrypoint" / "Process session remote script".
label: string;
// Short action label for `requireSuccessfulResult`, e.g.
// "sync sandbox callback bridge entrypoint".
action: string;
lockDir: string;
timeoutMs?: number | null;
shellCommand?: "bash" | "sh" | null;
}): Promise<{ remoteEntrypoint: string; sha256: string; uploaded: boolean }> {
}): Promise<{ uploaded: boolean; sha256: string }> {
const timeoutMs = normalizeTimeoutMs(input.timeoutMs, DEFAULT_BRIDGE_RESPONSE_TIMEOUT_MS);
const shellCommand = preferredShellForSandbox(input.shellCommand);
const remoteEntrypoint = path.posix.join(input.assetRemoteDir, SANDBOX_CALLBACK_BRIDGE_ENTRYPOINT);
const remoteEntrypointPartial = `${remoteEntrypoint}.partial`;
const remoteUploadPath = `${remoteEntrypoint}.paperclip-upload.b64`;
const remoteLockDir = path.posix.join(input.assetRemoteDir, ".paperclip-bridge-upload.lock");
const entrypointSource = await fs.readFile(input.bridgeAsset.entrypoint, "utf8");
const entrypointBase64 = toBuffer(Buffer.from(entrypointSource, "utf8")).toString("base64");
const sha256 = createHash("sha256").update(entrypointSource, "utf8").digest("hex");
const remotePartial = `${input.remotePath}.partial`;
const remoteUploadPath = `${input.remotePath}.paperclip-upload.b64`;
const base64Body = toBuffer(Buffer.from(input.body, "utf8")).toString("base64");
const sha256 = createHash("sha256").update(input.body, "utf8").digest("hex");
const syncResult = await runShell(
input.runner,
input.remoteCwd,
[
"set -eu",
`remote_dir=${shellQuote(input.assetRemoteDir)}`,
`remote_path=${shellQuote(remoteEntrypoint)}`,
`remote_partial=${shellQuote(remoteEntrypointPartial)}`,
`remote_dir=${shellQuote(input.remoteDir)}`,
`remote_path=${shellQuote(input.remotePath)}`,
`remote_partial=${shellQuote(remotePartial)}`,
`remote_upload=${shellQuote(remoteUploadPath)}`,
`lock_dir=${shellQuote(remoteLockDir)}`,
`lock_dir=${shellQuote(input.lockDir)}`,
`expected_sha=${shellQuote(sha256)}`,
"hash_file() {",
" if command -v sha256sum >/dev/null 2>&1; then",
@@ -818,7 +866,7 @@ export async function syncSandboxCallbackBridgeEntrypoint(input: {
" return 127",
"}",
"mkdir -p \"$remote_dir\"",
...buildRemotePidLockAcquireScript("\"$lock_dir\"", "Timed out acquiring sandbox callback bridge upload lock."),
...buildRemotePidLockAcquireScript("\"$lock_dir\"", `Timed out acquiring ${input.label} upload lock.`),
...buildRemotePidLockCleanupScript("\"$lock_dir\"", [
"rm -f \"$remote_upload\" \"$remote_partial\"",
]),
@@ -839,30 +887,56 @@ export async function syncSandboxCallbackBridgeEntrypoint(input: {
// best-effort and we trust base64-decode + atomic rename below.
"if partial_sha=\"$(hash_file \"$remote_partial\" 2>/dev/null)\"; then",
" if [ \"$partial_sha\" != \"$expected_sha\" ]; then",
" echo \"Sandbox callback bridge entrypoint upload sha mismatch.\" >&2",
` echo ${shellQuote(`${input.label} upload sha mismatch.`)} >&2`,
" exit 1",
" fi",
"else",
" echo \"Sandbox callback bridge entrypoint sha verify skipped: no sha256sum/shasum on remote.\" >&2",
` echo ${shellQuote(`${input.label} sha verify skipped: no sha256sum/shasum on remote.`)} >&2`,
"fi",
"mv \"$remote_partial\" \"$remote_path\"",
"printf '{\"uploaded\":true}\\n'",
].join("\n"),
timeoutMs,
shellCommand,
entrypointBase64,
base64Body,
);
requireSuccessfulResult("sync sandbox callback bridge entrypoint", syncResult);
requireSuccessfulResult(input.action, syncResult);
let uploaded = false;
try {
uploaded = JSON.parse(syncResult.stdout.trim())?.uploaded === true;
} catch (error) {
throw new Error(
`Sandbox callback bridge sync wrote invalid result JSON: ${error instanceof Error ? error.message : String(error)}`,
`${input.label} sync wrote invalid result JSON: ${error instanceof Error ? error.message : String(error)}`,
);
}
return { uploaded, sha256 };
}
export async function syncSandboxCallbackBridgeEntrypoint(input: {
runner: CommandManagedRuntimeRunner;
remoteCwd: string;
assetRemoteDir: string;
bridgeAsset: SandboxCallbackBridgeAsset;
timeoutMs?: number | null;
shellCommand?: "bash" | "sh" | null;
}): Promise<{ remoteEntrypoint: string; sha256: string; uploaded: boolean }> {
const remoteEntrypoint = path.posix.join(input.assetRemoteDir, SANDBOX_CALLBACK_BRIDGE_ENTRYPOINT);
const entrypointSource = await fs.readFile(input.bridgeAsset.entrypoint, "utf8");
const { uploaded, sha256 } = await syncRemoteTextFileWithHashSkip({
runner: input.runner,
remoteCwd: input.remoteCwd,
remoteDir: input.assetRemoteDir,
remotePath: remoteEntrypoint,
body: entrypointSource,
label: "Sandbox callback bridge entrypoint",
action: "sync sandbox callback bridge entrypoint",
lockDir: path.posix.join(input.assetRemoteDir, ".paperclip-bridge-upload.lock"),
timeoutMs: input.timeoutMs,
shellCommand: input.shellCommand,
});
return {
remoteEntrypoint,
sha256,
packages/adapter-utils/src/sandbox-file-sync.test.ts
+188 -17
View File
@@ -21,8 +21,10 @@ interface RecordingClient {
}
// A filesystem-backed client that additionally exposes native syncIn/syncOut,
// mirroring a provider that opted into the sync verbs. The native transfer is a
// faithful destroy-then-replace directory copy honoring followSymlinks.
// mirroring a provider that opted into the sync verbs and HONORS an operation's
// ordered `postUploadCommands` after its files land (PR-2 contract: execute or
// fail-closed, never silently ignore). The native transfer is a faithful copy
// honoring followSymlinks; single-file mappings stream verbatim.
function makeNativeClient(): RecordingClient {
const syncInOps: SandboxSyncOperation[][] = [];
const syncOutOps: SandboxSyncOperation[][] = [];
@@ -53,6 +55,10 @@ function makeNativeClient(): RecordingClient {
filesTransferred += 1;
}
}
// Honor the operation's ordered post-upload commands (PR-2), fail-fast.
for (const command of operation.postUploadCommands ?? []) {
await execFile("sh", ["-c", command.command], { maxBuffer: 32 * 1024 * 1024 });
}
return { operationId: operation.operationId, filesTransferred, bytesTransferred: 0 };
})),
});
@@ -106,16 +112,25 @@ describe("sandbox native file sync", () => {
assets: [{ key: "skills", localDir: localAssetsDir }],
});
// The default-provision asset was transferred through syncIn as a single
// directory mapping with an opaque operationId; the file landed in place.
// Both the workspace and the default-provision asset stage through syncIn:
// each uploads a single tar as a `file` mapping with an opaque operationId,
// and the extract runs as the operation's ordered post-upload command.
const inboundOps = syncInOps.flat();
expect(inboundOps.length).toBe(1);
const assetOp = inboundOps[0];
expect(assetOp.operationId).toMatch(/^sync-op-\d+$/);
expect(assetOp.operationId).not.toContain("skills");
expect(assetOp.files).toEqual([
{ sourcePath: localAssetsDir, targetPath: prepared.assetDirs.skills, kind: "directory", exclude: undefined, followSymlinks: undefined },
]);
expect(inboundOps.length).toBe(2);
const assetOp = inboundOps.find((op) =>
op.files.some((mapping) => mapping.targetPath.endsWith("skills-upload.tar")),
);
expect(assetOp).toBeDefined();
expect(assetOp!.operationId).toMatch(/^sync-op-\d+$/);
expect(assetOp!.operationId).not.toContain("skills");
expect(assetOp!.files).toHaveLength(1);
expect(assetOp!.files[0]).toMatchObject({
targetPath: path.posix.join(prepared.runtimeRootDir, "skills-upload.tar"),
kind: "file",
});
// Default provision → a plain destroy-then-replace tar extract post-command.
expect(assetOp!.postUploadCommands).toHaveLength(1);
expect(assetOp!.postUploadCommands![0].command).toContain("tar -xf");
expect(await readFile(path.join(prepared.assetDirs.skills, "skill.md"), "utf8")).toBe("skill body\n");
// Mutate the sandbox workspace, then restore through the native outbound path.
@@ -130,7 +145,54 @@ describe("sandbox native file sync", () => {
expect(await readFile(path.join(localWorkspaceDir, "new.txt"), "utf8")).toBe("added\n");
});
it("keeps a custom-provision asset on the tar fallback even when native sync is available", async () => {
it("stamps the run-specific timeout onto every delegated post-upload command", async () => {
// The extract/wipe/merge commands are delegated to the provider through
// `syncIn` as `postUploadCommands`. They MUST carry the run-specific timeout
// (`spec.timeoutMs`) — the same limit the pre-syncIn code passed to
// `client.run` — not the provider sync client's own default. When the two
// differ, a command left without a `timeoutMs` outlives (or is killed under)
// the wrong limit; here a distinctive `spec.timeoutMs` proves propagation.
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-timeout-"));
cleanupDirs.push(rootDir);
const localWorkspaceDir = path.join(rootDir, "local-workspace");
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
const defaultAssetDir = path.join(rootDir, "default-asset");
const customAssetDir = path.join(rootDir, "custom-asset");
await mkdir(localWorkspaceDir, { recursive: true });
await mkdir(defaultAssetDir, { recursive: true });
await mkdir(customAssetDir, { recursive: true });
await writeFile(path.join(localWorkspaceDir, "README.md"), "ws\n", "utf8");
await writeFile(path.join(defaultAssetDir, "skill.md"), "skill\n", "utf8");
await writeFile(path.join(customAssetDir, "cred.txt"), "secret\n", "utf8");
const runTimeoutMs = 7_000;
const { client, syncInOps } = makeNativeClient();
await prepareSandboxManagedRuntime({
spec: { transport: "sandbox", provider: "test", sandboxId: "s1", remoteCwd: remoteWorkspaceDir, timeoutMs: runTimeoutMs, apiKey: null },
adapterKey: "test-adapter",
client,
workspaceLocalDir: localWorkspaceDir,
assets: [
{ key: "skills", localDir: defaultAssetDir },
{
key: "creds",
localDir: customAssetDir,
provision: { postUploadCommand: ({ assetTarPath, assetDir }) =>
`rm -rf ${assetDir} && mkdir -p ${assetDir} && tar -xf ${assetTarPath} -C ${assetDir} && rm -f ${assetTarPath}` },
},
],
});
// Workspace extract + default-asset extract + custom-provision merge — every
// delegated command across every operation carries the run timeout.
const commands = syncInOps.flat().flatMap((op) => op.postUploadCommands ?? []);
expect(commands.length).toBeGreaterThanOrEqual(3);
for (const command of commands) {
expect(command.timeoutMs).toBe(runTimeoutMs);
}
});
it("routes a custom-provision asset through syncIn with its bespoke post-upload command (native)", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-custom-"));
cleanupDirs.push(rootDir);
const localWorkspaceDir = path.join(rootDir, "local-workspace");
@@ -150,18 +212,127 @@ describe("sandbox native file sync", () => {
assets: [{
key: "creds",
localDir: localAssetsDir,
// A bespoke extract command (e.g. a credential merge) cannot be a generic
// file mapping, so the orchestrator keeps it on the tar path.
provision: { extractCommand: ({ assetTarPath, assetDir }) =>
// A bespoke post-upload command (e.g. a credential merge) rides syncIn as
// the operation's ordered post-upload command — no native-diversion gate.
provision: { postUploadCommand: ({ assetTarPath, assetDir }) =>
`rm -rf ${assetDir} && mkdir -p ${assetDir} && tar -xf ${assetTarPath} -C ${assetDir} && rm -f ${assetTarPath}` },
}],
});
// No syncIn operation for the custom asset; it still materializes via tar.
expect(syncInOps.flat().length).toBe(0);
// The custom asset now rides syncIn (native uploadFiles), carrying its
// bespoke command as the operation's ordered post-upload command.
const credsOp = syncInOps.flat().find((op) =>
op.files.some((mapping) => mapping.targetPath.endsWith("creds-upload.tar")),
);
expect(credsOp).toBeDefined();
expect(credsOp!.files.every((mapping) => mapping.kind === "file")).toBe(true);
expect(credsOp!.postUploadCommands).toHaveLength(1);
expect(credsOp!.postUploadCommands![0].command).toContain("tar -xf");
expect(await readFile(path.join(prepared.assetDirs.creds, "cred.txt"), "utf8")).toBe("secret\n");
});
it("stages each additional project into its own isolated dir via a native directory syncIn", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-additional-"));
cleanupDirs.push(rootDir);
const localWorkspaceDir = path.join(rootDir, "local-workspace");
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
await mkdir(localWorkspaceDir, { recursive: true });
await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor\n", "utf8");
// Three referenced projects, each with a distinctive file, staged as plain
// read-only trees.
const projects = [
{ projectId: "alpha", localDir: path.join(rootDir, "src-alpha"), file: "alpha.txt", body: "alpha body\n" },
{ projectId: "bravo", localDir: path.join(rootDir, "src-bravo"), file: "bravo.txt", body: "bravo body\n" },
{ projectId: "charlie", localDir: path.join(rootDir, "src-charlie"), file: "charlie.txt", body: "charlie body\n" },
];
for (const project of projects) {
await mkdir(project.localDir, { recursive: true });
await writeFile(path.join(project.localDir, project.file), project.body, "utf8");
}
const { client, syncInOps } = makeNativeClient();
const prepared = await prepareSandboxManagedRuntime({
spec: { transport: "sandbox", provider: "test", sandboxId: "s1", remoteCwd: remoteWorkspaceDir, timeoutMs: 30_000, apiKey: null },
adapterKey: "test-adapter",
client,
workspaceLocalDir: localWorkspaceDir,
additionalSources: projects.map((project) => ({ localPath: project.localDir, projectId: project.projectId })),
});
// Each project lands in its OWN `project-<projectId>` directory under the
// runtime root, and its file materializes there.
const projectDirs = projects.map((project) => {
const dir = prepared.additionalSourceDirs[project.projectId];
expect(dir).toBe(path.posix.join(prepared.runtimeRootDir, `project-${project.projectId}`));
return dir;
});
for (const [index, project] of projects.entries()) {
expect(await readFile(path.join(projectDirs[index], project.file), "utf8")).toBe(project.body);
}
// The target dirs are pairwise distinct and never nested inside one another.
for (const outer of projectDirs) {
for (const inner of projectDirs) {
if (outer === inner) continue;
expect(inner.startsWith(`${outer}/`)).toBe(false);
}
}
// Each project rides its own `syncIn` operation as a single `directory`
// mapping, source = the host checkout dir, target = the isolated project dir.
const inboundOps = syncInOps.flat();
for (const [index, project] of projects.entries()) {
const op = inboundOps.find((candidate) =>
candidate.files.some((mapping) => mapping.targetPath === projectDirs[index]),
);
expect(op).toBeDefined();
expect(op!.operationId).toMatch(/^sync-op-\d+$/);
expect(op!.operationId).not.toContain(project.projectId);
expect(op!.files).toHaveLength(1);
expect(op!.files[0]).toMatchObject({
sourcePath: project.localDir,
targetPath: projectDirs[index],
kind: "directory",
});
// Plain read-only tree — no post-upload extract/wipe/merge command.
expect(op!.postUploadCommands ?? []).toHaveLength(0);
}
});
it("isolates one additional project's sync failure and stages the rest", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-additional-fail-"));
cleanupDirs.push(rootDir);
const localWorkspaceDir = path.join(rootDir, "local-workspace");
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
const goodDir = path.join(rootDir, "src-good");
await mkdir(localWorkspaceDir, { recursive: true });
await mkdir(goodDir, { recursive: true });
await writeFile(path.join(localWorkspaceDir, "README.md"), "anchor\n", "utf8");
await writeFile(path.join(goodDir, "good.txt"), "good body\n", "utf8");
// The middle source points at a directory that does not exist, so its native
// transfer fails. Failure isolation must skip only it and stage the rest.
const { client } = makeNativeClient();
const prepared = await prepareSandboxManagedRuntime({
spec: { transport: "sandbox", provider: "test", sandboxId: "s1", remoteCwd: remoteWorkspaceDir, timeoutMs: 30_000, apiKey: null },
adapterKey: "test-adapter",
client,
workspaceLocalDir: localWorkspaceDir,
additionalSources: [
{ localPath: goodDir, projectId: "good-a" },
{ localPath: path.join(rootDir, "does-not-exist"), projectId: "broken" },
{ localPath: goodDir, projectId: "good-b" },
],
});
// Both healthy projects staged; the broken one is absent, not fatal.
expect(Object.keys(prepared.additionalSourceDirs).sort()).toEqual(["good-a", "good-b"]);
expect(prepared.additionalSourceDirs.broken).toBeUndefined();
expect(await readFile(path.join(prepared.additionalSourceDirs["good-a"], "good.txt"), "utf8")).toBe("good body\n");
expect(await readFile(path.join(prepared.additionalSourceDirs["good-b"], "good.txt"), "utf8")).toBe("good body\n");
});
it("dereferences symlinks only when followSymlinks is true (native honors the flag)", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-native-symlink-"));
cleanupDirs.push(rootDir);
packages/adapter-utils/src/sandbox-managed-runtime.test.ts
+886 -5
View File
File diff suppressed because it is too large. Load diff
packages/adapter-utils/src/sandbox-managed-runtime.ts
+391 -169
View File
@@ -1,4 +1,5 @@
import { execFile as execFileCallback } from "node:child_process";
import { randomUUID } from "node:crypto";
import { constants as fsConstants, promises as fs } from "node:fs";
import os from "node:os";
import path from "node:path";
@@ -55,7 +56,7 @@ export interface SandboxRemoteExecutionSpec {
}
/**
* Remote paths handed to an asset's `provision.extractCommand`. All are POSIX
* Remote paths handed to an asset's `provision.postUploadCommand`. All are POSIX
* paths inside the sandbox: `assetTarPath` is the uploaded asset tarball,
* `assetDir` is where the asset should be materialized, and `runtimeRootDir`
* is the directory any `stageFiles` were written into.
@@ -68,23 +69,32 @@ export interface SandboxManagedRuntimeAssetProvisionContext {
/**
* Per-asset inbound provisioning contribution. The core is adapter-agnostic:
* an asset that supplies neither `stageFiles` nor `extractCommand` is extracted
* with a plain `tar -xf`. An adapter that needs custom provisioning (e.g. a
* credential merge) supplies helper files via `stageFiles` and the shell
* command that consumes them via `extractCommand`.
* an asset that supplies neither `stageFiles` nor `postUploadCommand` is
* materialized with a plain destroy-then-replace `tar -xf`. An adapter that
* needs custom provisioning (e.g. a credential merge) supplies helper files via
* `stageFiles` and the shell command that consumes them via `postUploadCommand`.
*
* Both contributions ride the unified {@link SandboxSyncOperation} the core
* builds per asset: `stageFiles` become additional `files` mappings placed
* alongside the asset tar, and `postUploadCommand` becomes the operation's
* ordered `postUploadCommands`. See {@link SandboxPostUploadCommand} for the
* command-origin / confinement security contract (C1–C3).
*/
export interface SandboxManagedRuntimeAssetProvision {
/**
* Extra files written into `runtimeRootDir` (alongside the asset tar) before
* the extract command runs — typically helper scripts the extract command
* Extra files placed into `runtimeRootDir` (alongside the asset tar) before
* the post-upload command runs — typically helper scripts the command
* invokes. Contents may be raw bytes or a UTF-8 string.
*/
stageFiles?: { name: string; contents: Buffer | string }[];
/**
* Builds the shell command that materializes the uploaded asset tar into
* `assetDir`. Defaults to a plain `tar -xf` extraction when omitted.
* Builds the opaque, adapter-authored shell command that materializes the
* uploaded asset tar into `assetDir`, run as the operation's ordered
* post-upload command after every mapping has landed. Defaults to a plain
* destroy-then-replace `tar -xf` extraction when omitted. Any path embedded in
* the command MUST be built from already-confined paths and shell-quoted (C3).
*/
extractCommand?: (ctx: SandboxManagedRuntimeAssetProvisionContext) => string;
postUploadCommand?: (ctx: SandboxManagedRuntimeAssetProvisionContext) => string;
}
/**
@@ -111,6 +121,22 @@ export interface SandboxManagedRuntimeAsset {
restore?: (ctx: SandboxManagedRuntimeAssetRestoreContext) => Promise<void>;
}
/**
* A referenced (additional) project to stage into the run sandbox as a plain,
* read-only tree. `localPath` is the host checkout directory. Upstream code
* already authorized and realized this directory (`project:read`); this layer
* adds no authorization logic. `projectId` names the isolated remote
* subdirectory (`project-<projectId>` under the runtime root) the tree lands in.
*
* Additional sources are plain trees only. They never carry the anchor
* workspace's git-history, overlay, or `.paperclip-runtime` preservation
* semantics — those stay anchor-only.
*/
export interface SandboxAdditionalSource {
localPath: string;
projectId: string;
}
/**
* Per-call byte-level progress hook. `transferredBytes`/`totalBytes` are decoded
* file bytes (not the base64 wire size). `totalBytes` is null when the size is
@@ -135,6 +161,53 @@ export interface SandboxSyncFileMapping {
mode?: number;
exclude?: string[];
followSymlinks?: boolean;
/**
* Advisory read-write intent for the sandbox target. `"rw"` marks a target the
* agent may change and keep; `"ro"` marks a read-only tree. An absent value
* defaults to `"ro"` (read-only is the safe default for an advisory signal).
* The field is advisory metadata for an optional sandbox feedback wrapper. It
* does not change the transfer and adds no security.
*/
access?: "rw" | "ro";
/**
* The sandbox directory that becomes read-write when `access` is `"rw"` and a
* post-upload command extracts `targetPath` into a different directory. A tar
* mapping uploads an archive under the runtime root, so its `targetPath` is the
* staging archive, not the directory the extract command fills. This field
* names that final destination directory. When absent, the read-write
* destination is the parent directory of `targetPath`. Advisory; ignored when
* `access` is not `"rw"`.
*/
writablePath?: string;
}
/**
* A control command run against the sandbox after a sync operation's files have
* landed. Mirrors the plugin SDK `PluginPostUploadCommand`; kept as a local
* structural type so `adapter-utils` does not depend on the plugin SDK. Ordered
* within {@link SandboxSyncOperation.postUploadCommands} and executed in array
* order, fail-fast (first non-zero exit or timeout aborts the operation).
*
* SECURITY — command origin (Stage-1 design review, condition C1). `command` is
* a **Paperclip/adapter-authored control operation**: it may be supplied ONLY by
* core/adapter code. No server route, issue/comment content, project/workspace
* file content, provider-plugin callback, or arbitrary adapter config may supply
* a raw `command` string; any path embedded in it MUST be built by adapter/core
* helpers from already-confined paths and shell-quoted (C3). Providers treat the
* command as **opaque** — execute or reject, never rewrite/concatenate/append.
*/
export interface SandboxPostUploadCommand {
/** The opaque, adapter-authored shell command to run after upload. */
command: string;
/**
* Working directory for the command. When present, MUST be an absolute POSIX
* path confined under the operation's allowed sandbox target root (C2). When
* absent, defaults to the runtime's stable command cwd — never a process
* default cwd.
*/
cwd?: string;
/** Optional per-command timeout in milliseconds. */
timeoutMs?: number;
}
/**
@@ -146,6 +219,13 @@ export interface SandboxSyncFileMapping {
export interface SandboxSyncOperation {
operationId: string;
files: SandboxSyncFileMapping[];
/**
* Optional ordered control commands run after this operation's files land, in
* array order, fail-fast. Absent means "no commands" — byte-identical to a
* pre-contract operation. See {@link SandboxPostUploadCommand} for the command
* origin/confinement security contract (C1–C4).
*/
postUploadCommands?: SandboxPostUploadCommand[];
}
export interface SandboxSyncResult {
@@ -213,6 +293,13 @@ export interface PreparedSandboxManagedRuntime {
workspaceRemoteDir: string;
runtimeRootDir: string;
assetDirs: Record<string, string>;
/**
* Remote directory of each additional (referenced) project that staged
* successfully, keyed by `projectId`. A project whose staging failed is
* absent (per-project failure isolation). Empty when no additional sources
* were requested.
*/
additionalSourceDirs: Record<string, string>;
restoreWorkspace(onProgress?: RuntimeProgressSink): Promise<void>;
}
@@ -244,6 +331,43 @@ function buildDefaultExtractRuntimeAssetCommand(input: {
`rm -f ${shellQuote(input.remoteAssetTar)}`;
}
// Named builder (Security Condition C3): extract an uploaded workspace tarball
// into `workspaceRemoteDir`, then remove the tarball. When `wipeExceptNames` is
// present the target's direct children (except the preserved names) are removed
// before extraction (destroy-then-replace); when null the tarball is overlaid on
// top of the existing tree (e.g. a git overlay merge). Every path is
// shell-quoted; no untrusted value is concatenated into the shell (C1/C3).
function buildWorkspaceTarExtractCommand(input: {
workspaceRemoteDir: string;
remoteTar: string;
wipeExceptNames: string[] | null;
}): string {
const wipe = input.wipeExceptNames
? ` && find ${shellQuote(input.workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ` +
`${preserveFindArgs(input.wipeExceptNames)} -exec rm -rf -- {} +`
: "";
return (
`mkdir -p ${shellQuote(input.workspaceRemoteDir)}${wipe} && ` +
`tar -xf ${shellQuote(input.remoteTar)} -C ${shellQuote(input.workspaceRemoteDir)} && ` +
`rm -f ${shellQuote(input.remoteTar)}`
);
}
// Named builder (C3): remove paths deleted in the host git worktree from the
// sandbox workspace. Every path is shell-quoted; the caller supplies only
// already-confined relative paths from the git snapshot.
function buildRemoveDeletedPathsCommand(input: {
remoteDir: string;
deletedPaths: string[];
}): string {
const quotedPaths = input.deletedPaths.map((entry) => shellQuote(entry)).join(" ");
return `cd ${shellQuote(input.remoteDir)} && rm -rf -- ${quotedPaths}`;
}
function buildUniqueStagingPath(input: { targetPath: string; suffix: string }): string {
return `${input.targetPath}${input.suffix}.${randomUUID()}`;
}
export function parseSandboxRemoteExecutionSpec(value: unknown): SandboxRemoteExecutionSpec | null {
const parsed = asObject(value);
const transport = asString(parsed.transport).trim();
@@ -314,7 +438,7 @@ async function execTar(args: string[]): Promise<void> {
});
}
async function createTarballFromDirectory(input: {
export async function createTarballFromDirectory(input: {
localDir: string;
archivePath: string;
exclude?: string[];
@@ -397,10 +521,17 @@ async function copyWorkspaceEntry(sourceRoot: string, targetRoot: string, relati
return;
}
await fs.copyFile(sourcePath, targetPath, fsConstants.COPYFILE_FICLONE).catch(async () => {
await fs.copyFile(sourcePath, targetPath);
});
await fs.chmod(targetPath, stats.mode);
const stagedTargetPath = buildUniqueStagingPath({ targetPath, suffix: ".paperclip-copy" });
await fs.rm(stagedTargetPath, { recursive: true, force: true }).catch(() => undefined);
try {
await fs.copyFile(sourcePath, stagedTargetPath, fsConstants.COPYFILE_FICLONE).catch(async () => {
await fs.copyFile(sourcePath, stagedTargetPath);
});
await fs.chmod(stagedTargetPath, stats.mode);
await fs.rename(stagedTargetPath, targetPath);
} finally {
await fs.rm(stagedTargetPath, { recursive: true, force: true }).catch(() => undefined);
}
}
export async function mirrorDirectory(
@@ -443,10 +574,6 @@ async function copySelectedWorkspaceEntries(input: {
}
}
function toArrayBuffer(bytes: Buffer): ArrayBuffer {
return Uint8Array.from(bytes).buffer;
}
function toBuffer(bytes: Buffer | Uint8Array | ArrayBuffer): Buffer {
if (Buffer.isBuffer(bytes)) return bytes;
if (bytes instanceof ArrayBuffer) return Buffer.from(bytes);
@@ -493,20 +620,6 @@ function preserveFindArgs(entries: string[]): string {
return entries.map((entry) => `! -name ${shellQuote(entry)}`).join(" ");
}
async function removeDeletedPathsInSandbox(input: {
client: SandboxManagedRuntimeClient;
spec: SandboxRemoteExecutionSpec;
remoteDir: string;
deletedPaths: string[];
}): Promise<void> {
if (input.deletedPaths.length === 0) return;
const quotedPaths = input.deletedPaths.map((entry) => shellQuote(entry)).join(" ");
await input.client.run(
`sh -c ${shellQuote(`cd ${shellQuote(input.remoteDir)} && rm -rf -- ${quotedPaths}`)}`,
{ timeoutMs: input.spec.timeoutMs },
);
}
// Bridge a single byte-level transfer to the throttled progress reporter. The
// transport reports decoded bytes via `options.onProgress`; the reporter turns
// them into a throttled, fully-formatted log line. `finish()` emits the terminal
@@ -565,6 +678,12 @@ export async function prepareSandboxManagedRuntime(input: {
workspaceExclude?: string[];
preserveAbsentOnRestore?: string[];
assets?: SandboxManagedRuntimeAsset[];
/**
* Referenced (additional) projects to stage into the sandbox as plain,
* read-only trees, each in its own isolated `project-<projectId>` directory.
* Defaults to none, so a legacy/anchor-only call is behavior-identical.
*/
additionalSources?: SandboxAdditionalSource[];
// Upload progress sink. Threaded for the byte-counting transport rewrite; the
// child task wires it into writeFile/readFile.
onProgress?: RuntimeProgressSink;
@@ -593,67 +712,145 @@ export async function prepareSandboxManagedRuntime(input: {
? await captureDirectorySnapshot(input.workspaceLocalDir, { exclude: restoreExclude })
: null;
// Prefer the provider's native file transport when it advertised the sync
// verbs; otherwise every branch below falls back to the byte-identical tar +
// base64 `writeFile`/`run` path. `nextSyncOperationId` emits opaque, ordered,
// non-sensitive tokens — never a caller/asset identifier.
const nativeSyncIn = typeof input.client.syncIn === "function";
// Every inbound staging step delegates to the provider through `client.syncIn`:
// the orchestrator no longer inlines `writeFile`+`run` or chooses a transport,
// and there is no `usesCustomProvision` native-diversion gate. `syncIn` is
// ALWAYS present in production — the command-managed client exposes a native
// transport (Daytona/Kubernetes `uploadFiles` + provider-executed post-upload
// commands) or a byte-identical base64-tar fallback that reproduces the prior
// `writeFile`+`run` sequence. Require it explicitly so a misconfigured client
// fails loud rather than silently skipping staging. `syncOut` stays optional
// (native-only) with a tar fallback on the restore path below.
const syncIn = input.client.syncIn;
if (typeof syncIn !== "function") {
throw new Error(
"prepareSandboxManagedRuntime requires a client that exposes syncIn " +
"(createCommandManagedRuntimeClient provides a native-or-fallback implementation).",
);
}
const nativeSyncOut = typeof input.client.syncOut === "function";
let syncOperationSeq = 0;
// Opaque, ordered, non-sensitive operation tokens — never a caller/asset id.
const nextSyncOperationId = () => `sync-op-${++syncOperationSeq}`;
// Remote directory of each additional (referenced) project that stages
// successfully, keyed by projectId. A project that fails to stage is absent.
const additionalSourceDirs: Record<string, string> = {};
// Additional projects stage as plain trees. Drop the heavy build/cache dirs a
// reference tree does not need, and `.git` — additional sources never carry
// git-history semantics (anchor-only).
const additionalSourceExclude = mergeExcludes(SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES, [".git"]);
// Every delegated post-upload command (extract/wipe/remove-deleted/asset merge)
// must run under the run-specific timeout (`spec.timeoutMs`), not the provider
// sync client's default timeout — the two can differ, and before staging was
// routed through `syncIn` each of these ran via
// `client.run(cmd, { timeoutMs: spec.timeoutMs })`. When they mismatch, a
// command left without a `timeoutMs` outlives (or is killed under) the wrong
// limit. Stamp the run timeout onto every delegated command, preserving any
// command that already carries its own explicit timeout.
const withRunTimeout = (
commands: SandboxPostUploadCommand[],
): SandboxPostUploadCommand[] =>
commands.map((command) => ({
...command,
timeoutMs: command.timeoutMs ?? input.spec.timeoutMs,
}));
await withTempDir("paperclip-sandbox-sync-", async (tempDir) => {
const preservedNames = new Set([
".paperclip-runtime",
...(gitSnapshot ? [".git"] : []),
...(input.preserveAbsentOnRestore ?? []),
]);
if (syncWorkspace && gitSnapshot) {
await emitRuntimeStatus(input.onRuntimeProgress, "git_sync", "Syncing git history to sandbox");
await withShallowGitWorkspaceClone({
localDir: input.workspaceLocalDir,
snapshot: gitSnapshot,
}, async (cloneDir) => {
// The git-workspace and workspace transfers preserve `.paperclip-runtime`
// on the target (and the git overlay merges on top rather than replacing),
// which the generic destroy-then-replace file mapping cannot express, so
// they always take the tar path. Native transfer is used for the clean
// destroy-then-replace cases (default-provision assets inbound; the
// workspace download into a fresh host dir outbound).
const gitTarPath = path.join(tempDir, "git-workspace.tar");
await createTarballFromDirectory({
localDir: cloneDir,
archivePath: gitTarPath,
exclude: [".paperclip-runtime"],
});
const gitTarBytes = await fs.readFile(gitTarPath);
const remoteGitTar = path.posix.join(runtimeRootDir, "git-workspace-upload.tar");
await input.client.makeDir(runtimeRootDir);
const gitUpload = makeTransferProgress(
input.onProgress,
"Syncing",
"to",
"git history",
{ sink: input.onRuntimeProgress, phase: "git_sync" },
);
await input.client.writeFile(remoteGitTar, toArrayBuffer(gitTarBytes), gitUpload.options);
await gitUpload.finish(gitTarBytes.byteLength, gitTarBytes.byteLength);
await input.client.run(
`sh -c ${shellQuote(
`mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
`find ${shellQuote(workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ${preserveFindArgs([".paperclip-runtime"])} -exec rm -rf -- {} + && ` +
`tar -xf ${shellQuote(remoteGitTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
`rm -f ${shellQuote(remoteGitTar)}`,
)}`,
{ timeoutMs: input.spec.timeoutMs },
);
// Stage one source directory into an isolated remote subdirectory through the
// unified `syncIn` seam. Both the workspace/asset tar path and the additional
// (referenced) project path use it: build one `SandboxSyncOperation` from the
// caller's `files` mappings (a host tarball as a single `file` mapping, or a
// whole `directory` mapping), carry any extract/wipe/merge steps as ordered
// `postUploadCommands`, confine the operation's source and target to their
// own roots (fail-closed), and delegate to `syncIn` (native transfer, or the
// base64-tar fallback). `finish` emits the terminal progress line.
const stageConfinedSyncIn = async (params: {
files: SandboxSyncFileMapping[];
postUploadCommands?: SandboxPostUploadCommand[];
sourceRoots: string[];
targetRoots: string[];
progressLabel: string;
statusPhase: RuntimeStatusPhase;
progressBytes: number;
}): Promise<void> => {
const operations: SandboxSyncOperation[] = [{
operationId: nextSyncOperationId(),
files: params.files,
...(params.postUploadCommands
? { postUploadCommands: withRunTimeout(params.postUploadCommands) }
: {}),
}];
assertSyncOperationsConfined(operations, {
sourceRoots: params.sourceRoots,
targetRoots: params.targetRoots,
});
}
const upload = makeTransferProgress(
input.onProgress,
"Syncing",
"to",
params.progressLabel,
{ sink: input.onRuntimeProgress, phase: params.statusPhase },
);
await syncIn(operations);
await upload.finish(params.progressBytes, params.progressBytes);
};
if (syncWorkspace) {
// A git-backed workspace and a plain workspace both stage through ONE
// confined `syncIn` operation. A git-backed workspace carries TWO host tars —
// the git-history clone and the working-tree overlay — as two `file` mappings
// on the SAME operation, with their extract commands as ordered
// `postUploadCommands`. One operation shares one mkdir, one confine guard, one
// `uploadFiles`, and one rename exec, so the second `syncIn` round trip is
// removed. Build the whole merged file set and command list BEFORE the confine
// guard runs (inside `stageConfinedSyncIn`); never append a mapping after it.
const workspaceFiles: SandboxSyncFileMapping[] = [];
const workspacePostUploadCommands: SandboxPostUploadCommand[] = [];
let workspaceUploadBytes = 0;
// 1. git-history tar (git-backed workspace only). Both tar targets live under
// `runtimeRootDir` (`.paperclip-runtime/<adapterKey>`). The git extract
// wipes the target tree EXCEPT `.paperclip-runtime`, so the overlay tar,
// which sits under `.paperclip-runtime`, survives to run its own extract.
if (gitSnapshot) {
await emitRuntimeStatus(input.onRuntimeProgress, "git_sync", "Syncing git history to sandbox");
const gitTarPath = path.join(tempDir, "git-workspace.tar");
const remoteGitTar = path.posix.join(runtimeRootDir, "git-workspace-upload.tar");
await withShallowGitWorkspaceClone({
localDir: input.workspaceLocalDir,
snapshot: gitSnapshot,
}, async (cloneDir) => {
await createTarballFromDirectory({
localDir: cloneDir,
archivePath: gitTarPath,
exclude: [".paperclip-runtime"],
});
});
workspaceFiles.push({ sourcePath: gitTarPath, targetPath: remoteGitTar, kind: "file", access: "rw", writablePath: workspaceRemoteDir });
workspacePostUploadCommands.push({
command: buildWorkspaceTarExtractCommand({
workspaceRemoteDir,
remoteTar: remoteGitTar,
wipeExceptNames: [".paperclip-runtime"],
}),
});
workspaceUploadBytes += (await fs.stat(gitTarPath)).size;
}
// 2. workspace-overlay tar. A git-backed overlay merges on top of the just
// extracted git tree (no wipe); a plain workspace wipes every child except
// the preserved names first. The extract runs AFTER the git extract.
await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing workspace to sandbox");
const workspaceTarPath = path.join(tempDir, "workspace.tar");
const workspaceArchiveDir = gitSnapshot ? path.join(tempDir, "workspace-overlay") : input.workspaceLocalDir;
await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing workspace to sandbox");
if (gitSnapshot) {
await copySelectedWorkspaceEntries({
sourceDir: input.workspaceLocalDir,
@@ -667,81 +864,50 @@ export async function prepareSandboxManagedRuntime(input: {
archivePath: workspaceTarPath,
exclude: gitSnapshot ? undefined : workspaceArchiveExclude,
});
const workspaceTarBytes = await fs.readFile(workspaceTarPath);
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-upload.tar");
await input.client.makeDir(runtimeRootDir);
const workspaceUpload = makeTransferProgress(
input.onProgress,
"Syncing",
"to",
"workspace",
{ sink: input.onRuntimeProgress, phase: "config_sync" },
);
await input.client.writeFile(
remoteWorkspaceTar,
toArrayBuffer(workspaceTarBytes),
workspaceUpload.options,
);
await workspaceUpload.finish(workspaceTarBytes.byteLength, workspaceTarBytes.byteLength);
const extractWorkspaceTarCommand = gitSnapshot
? `mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
`tar -xf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
`rm -f ${shellQuote(remoteWorkspaceTar)}`
: `mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
`find ${shellQuote(workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ${preserveFindArgs([...preservedNames])} -exec rm -rf -- {} + && ` +
`tar -xf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
`rm -f ${shellQuote(remoteWorkspaceTar)}`;
await input.client.run(
`sh -c ${shellQuote(extractWorkspaceTarCommand)}`,
{ timeoutMs: input.spec.timeoutMs },
);
if (gitSnapshot) {
await removeDeletedPathsInSandbox({
client: input.client,
spec: input.spec,
remoteDir: workspaceRemoteDir,
deletedPaths: gitSnapshot.deletedPaths,
workspaceFiles.push({ sourcePath: workspaceTarPath, targetPath: remoteWorkspaceTar, kind: "file", access: "rw", writablePath: workspaceRemoteDir });
workspacePostUploadCommands.push({
command: buildWorkspaceTarExtractCommand({
workspaceRemoteDir,
remoteTar: remoteWorkspaceTar,
wipeExceptNames: gitSnapshot ? null : [...preservedNames],
}),
});
// 3. Optional remove-deleted-paths command runs LAST, after both extracts.
if (gitSnapshot && gitSnapshot.deletedPaths.length > 0) {
workspacePostUploadCommands.push({
command: buildRemoveDeletedPathsCommand({
remoteDir: workspaceRemoteDir,
deletedPaths: gitSnapshot.deletedPaths,
}),
});
}
workspaceUploadBytes += (await fs.stat(workspaceTarPath)).size;
// One confined `syncIn` for the whole merged workspace file set. The confine
// guard covers every mapping BEFORE any bytes upload (fail-closed): a source
// or target escape in EITHER tar mapping stops the upload of both.
await stageConfinedSyncIn({
files: workspaceFiles,
postUploadCommands: workspacePostUploadCommands,
sourceRoots: [tempDir],
targetRoots: [runtimeRootDir],
progressLabel: "workspace",
statusPhase: "config_sync",
progressBytes: workspaceUploadBytes,
});
}
for (const asset of input.assets ?? []) {
await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing runtime assets to sandbox");
const remoteAssetDir = path.posix.join(runtimeRootDir, asset.key);
// Assets with custom provisioning (staged helper files or a bespoke extract
// command such as a credential merge) do more than a plain directory
// replacement, so they cannot be expressed as a generic file mapping and
// always take the tar path. A default-provisioned asset is a clean
// destroy-then-replace of its own directory, which the native transport
// reproduces exactly.
const usesCustomProvision =
Boolean(asset.provision?.extractCommand) || (asset.provision?.stageFiles?.length ?? 0) > 0;
if (nativeSyncIn && !usesCustomProvision) {
const operations: SandboxSyncOperation[] = [{
operationId: nextSyncOperationId(),
files: [{
sourcePath: asset.localDir,
targetPath: remoteAssetDir,
kind: "directory",
exclude: asset.exclude,
followSymlinks: asset.followSymlinks,
}],
}];
assertSyncOperationsConfined(operations, {
sourceRoots: [asset.localDir],
targetRoots: [runtimeRootDir],
});
const assetUpload = makeTransferProgress(
input.onProgress,
"Syncing",
"to",
asset.key,
{ sink: input.onRuntimeProgress, phase: "config_sync" },
);
await input.client.syncIn!(operations);
await assetUpload.finish(0, 0);
continue;
}
const remoteAssetTar = path.posix.join(runtimeRootDir, `${asset.key}-upload.tar`);
// Every asset — default OR custom-provisioned (e.g. an adapter credential
// merge) — rides one `syncIn` operation: the asset tar plus any staged
// helper files as `files` mappings, and the extract/merge command as the
// ordered post-upload command. There is no native-diversion gate; a
// custom-provisioned asset's bytes now ride native `uploadFiles` and its
// command runs as a provider-executed post-upload command.
const assetTarPath = path.join(tempDir, `${asset.key}.tar`);
await createTarballFromDirectory({
localDir: asset.localDir,
@@ -749,39 +915,94 @@ export async function prepareSandboxManagedRuntime(input: {
followSymlinks: asset.followSymlinks,
exclude: asset.exclude,
});
const assetTarBytes = await fs.readFile(assetTarPath);
const remoteAssetTar = path.posix.join(runtimeRootDir, `${asset.key}-upload.tar`);
const assetUpload = makeTransferProgress(
input.onProgress,
"Syncing",
"to",
asset.key,
{ sink: input.onRuntimeProgress, phase: "config_sync" },
);
await input.client.writeFile(remoteAssetTar, toArrayBuffer(assetTarBytes), assetUpload.options);
await assetUpload.finish(assetTarBytes.byteLength, assetTarBytes.byteLength);
const files: SandboxSyncFileMapping[] = [
{ sourcePath: assetTarPath, targetPath: remoteAssetTar, kind: "file", access: "rw", writablePath: remoteAssetDir },
];
// Stage provision helper files (e.g. the merge scripts) into the temp dir
// and map them alongside the asset tar so they ride the same native upload.
for (const stageFile of asset.provision?.stageFiles ?? []) {
const stageBytes = typeof stageFile.contents === "string"
? Buffer.from(stageFile.contents)
: stageFile.contents;
const safeName = stageFile.name;
if (/[\\/]|\.\.(\.|$)/.test(safeName) || safeName === "..") {
throw new Error(`provision stageFile.name must be a simple basename, got: ${safeName}`);
}
await input.client.writeFile(
path.posix.join(runtimeRootDir, safeName),
toArrayBuffer(stageBytes),
);
const stageBytes = typeof stageFile.contents === "string"
? Buffer.from(stageFile.contents)
: stageFile.contents;
const stageHostPath = path.join(tempDir, `${asset.key}.stage.${safeName}`);
await fs.writeFile(stageHostPath, stageBytes);
// A stage helper file (for example a merge script) is a read-only input
// that the provision command reads; the agent does not change it and does
// not keep it. So it is `access: "ro"` and never joins the writable set.
files.push({
sourcePath: stageHostPath,
targetPath: path.posix.join(runtimeRootDir, safeName),
kind: "file",
access: "ro",
});
}
const extractCommand = asset.provision?.extractCommand?.({
const postUploadCommand = asset.provision?.postUploadCommand?.({
assetTarPath: remoteAssetTar,
assetDir: remoteAssetDir,
runtimeRootDir,
}) ?? buildDefaultExtractRuntimeAssetCommand({ remoteAssetDir, remoteAssetTar });
await input.client.run(
`sh -c ${shellQuote(extractCommand)}`,
{ timeoutMs: input.spec.timeoutMs },
);
const assetTarSize = (await fs.stat(assetTarPath)).size;
await stageConfinedSyncIn({
files,
postUploadCommands: [{ command: postUploadCommand }],
sourceRoots: [tempDir],
targetRoots: [runtimeRootDir],
progressLabel: asset.key,
statusPhase: "config_sync",
progressBytes: assetTarSize,
});
}
// Stage each referenced (additional) project as a plain, read-only tree in
// its OWN isolated remote directory (`project-<projectId>`). An additional
// project rides one confined `syncIn` directory mapping — a native directory
// transfer, or the base64-tar fallback — with source and target confined to
// their own roots. No workspace, git-history, or `.paperclip-runtime`
// semantics apply; those stay anchor-only. Per-project failure isolation: one
// project's confinement or sync failure logs a warning and is skipped, and
// the run plus the other projects continue. Only a project that stages
// successfully appears in `additionalSourceDirs`.
for (const source of input.additionalSources ?? []) {
const { localPath, projectId } = source;
const label = `project-${projectId}`;
try {
if (!path.posix.isAbsolute(localPath)) {
throw new Error(`additional source localPath is not an absolute path: ${localPath}`);
}
if (
projectId.length === 0 ||
projectId.includes("/") ||
projectId.includes("\\") ||
projectId.includes("..")
) {
throw new Error(`additional source projectId is not a simple path segment: ${projectId}`);
}
const remoteProjectDir = path.posix.join(runtimeRootDir, label);
await emitRuntimeStatus(input.onRuntimeProgress, "config_sync", "Syncing referenced project to sandbox");
await stageConfinedSyncIn({
files: [{
sourcePath: localPath,
targetPath: remoteProjectDir,
kind: "directory",
exclude: additionalSourceExclude,
access: "ro",
}],
sourceRoots: [localPath],
targetRoots: [remoteProjectDir],
progressLabel: label,
statusPhase: "config_sync",
progressBytes: 0,
});
additionalSourceDirs[projectId] = remoteProjectDir;
} catch (error) {
console.warn(
`[paperclip] Failed to stage referenced project ${projectId}; skipping it. ${String(error)}`,
);
}
}
});
@@ -795,6 +1016,7 @@ export async function prepareSandboxManagedRuntime(input: {
workspaceRemoteDir,
runtimeRootDir,
assetDirs,
additionalSourceDirs,
restoreWorkspace: async (onProgress?: RuntimeProgressSink) => {
const restoreSink = onProgress ?? input.onProgress;
if (!syncWorkspace) {
packages/adapter-utils/src/server-utils.test.ts
+46
View File
@@ -843,6 +843,8 @@ describe("renderPaperclipWakePrompt", () => {
expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain("evidence, not valid liveness paths by themselves");
expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain("keep `in_progress` only when a live continuation path exists");
expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain("Prefer the smallest verification that proves the change");
expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain("After 2 consecutive failures of the same control-plane write");
expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain("adapter/runtime status channel as the sanctioned fallback");
expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain("Use child issues");
expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain("instead of polling agents, sessions, or processes");
expect(DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE).toContain("Create child issues directly when you know what needs to be done");
@@ -905,6 +907,8 @@ describe("renderPaperclipWakePrompt", () => {
expect(prompt).toContain("Immediately before returning, verify that Paperclip records one of those dispositions");
expect(prompt).toContain("a successful process exit or final response is not sufficient");
expect(prompt).toContain("If no valid disposition is recorded, record it now and do not end the run");
expect(prompt).toContain("After 2 consecutive failures of the same control-plane write");
expect(prompt).toContain("adapter/runtime status channel as the sanctioned fallback");
expect(prompt).toContain("evidence, not valid liveness paths by themselves");
expect(prompt).toContain("Use child issues for long or parallel delegated work instead of polling");
expect(prompt).toContain("named unblock owner/action");
@@ -969,6 +973,48 @@ describe("renderPaperclipWakePrompt", () => {
expect(prompt).toContain("- recovery attempt: 2/3");
expect(prompt).toContain("- next action: Restore the execution path.");
expect(prompt).not.toContain("Execution contract: take concrete action");
if (cause === "successful_run_missing_state") {
expect(prompt).not.toContain("Any comment you post on the source issue must be ≤3 lines");
} else {
expect(prompt).toContain("Record the outcome in the resolve call's `resolutionNote`");
expect(prompt).toContain("Any comment you post on the source issue must be ≤3 lines");
expect(prompt).toContain("No headings, no run-by-run narrative.");
}
});
it("asks process-loss retries to lead with the work instead of narrating recovery", () => {
const prompt = renderPaperclipWakePrompt({
reason: "source_scoped_recovery_action",
issue: { id: "issue-1", identifier: "PAP-14092", title: "Recover work", status: "blocked" },
recovery: {
cause: "process_lost",
failureSummary: "adapter stopped",
originalAssignee: { id: "agent-1", name: "Coder" },
attemptCount: 1,
nextAction: "Restore the execution path.",
},
commentWindow: { requestedCount: 0, includedCount: 0, missingCount: 0 },
comments: [],
fallbackFetchNeeded: false,
});
expect(prompt).toContain(
"Do not narrate the recovery in your next comment — at most one short sentence; lead with the work.",
);
});
it("asks restored source owners to lead with work instead of narrating recovery", () => {
const prompt = renderPaperclipWakePrompt({
reason: "issue_recovery_action_restored",
issue: { id: "issue-1", identifier: "PAP-14092", title: "Continue work", status: "todo" },
commentWindow: { requestedCount: 0, includedCount: 0, missingCount: 0 },
comments: [],
fallbackFetchNeeded: false,
});
expect(prompt).toContain(
"Do not narrate the recovery in your next comment — at most one short sentence; lead with the work.",
);
});
it("keeps exactly one execution contract in a composed fresh heartbeat prompt", () => {
packages/adapter-utils/src/server-utils.ts
+10 -2
View File
@@ -158,6 +158,7 @@ export const DEFAULT_PAPERCLIP_AGENT_PROMPT_TEMPLATE = [
"- Comments, documents, screenshots, work products, and `Remaining` bullets are evidence, not valid liveness paths by themselves.",
"- Final disposition checklist: mark `done` when complete; use `in_review` only with a real reviewer, approval, interaction, or monitor path; use `blocked` only with first-class blockers or a named unblock owner/action; create delegated follow-up issues with blockers when another agent owns the next step; keep `in_progress` only when a live continuation path exists.",
"- Prefer the smallest verification that proves the change; do not default to full workspace typecheck/build/test on every heartbeat unless the task scope warrants it.",
"- After 2 consecutive failures of the same control-plane write, stop retrying that write for the rest of the heartbeat. Continue useful work, report the failure in the final response, and rely on the adapter/runtime status channel as the sanctioned fallback.",
"- Use child issues for parallel or long delegated work instead of polling agents, sessions, or processes.",
"- If woken by a human comment on a dependency-blocked issue, respond or triage the comment without treating the blocked deliverable work as unblocked.",
"- Create child issues directly when you know what needs to be done; use issue-thread interactions when the board/user must choose suggested tasks, answer structured questions, or confirm a proposal.",
@@ -1436,7 +1437,7 @@ export function renderPaperclipWakePrompt(
const recoveryInstruction = (() => {
switch (recovery?.cause) {
case "process_lost":
return `Your previous run on this issue was lost (${recovery.failureSummary ?? "no failure summary available"}). Try again — resume from durable progress; don't redo completed steps.`;
return `Your previous run on this issue was lost (${recovery.failureSummary ?? "no failure summary available"}). Try again — resume from durable progress; don't redo completed steps. Do not narrate the recovery in your next comment — at most one short sentence; lead with the work.`;
case "successful_run_missing_state":
case "successful_run_missing_issue_disposition":
return "Your run completed but left no final disposition. Post a comment summarizing the state and set the correct disposition (`done` / `in_review` / `blocked` / `in_progress` with a live path). Do not start new work.";
@@ -1479,12 +1480,16 @@ export function renderPaperclipWakePrompt(
? [
"Recovery contract: your job is to RECOVER this task, not to do the work. Do not produce the deliverable yourself.",
`Cause-specific instruction: ${recoveryInstruction}`,
...(recovery?.cause === "successful_run_missing_state" ||
recovery?.cause === "successful_run_missing_issue_disposition"
? []
: ["Record the outcome in the resolve call's `resolutionNote`. Any comment you post on the source issue must be ≤3 lines (cause → what you did → hand-back). No headings, no run-by-run narrative."]),
`Fallback preference order: (1) send back to ${originalAssigneeLabel} with a retry instruction; (2) fix the runtime/adapter/workspace problem, then send it back; (3) reassign to another agent with the right specialty; (4) convert to an explicit manual-review state for the board.`,
"",
]
: includeExecutionContract
? [
"Execution contract: take concrete action in this heartbeat when the issue is actionable; do not stop at a plan unless planning was requested. Leave durable progress and then give the issue a clear final disposition before ending the heartbeat: `done`, `in_review` with a real reviewer/approval/interaction path, `blocked` with first-class blockers or a named unblock owner/action, delegated follow-up issues with blockers, or `in_progress` only when a live continuation path exists. Immediately before returning, verify that Paperclip records one of those dispositions; a successful process exit or final response is not sufficient. If no valid disposition is recorded, record it now and do not end the run. Use child issues for long or parallel delegated work instead of polling. Comments, documents, screenshots, work products, and `Remaining` bullets are evidence, not valid liveness paths by themselves.",
"Execution contract: take concrete action in this heartbeat when the issue is actionable; do not stop at a plan unless planning was requested. Leave durable progress and then give the issue a clear final disposition before ending the heartbeat: `done`, `in_review` with a real reviewer/approval/interaction path, `blocked` with first-class blockers or a named unblock owner/action, delegated follow-up issues with blockers, or `in_progress` only when a live continuation path exists. Immediately before returning, verify that Paperclip records one of those dispositions; a successful process exit or final response is not sufficient. If no valid disposition is recorded, record it now and do not end the run. After 2 consecutive failures of the same control-plane write, stop retrying it for the rest of the heartbeat, continue useful work, report the failure in the final response, and rely on the adapter/runtime status channel as the sanctioned fallback. Use child issues for long or parallel delegated work instead of polling. Comments, documents, screenshots, work products, and `Remaining` bullets are evidence, not valid liveness paths by themselves.",
"",
]
: [];
@@ -1510,6 +1515,9 @@ export function renderPaperclipWakePrompt(
: []),
]
: []),
...(normalized.reason === "issue_recovery_action_restored"
? ["- instruction: Do not narrate the recovery in your next comment — at most one short sentence; lead with the work."]
: []),
];
const lines = resumedSession
? [
packages/adapter-utils/src/ssh-fixture.test.ts
+43
View File
@@ -170,6 +170,49 @@ describe("ssh env-lab fixture", () => {
await stopSshEnvLabFixture(statePath);
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("builds a remote script that sources login profiles but no nvm", async () => {
const target = await buildSshSpawnTarget({
spec: {
host: "ssh.example.test",
port: 22,
username: "ssh-user",
remoteCwd: "/srv/paperclip/workspace",
remoteWorkspacePath: "/srv/paperclip/workspace",
privateKey: null,
knownHosts: null,
strictHostKeyChecking: true,
},
command: "node",
args: ["--version"],
env: { FOO: "bar" },
});
// The remote script rides the last ssh argument. The SSH target is an
// operator-configured host that can expose `node` only through a login
// profile, so the wrapper sources the profiles. It no longer sources
// `nvm.sh`; a profile that adds nvm still runs.
const remoteScript = String(target.args.at(-1) ?? "");
expect(remoteScript).not.toContain("nvm.sh");
expect(remoteScript).not.toContain("NVM_DIR");
// Source /etc/profile so a host that exposes the PATH through
// /etc/profile.d scripts still resolves node and the agent CLI.
expect(remoteScript).toContain("/etc/profile");
expect(remoteScript).toContain(".profile");
expect(remoteScript).toContain(".bash_profile");
expect(remoteScript).toContain(".zprofile");
// Fall back to .bashrc when no .bash_profile exists, so a host that adds
// nvm in .bashrc still resolves node under a non-login SSH command.
expect(remoteScript).toContain(".bashrc");
// The last ssh argument wraps the script as `sh -c '...'`, so the inner
// quotes are escaped. Assert the command still runs: cd, env, and the argv.
expect(remoteScript).toContain("cd ");
expect(remoteScript).toContain("/srv/paperclip/workspace");
expect(remoteScript).toContain("exec env ");
expect(remoteScript).toContain("node");
expect(remoteScript).toContain("--version");
await target.cleanup();
});
it("rejects invalid environment variable keys when constructing SSH spawn targets", async () => {
await expect(
buildSshSpawnTarget({
packages/adapter-utils/src/ssh.ts
+26 -8
View File
@@ -1166,14 +1166,22 @@ export async function runSshCommand(
}
}
// Mirror buildSshSpawnTarget: source login profiles first, then run
// `env KEY=VAL cmd` so user-supplied identity overrides win over anything
// a profile re-exports. Without this, a remote profile that resets HOME
// / NVM_DIR / etc. would silently undo the explicit env passed in here.
// Mirror buildSshSpawnTarget: source the login profiles first, then run
// `env KEY=VAL cmd` so user-supplied identity overrides win over anything a
// profile re-exports. The SSH target is an operator-configured host, not a
// Paperclip sandbox image, so it can expose `node` or an agent CLI only
// through a login profile; a non-login SSH command would miss that PATH.
// Source `/etc/profile` first so a host that exposes the PATH through
// `/etc/profile.d` scripts still resolves node and the agent CLI.
// The script no longer sources `nvm.sh`; a profile that adds nvm still runs.
// .bash_profile typically sources .bashrc itself; only source .bashrc
// directly when no .bash_profile exists, so a host that adds nvm in
// .bashrc still resolves node without a double-run of the setup.
const envArgs = envEntries.map(([key, value]) => `${key}=${shellQuote(value)}`);
const remoteScript = [
'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
envArgs.length > 0
? `exec env ${envArgs.join(" ")} sh -c ${shellQuote(remoteCommand)}`
@@ -1223,12 +1231,22 @@ export async function buildSshSpawnTarget(input: {
.filter((entry): entry is [string, string] => typeof entry[1] === "string")
.map(([key, value]) => `${key}=${shellQuote(value)}`);
const remoteCommandParts = [shellQuote(input.command), ...input.args.map((arg) => shellQuote(arg))].join(" ");
// Source the login profiles first, then run `env KEY=VAL cmd` so
// user-supplied identity overrides win over anything a profile re-exports.
// The SSH target is an operator-configured host, not a Paperclip sandbox
// image, so it can expose `node` or an agent CLI only through a login
// profile; a non-login SSH command would miss that PATH. Source
// `/etc/profile` first so a host that exposes the PATH through
// `/etc/profile.d` scripts still resolves node and the agent CLI. The script
// no longer sources `nvm.sh`; a profile that adds nvm still runs.
// .bash_profile typically sources .bashrc itself; only source .bashrc
// directly when no .bash_profile exists, so a host that adds nvm in
// .bashrc still resolves node without a double-run of the setup.
const remoteScript = [
'if [ -f /etc/profile ]; then . /etc/profile >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.profile" ]; then . "$HOME/.profile" >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.bash_profile" ]; then . "$HOME/.bash_profile" >/dev/null 2>&1 || true; elif [ -f "$HOME/.bashrc" ]; then . "$HOME/.bashrc" >/dev/null 2>&1 || true; fi',
'if [ -f "$HOME/.zprofile" ]; then . "$HOME/.zprofile" >/dev/null 2>&1 || true; fi',
'export NVM_DIR="${NVM_DIR:-$HOME/.nvm}"',
'[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh" >/dev/null 2>&1 || true',
`cd ${shellQuote(input.spec.remoteCwd)}`,
envArgs.length > 0
? `exec env ${envArgs.join(" ")} ${remoteCommandParts}`
packages/adapter-utils/src/types.ts
+8
View File
@@ -175,6 +175,14 @@ export interface AdapterExecutionContext {
onRuntimeProgress?: RuntimeStatusSink;
onSpawn?: (meta: { pid: number; processGroupId: number | null; startedAt: string }) => Promise<void>;
authToken?: string;
/**
* The injected OpenTelemetry startup trace context (tracer + root
* parent-context helper). The server passes the real, endpoint-gated
* implementation; when absent, the ACPX engine uses a no-op, so the whole
* span path stays inert. The type is an inline import so this module keeps no
* top-level dependency on the timing helper.
*/
startupTraceContext?: import("./acpx-engine/startup-timing.js").StartupTraceContext;
}
export interface AdapterModel {
packages/adapters/claude-local/package.json
+2 -2
View File
@@ -53,12 +53,12 @@
"probe:quota:raw": "pnpm exec tsx src/cli/quota-probe.ts --json --raw-cli"
},
"dependencies": {
"@agentclientprotocol/claude-agent-acp": "^0.59.0",
"@agentclientprotocol/claude-agent-acp": "^0.63.0",
"@paperclipai/adapter-utils": "workspace:*",
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"typescript": "^5.7.3"
}
}
packages/adapters/claude-local/src/index.ts
+2
View File
@@ -7,8 +7,10 @@ export const SANDBOX_INSTALL_COMMAND = "npm install -g @anthropic-ai/claude-code
export const models = [
{ id: "claude-opus-4-8", label: "Claude Opus 4.8" },
{ id: "claude-sonnet-5", label: "Claude Sonnet 5" },
{ id: "claude-fable-5", label: "Claude Fable 5" },
{ id: "claude-mythos-5", label: "Claude Mythos 5" },
{ id: "claude-opus-5", label: "Claude Opus 5" },
{ id: "claude-opus-4-7", label: "Claude Opus 4.7" },
{ id: "claude-opus-4-6", label: "Claude Opus 4.6" },
{ id: "claude-sonnet-4-6", label: "Claude Sonnet 4.6" },
packages/adapters/claude-local/src/server/execute.acp-fallback.test.ts
+27 -1
View File
@@ -1,4 +1,4 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
const {
ensureAdapterExecutionTargetCommandResolvable,
@@ -89,6 +89,10 @@ describe("claude_local ACP startup fallback", () => {
vi.clearAllMocks();
});
afterEach(() => {
vi.unstubAllEnvs();
});
it("falls back to Claude CLI when auto-selected ACP fails before execution starts", async () => {
const ctx = buildContext();
@@ -107,6 +111,28 @@ describe("claude_local ACP startup fallback", () => {
);
});
it("trusts the Paperclip API URL when network access is allowlisted", async () => {
const paperclipApiUrl = "http://127.0.0.1:4310";
vi.stubEnv("PAPERCLIP_RUNTIME_API_URL", paperclipApiUrl);
const ctx = buildContext({ networkScope: "allowlist" });
await execute(ctx as never);
expect(runAdapterExecutionTargetProcess).toHaveBeenCalledTimes(1);
expect(runAdapterExecutionTargetProcess).toHaveBeenCalledWith(
expect.any(String),
null,
expect.any(String),
expect.any(Array),
expect.objectContaining({
localProcessSandbox: expect.objectContaining({
networkScope: "allowlist",
networkTrustedUrls: [paperclipApiUrl],
}),
}),
);
});
it("keeps explicit ACP strict when startup fails", async () => {
const ctx = buildContext({ engine: "acp" });
packages/adapters/claude-local/src/server/execute.ts
+4
View File
@@ -543,6 +543,10 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
homeDir: filesystemScope ? path.dirname(sharedClaudeConfigDir) : null,
networkScope,
networkAllowlist: parseLocalProcessNetworkAllowlist(config.networkAllowlist),
networkTrustedUrls: [
env.PAPERCLIP_API_URL,
...runtimeMcpServers.map((server) => server.url),
].filter((value): value is string => typeof value === "string" && value.length > 0),
command: asString(config.filesystemSandboxCommand, "bwrap"),
}
: null;
packages/adapters/codex-local/package.json
+2 -2
View File
@@ -52,12 +52,12 @@
"probe:quota": "pnpm exec tsx src/cli/quota-probe.ts --json"
},
"dependencies": {
"@agentclientprotocol/codex-acp": "^1.1.0",
"@agentclientprotocol/codex-acp": "^1.1.4",
"@paperclipai/adapter-utils": "workspace:*",
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"typescript": "^5.7.3"
}
}
packages/adapters/codex-local/src/index.test.ts
+23 -4
View File
@@ -1,18 +1,37 @@
import { describe, expect, it } from "vitest";
import { DEFAULT_CODEX_LOCAL_MODEL, isCodexLocalFastModeSupported, models } from "./index.js";
import {
DEFAULT_CODEX_LOCAL_MODEL,
isCodexLocalFastModeSupported,
models,
normalizeCodexModel,
} from "./index.js";
describe("codex local adapter metadata", () => {
it("advertises current GPT-5.6 Codex-capable OpenAI models by default", () => {
const modelIds = models.map((model) => model.id);
expect(DEFAULT_CODEX_LOCAL_MODEL).toBe("gpt-5.6");
expect(modelIds.slice(0, 4)).toEqual([
"gpt-5.6",
// Default to the concrete gpt-5.6-sol slug — Codex ships no metadata for the bare gpt-5.6
// alias, so it must not be advertised or used as the default (it triggers a fallback warning).
expect(DEFAULT_CODEX_LOCAL_MODEL).toBe("gpt-5.6-sol");
expect(modelIds.slice(0, 3)).toEqual([
"gpt-5.6-sol",
"gpt-5.6-terra",
"gpt-5.6-luna",
]);
expect(modelIds).not.toContain("gpt-5.6");
expect(isCodexLocalFastModeSupported(DEFAULT_CODEX_LOCAL_MODEL)).toBe(true);
expect(modelIds).not.toContain("gpt-5.3-codex");
expect(modelIds).not.toContain("gpt-5.3-codex-spark");
});
it("normalizes the legacy bare gpt-5.6 alias to the concrete gpt-5.6-sol slug", () => {
expect(normalizeCodexModel("gpt-5.6")).toBe("gpt-5.6-sol");
expect(normalizeCodexModel(" gpt-5.6 ")).toBe("gpt-5.6-sol");
// Concrete slugs and unknown/manual model IDs pass through untouched.
expect(normalizeCodexModel("gpt-5.6-sol")).toBe("gpt-5.6-sol");
expect(normalizeCodexModel("gpt-5.5")).toBe("gpt-5.5");
expect(normalizeCodexModel("future-model")).toBe("future-model");
expect(normalizeCodexModel("")).toBe("");
expect(normalizeCodexModel(null)).toBe("");
});
});
packages/adapters/codex-local/src/index.ts
+29 -12
View File
@@ -5,14 +5,36 @@ export const label = "Codex";
export const SANDBOX_INSTALL_COMMAND = "npm install -g @openai/codex";
export const DEFAULT_CODEX_LOCAL_MODEL = "gpt-5.6";
// Use the concrete `gpt-5.6-sol` slug (Codex's own default for the 5.6 family) rather than the
// bare `gpt-5.6` alias: OpenAI ships no model metadata for the bare slug, so passing it makes the
// Codex CLI warn ("Model metadata for `gpt-5.6` not found") and fall back to generic context limits.
export const DEFAULT_CODEX_LOCAL_MODEL = "gpt-5.6-sol";
export const DEFAULT_CODEX_LOCAL_BYPASS_APPROVALS_AND_SANDBOX = true;
export const CODEX_LOCAL_FAST_MODE_SUPPORTED_MODELS = ["gpt-5.6", "gpt-5.5", "gpt-5.4"] as const;
export const CODEX_LOCAL_FAST_MODE_SUPPORTED_MODELS = [
"gpt-5.6-sol",
"gpt-5.6-terra",
"gpt-5.6-luna",
"gpt-5.5",
"gpt-5.4",
] as const;
function normalizeModelId(model: string | null | undefined): string {
return typeof model === "string" ? model.trim() : "";
}
// Legacy model aliases with no OpenAI-published metadata are rewritten to the concrete slug the
// Codex CLI actually knows. Without this, agents still configured with the bare `gpt-5.6` alias
// (the old default) keep triggering "Model metadata for `gpt-5.6` not found" warnings and Codex
// falls back to generic context-window limits, even on a Codex build that ships the 5.6 family.
const CODEX_LOCAL_MODEL_ALIASES: Readonly<Record<string, string>> = {
"gpt-5.6": "gpt-5.6-sol",
};
export function normalizeCodexModel(model: string | null | undefined): string {
const normalizedModel = normalizeModelId(model);
return CODEX_LOCAL_MODEL_ALIASES[normalizedModel] ?? normalizedModel;
}
export function isCodexLocalKnownModel(model: string | null | undefined): boolean {
const normalizedModel = normalizeModelId(model);
if (!normalizedModel) return false;
@@ -37,13 +59,12 @@ export function isCodexLocalFastModeSupported(model: string | null | undefined):
}
export const models = [
// DEFAULT_CODEX_LOCAL_MODEL is gpt-5.6-sol, so it doubles as the first (default) 5.6 entry.
{ id: DEFAULT_CODEX_LOCAL_MODEL, label: DEFAULT_CODEX_LOCAL_MODEL },
{ id: "gpt-5.6-sol", label: "gpt-5.6-sol" },
{ id: "gpt-5.6-terra", label: "gpt-5.6-terra" },
{ id: "gpt-5.6-luna", label: "gpt-5.6-luna" },
{ id: "gpt-5.4", label: "gpt-5.4" },
{ id: "gpt-5.4-mini", label: "gpt-5.4-mini" },
{ id: "gpt-5.3-codex-spark", label: "gpt-5.3-codex-spark" },
{ id: "gpt-5", label: "gpt-5" },
{ id: "o3", label: "o3" },
{ id: "o4-mini", label: "o4-mini" },
@@ -57,12 +78,8 @@ export const modelProfiles: AdapterModelProfileDefinition[] = [
{
key: "cheap",
label: "Cheap",
description: "Use the lowest-cost known Codex local model lane without changing the primary model.",
adapterConfig: {
model: "gpt-5.3-codex-spark",
// Spark is the cheap lane by model price; high effort keeps Codex coding behavior usable for delegated work.
modelReasoningEffort: "high",
},
description: "Use an explicitly configured lower-cost Codex model without changing the primary model.",
adapterConfig: {},
source: "adapter_default",
},
];
@@ -79,7 +96,7 @@ Core fields:
- modelReasoningEffort (string, optional): reasoning effort override (minimal|low|medium|high|xhigh) passed via -c model_reasoning_effort=...
- promptTemplate (string, optional): run prompt template
- search (boolean, optional): run codex with --search
- fastMode (boolean, optional): enable Codex Fast mode; supported on GPT-5.6, GPT-5.5, GPT-5.4 and passed through for manual model IDs
- fastMode (boolean, optional): enable Codex Fast mode; supported on GPT-5.6 (sol/terra/luna), GPT-5.5, GPT-5.4 and passed through for manual model IDs
- dangerouslyBypassApprovalsAndSandbox (boolean, optional): run with bypass flag
- command (string, optional): defaults to "codex"
- extraArgs (string[], optional): additional CLI args
@@ -110,7 +127,7 @@ Notes:
- Paperclip injects desired local skills into the effective CODEX_HOME/skills/ directory at execution time so Codex can discover "$paperclip" and related skills without polluting the project working directory. For new and updated agents, Paperclip assigns an isolated managed home at ~/.paperclip/instances/<id>/companies/<companyId>/agents/<agentId>/codex-home/skills/; when CODEX_HOME is explicitly overridden in adapter config, that override is used instead.
- New and updated codex_local agents persist an empty OPENAI_API_KEY override by default so a host-level OPENAI_API_KEY cannot leak into Codex runs through process inheritance. Explicit CODEX_HOME overrides must not point at the shared company codex-home, $CODEX_HOME, or ~/.codex.
- Some model/tool combinations reject certain effort levels (for example minimal with web search enabled).
- Fast mode is supported on GPT-5.6, GPT-5.5, GPT-5.4 and manual model IDs. When enabled for those models, Paperclip applies \`service_tier="fast"\` and \`features.fast_mode=true\`.
- Fast mode is supported on GPT-5.6 (sol/terra/luna), GPT-5.5, GPT-5.4 and manual model IDs. When enabled for those models, Paperclip applies \`service_tier="fast"\` and \`features.fast_mode=true\`.
- When Paperclip realizes a workspace/runtime for a run, it injects PAPERCLIP_WORKSPACE_* and PAPERCLIP_RUNTIME_* env vars for agent-side tooling.
- Codex ACP is the preferred auto lane when Node >=22.13.0 and the Codex ACP server are available. It reuses shared ACP prompt/runtime guidance, selected skill materialization into CODEX_HOME/skills, model/reasoning/fast-mode session config, and existing quota-window reporting. Auto selection falls back to CLI when ACP prerequisites are unavailable; explicit engine="acp" fails loudly.
`;
packages/adapters/codex-local/src/server/acp.test.ts
+6
View File
@@ -469,6 +469,12 @@ describe("codex_local ACP lane", () => {
});
});
it("normalizes the legacy bare gpt-5.6 alias to gpt-5.6-sol", () => {
expect(buildCodexAcpConfig({ engine: "acp", model: "gpt-5.6" })).toMatchObject({
model: "gpt-5.6-sol",
});
});
it("checks the Node version required by the ACPX runtime", () => {
setNodeVersion("v22.12.0");
expect(nodeVersionMeetsCodexAcpMinimum()).toBe(false);
packages/adapters/codex-local/src/server/acp.ts
+7
View File
@@ -35,6 +35,7 @@ import {
asString,
parseObject,
} from "@paperclipai/adapter-utils/server-utils";
import { normalizeCodexModel } from "../index.js";
import { classifyCodexAuthRefreshFailure } from "./parse.js";
import { copyBackCodexAuth } from "./codex-auth-copyback.js";
import { buildCodexAuthInboundProvision } from "./codex-auth-merge-scripts.js";
@@ -145,6 +146,11 @@ export function buildCodexAcpConfig(config: Record<string, unknown>): Record<str
config.warmHandleIdleMs ??
config.acpWarmHandleIdleMs ??
DEFAULT_ACP_ENGINE_WARM_HANDLE_IDLE_MS;
// Rewrite legacy model aliases (e.g. bare gpt-5.6) to the concrete slug Codex has metadata for,
// so the ACP session config matches the CLI lane and avoids the fallback-metadata warning.
const normalizedModel = normalizeCodexModel(
typeof config.model === "string" ? config.model : "",
);
return {
...config,
@@ -153,6 +159,7 @@ export function buildCodexAcpConfig(config: Record<string, unknown>): Record<str
permissionMode,
nonInteractivePermissions,
warmHandleIdleMs,
...(normalizedModel ? { model: normalizedModel } : {}),
...(agentCommand ? { agentCommand } : {}),
...(stateDir ? { stateDir } : {}),
};
packages/adapters/codex-local/src/server/codex-args.test.ts
+17 -4
View File
@@ -2,6 +2,19 @@ import { describe, expect, it } from "vitest";
import { buildCodexExecArgs } from "./codex-args.js";
describe("buildCodexExecArgs", () => {
it("rewrites the legacy bare gpt-5.6 alias to gpt-5.6-sol and applies fast mode", () => {
const result = buildCodexExecArgs({
model: "gpt-5.6",
fastMode: true,
});
expect(result.model).toBe("gpt-5.6-sol");
expect(result.args).toContain("gpt-5.6-sol");
expect(result.args).not.toContain("gpt-5.6");
expect(result.fastModeApplied).toBe(true);
expect(result.fastModeIgnoredReason).toBeNull();
});
it("enables Codex fast mode overrides for GPT-5.4", () => {
const result = buildCodexExecArgs({
model: "gpt-5.4",
@@ -89,22 +102,22 @@ describe("buildCodexExecArgs", () => {
]);
});
it("ignores fast mode for unsupported models", () => {
it("ignores fast mode for known unsupported models", () => {
const result = buildCodexExecArgs({
model: "gpt-5.3-codex-spark",
model: "gpt-5",
fastMode: true,
});
expect(result.fastModeRequested).toBe(true);
expect(result.fastModeApplied).toBe(false);
expect(result.fastModeIgnoredReason).toContain(
"currently only supported on gpt-5.6, gpt-5.5, gpt-5.4 or manually configured model IDs",
"currently only supported on gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5, gpt-5.4 or manually configured model IDs",
);
expect(result.args).toEqual([
"exec",
"--json",
"--model",
"gpt-5.3-codex-spark",
"gpt-5",
"-",
]);
});
packages/adapters/codex-local/src/server/codex-args.ts
+2 -1
View File
@@ -2,6 +2,7 @@ import { asBoolean, asString, asStringArray } from "@paperclipai/adapter-utils/s
import {
CODEX_LOCAL_FAST_MODE_SUPPORTED_MODELS,
isCodexLocalFastModeSupported,
normalizeCodexModel,
} from "../index.js";
export type BuildCodexExecArgsResult = {
@@ -36,7 +37,7 @@ export function buildCodexExecArgs(
} = {},
): BuildCodexExecArgsResult {
const record = asRecord(config);
const model = asString(record.model, "").trim();
const model = normalizeCodexModel(asString(record.model, ""));
const modelReasoningEffort = asString(
record.modelReasoningEffort,
asString(record.reasoningEffort, ""),
packages/adapters/codex-local/src/server/codex-auth-merge-scripts.ts
+13 -6
View File
@@ -22,12 +22,19 @@ const CODEX_AUTH_MERGE_DECISION_SCRIPT_BYTES = readFileSync(
/**
* Builds the inbound (host→sandbox) provisioning contribution for the Codex
* managed-home asset: stage the two merge scripts into the runtime root and run
* the merge-extract script instead of a plain `tar -xf`, so a sandbox that
* already carries a Codex `auth.json` keeps whichever credential is newer.
* managed-home asset as **files + an ordered post-upload merge command**: the two
* merge scripts ride the sync operation's `files` (staged into the runtime root
* alongside the uploaded home tar via native `uploadFiles`), and the merge-extract
* script runs as the operation's ordered **post-upload command** instead of a
* plain `tar -xf`, so a sandbox that already carries a Codex `auth.json` keeps
* whichever credential is newer (newer-`auth.json`-wins, same-identity, atomic
* `0600` install — all inside the opaque script, unchanged).
*
* This is behaviour-identical to the extraction the sandbox core previously
* hardcoded for `adapterKey === "codex" && assetKey === "home"`.
* The command string handed to the provider is opaque and fully shell-quoted from
* already-confined paths (Security Conditions C1/C3): it invokes only the staged
* script by path — no `auth.json` bytes, token fields, or workspace content are
* interpolated into the shell (C5). This is behaviour-identical to the extraction
* the sandbox core previously drove through the custom-provision tar path.
*/
export function buildCodexAuthInboundProvision(): SandboxManagedRuntimeAssetProvision {
return {
@@ -35,7 +42,7 @@ export function buildCodexAuthInboundProvision(): SandboxManagedRuntimeAssetProv
{ name: CODEX_AUTH_MERGE_EXTRACT_SCRIPT_NAME, contents: CODEX_AUTH_MERGE_EXTRACT_SCRIPT_BYTES },
{ name: CODEX_AUTH_MERGE_DECISION_SCRIPT_NAME, contents: CODEX_AUTH_MERGE_DECISION_SCRIPT_BYTES },
],
extractCommand: ({ assetTarPath, assetDir, runtimeRootDir }) =>
postUploadCommand: ({ assetTarPath, assetDir, runtimeRootDir }) =>
`sh ${shellQuote(path.posix.join(runtimeRootDir, CODEX_AUTH_MERGE_EXTRACT_SCRIPT_NAME))} ` +
`${shellQuote(assetDir)} ${shellQuote(assetTarPath)}`,
};
packages/adapters/codex-local/src/server/codex-auth-merge.test.ts
+143
View File
@@ -9,6 +9,7 @@ import { afterEach, describe, expect, it } from "vitest";
import {
prepareSandboxManagedRuntime,
type SandboxManagedRuntimeClient,
type SandboxSyncOperation,
} from "@paperclipai/adapter-utils/sandbox-managed-runtime";
import { buildCodexAuthInboundProvision } from "./codex-auth-merge-scripts.js";
@@ -93,6 +94,27 @@ describe("codex home auth merge on sandbox asset extract", () => {
outputs.push(result.stdout, result.stderr);
},
};
// Non-native base64-tar fallback `syncIn`: place each file mapping via
// `writeFile`, then run the operation's ordered `postUploadCommands` — the
// same seam the command-managed client provides in production. The Codex home
// asset uploads its tar + the two merge scripts, then runs the auth-merge
// command as the operation's post-upload command.
client.syncIn = async (operations) => {
for (const operation of operations) {
for (const mapping of operation.files) {
const bytes = await readFile(mapping.sourcePath);
await mkdir(path.dirname(mapping.targetPath), { recursive: true });
await client.writeFile(mapping.targetPath, bytes.buffer.slice(
bytes.byteOffset,
bytes.byteOffset + bytes.byteLength,
) as ArrayBuffer);
}
for (const command of operation.postUploadCommands ?? []) {
await client.run(command.command, { timeoutMs: 30_000 });
}
}
return { operations: [] };
};
await prepareSandboxManagedRuntime({
spec: {
@@ -352,6 +374,127 @@ describe("codex home auth merge on sandbox asset extract", () => {
expect(result.commandText, entry.name).not.toContain("SENTINEL");
}
});
it("routes the Codex home asset through a single native syncIn operation whose post-command is the auth-merge (#4, C5/C6)", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-codex-native-route-"));
cleanupDirs.push(rootDir);
const localWorkspaceDir = path.join(rootDir, "local-workspace");
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
const localHomeDir = path.join(rootDir, "local-codex-home");
const remoteHomeDir = path.join(remoteWorkspaceDir, ".paperclip-runtime", "codex", "home");
await mkdir(localWorkspaceDir, { recursive: true });
await mkdir(localHomeDir, { recursive: true });
await mkdir(remoteHomeDir, { recursive: true });
await writeFile(path.join(localWorkspaceDir, "README.md"), "workspace\n", "utf8");
// Host credential is strictly newer → wins the merge (same identity).
const hostAuth = subscriptionAuth({
accountId: "acct-native",
lastRefresh: "2026-07-10T02:00:00Z",
marker: "host-newer-SENTINEL",
});
const sandboxAuth = subscriptionAuth({
accountId: "acct-native",
lastRefresh: "2026-07-10T01:00:00Z",
marker: "sandbox-older-SENTINEL",
});
await writeFile(path.join(localHomeDir, "auth.json"), hostAuth, { mode: 0o600 });
await writeFile(path.join(localHomeDir, "config.toml"), "model = \"gpt\"\n", "utf8");
await writeFile(path.join(remoteHomeDir, "auth.json"), sandboxAuth, { mode: 0o600 });
// A native runner: the orchestrator must make NO direct writeFile/run — every
// byte (incl. auth.json) rides `syncIn` (native uploadFiles), and the merge
// runs as the operation's ordered post-upload command.
const directWrites: string[] = [];
const directRuns: string[] = [];
const captured: SandboxSyncOperation[] = [];
const client: SandboxManagedRuntimeClient = {
makeDir: async (remotePath) => {
await mkdir(remotePath, { recursive: true });
},
writeFile: async (remotePath, bytes) => {
directWrites.push(remotePath);
await mkdir(path.dirname(remotePath), { recursive: true });
await writeFile(remotePath, Buffer.from(bytes));
},
readFile: async (remotePath) => await readFile(remotePath),
listFiles: async () => [],
remove: async (remotePath) => {
await rm(remotePath, { recursive: true, force: true });
},
run: async (command) => {
directRuns.push(command);
await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 });
},
};
client.syncIn = async (operations) => {
for (const operation of operations) {
captured.push(operation);
for (const mapping of operation.files) {
const bytes = await readFile(mapping.sourcePath);
await mkdir(path.dirname(mapping.targetPath), { recursive: true });
await writeFile(mapping.targetPath, bytes);
if (mapping.mode != null) await lstat(mapping.targetPath);
}
for (const command of operation.postUploadCommands ?? []) {
await execFile("sh", ["-c", command.command], { maxBuffer: 32 * 1024 * 1024 });
}
}
return { operations: [] };
};
await prepareSandboxManagedRuntime({
spec: {
transport: "sandbox",
provider: "test",
sandboxId: "sandbox-1",
remoteCwd: remoteWorkspaceDir,
timeoutMs: 30_000,
apiKey: null,
},
adapterKey: "codex",
client,
workspaceLocalDir: localWorkspaceDir,
assets: [{
key: "home",
localDir: localHomeDir,
followSymlinks: true,
provision: buildCodexAuthInboundProvision(),
}],
});
// 0 direct exec/writeFile — pure native delegation.
expect(directWrites).toEqual([]);
expect(directRuns).toEqual([]);
// One operation carries the home asset: the home tar + the two merge scripts
// as `files` mappings, and the auth-merge as the ordered post-upload command.
const homeOp = captured.find((op) =>
op.files.some((mapping) => mapping.targetPath.endsWith("home-upload.tar")),
);
expect(homeOp).toBeDefined();
const targets = homeOp!.files.map((mapping) => path.posix.basename(mapping.targetPath)).sort();
expect(targets).toEqual([
"codex-auth-merge-decision.cjs",
"codex-auth-merge-extract.sh",
"home-upload.tar",
]);
expect(homeOp!.files.every((mapping) => mapping.kind === "file")).toBe(true);
expect(homeOp!.postUploadCommands).toHaveLength(1);
// The post-command is the auth-merge script, NOT a plain `tar -xf` (C6).
const mergeCommand = homeOp!.postUploadCommands![0].command;
expect(mergeCommand).toContain("codex-auth-merge-extract.sh");
expect(mergeCommand).not.toMatch(/^\s*tar -xf/);
// C5: no token/credential material leaks into the operation metadata.
const opJson = JSON.stringify(captured);
expect(opJson).not.toContain("SENTINEL");
expect(opJson).not.toContain("refresh-token");
// C6: newer host credential won, installed atomically at mode 0600.
const finalAuthPath = path.join(remoteHomeDir, "auth.json");
expect(await readFile(finalAuthPath, "utf8")).toBe(hostAuth);
expect((await lstat(finalAuthPath)).mode & 0o777).toBe(0o600);
});
});
// The extract shell script consumes the decision predicate as a child process
packages/adapters/cursor-cloud/package.json
+2 -2
View File
@@ -50,12 +50,12 @@
"typecheck": "tsc --noEmit"
},
"dependencies": {
"@cursor/sdk": "^1.0.19",
"@cursor/sdk": "^1.0.24",
"@paperclipai/adapter-utils": "workspace:*",
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"typescript": "^5.7.3"
}
}
packages/adapters/cursor-local/package.json
+1 -1
View File
@@ -55,7 +55,7 @@
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"typescript": "^5.7.3"
}
}
packages/adapters/gemini-local/package.json
+1 -1
View File
@@ -55,7 +55,7 @@
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"typescript": "^5.7.3"
}
}
packages/adapters/gemini-local/src/server/index.ts
+1
View File
@@ -9,6 +9,7 @@ export {
isGeminiTransientNetworkError,
describeGeminiFailure,
detectGeminiAuthRequired,
detectGeminiQuotaExhausted,
isGeminiTurnLimitResult,
} from "./parse.js";
import type { AdapterSessionCodec } from "@paperclipai/adapter-utils";
packages/adapters/grok-local/package.json
+1 -1
View File
@@ -54,7 +54,7 @@
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"typescript": "^5.7.3"
}
}
packages/adapters/hermes-gateway/package.json
+1 -1
View File
@@ -71,7 +71,7 @@
"@paperclipai/hermes-paperclip-adapter": "workspace:*"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"typescript": "^5.7.3",
"vitest": "^4.1.10"
},
packages/adapters/hermes/package.json
+1 -1
View File
@@ -103,7 +103,7 @@
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"typescript": "^5.7.3",
"vitest": "^4.1.10"
},
packages/adapters/openclaw-gateway/package.json
+1 -1
View File
@@ -55,7 +55,7 @@
"ws": "^8.21.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"@types/ws": "^8.18.1",
"typescript": "^5.7.3"
}
packages/adapters/opencode-local/package.json
+1 -1
View File
@@ -55,7 +55,7 @@
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"typescript": "^5.7.3"
}
}
packages/adapters/pi-local/package.json
+1 -1
View File
@@ -54,7 +54,7 @@
"picocolors": "^1.1.1"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"typescript": "^5.7.3"
}
}
packages/db/package.json
+1 -1
View File
@@ -53,7 +53,7 @@
"postgres": "^3.4.9"
},
"devDependencies": {
"@types/node": "^22.19.21",
"@types/node": "^22.20.1",
"drizzle-kit": "^0.31.10",
"tsx": "^4.23.1",
"typescript": "^5.7.3",
packages/db/src/built-in-agent-unique-marker-migration.test.ts
+119
View File
@@ -0,0 +1,119 @@
import { createHash, randomUUID } from "node:crypto";
import fs from "node:fs";
import { afterEach, describe, expect, it } from "vitest";
import postgres from "postgres";
import { applyPendingMigrations } from "./client.js";
import {
getEmbeddedPostgresTestSupport,
startEmbeddedPostgresTestDatabase,
} from "./test-embedded-postgres.js";
const MIGRATION_FILE = "0195_built_in_agent_unique_marker.sql";
const UNIQUE_INDEX = "agents_company_built_in_agent_key_unique_idx";
const cleanups: Array<() => Promise<void>> = [];
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
async function migrationHash() {
const content = await fs.promises.readFile(new URL(`./migrations/${MIGRATION_FILE}`, import.meta.url), "utf8");
return createHash("sha256").update(content).digest("hex");
}
function markerObj(key: string) {
// Return a plain object (not a JSON string) so `sql.json` stores it as a jsonb
// object; a JSON string param + `::jsonb` would double-encode into a scalar.
return { paperclipBuiltInAgent: { key, featureKeys: [key] } };
}
describeEmbeddedPostgres("built-in agent unique marker migration", () => {
afterEach(async () => {
await Promise.all(cleanups.splice(0).map((cleanup) => cleanup()));
});
it("resolves pre-existing duplicates and enforces uniqueness going forward", async () => {
const database = await startEmbeddedPostgresTestDatabase("paperclip-built-in-unique-marker-");
cleanups.push(database.cleanup);
const sql = postgres(database.connectionString, { max: 1 });
cleanups.push(async () => sql.end());
// Rewind the migration so we can seed the pre-index (duplicate) state.
await sql`DELETE FROM "drizzle"."__drizzle_migrations" WHERE "hash" = ${await migrationHash()}`;
await sql`DROP INDEX IF EXISTS ${sql(UNIQUE_INDEX)}`;
const affectedCompanyId = randomUUID();
const cleanCompanyId = randomUUID();
const olderId = randomUUID();
const newerId = randomUUID();
const terminatedDupeId = randomUUID();
const cleanSummarizerId = randomUUID();
const approvalId = randomUUID();
const apiKeyId = randomUUID();
await sql`
INSERT INTO "companies" ("id", "name", "issue_prefix")
VALUES
(${affectedCompanyId}, 'Affected', 'AFF'),
(${cleanCompanyId}, 'Clean', 'CLN')
`;
await sql`
INSERT INTO "agents" ("id", "company_id", "name", "status", "created_at", "metadata")
VALUES
(${olderId}, ${affectedCompanyId}, 'Summarizer One', 'idle', '2026-07-18T00:00:00.000Z', ${sql.json(markerObj("summarizer"))}),
(${newerId}, ${affectedCompanyId}, 'Summarizer Two', 'pending_approval', '2026-07-18T00:00:00.025Z', ${sql.json(markerObj("summarizer"))}),
(${terminatedDupeId}, ${affectedCompanyId}, 'Summarizer Old', 'terminated', '2026-07-17T00:00:00.000Z', ${sql.json(markerObj("summarizer"))}),
(${cleanSummarizerId}, ${cleanCompanyId}, 'Summarizer', 'idle', '2026-07-18T00:00:00.000Z', ${sql.json(markerObj("summarizer"))})
`;
await sql`
INSERT INTO "approvals" ("id", "company_id", "type", "status", "payload")
VALUES (${approvalId}, ${affectedCompanyId}, 'hire_agent', 'pending', ${sql.json({ agentId: newerId, sourceBuiltInAgentKey: "summarizer" })})
`;
await sql`
INSERT INTO "agent_api_keys" ("id", "agent_id", "company_id", "name", "key_hash")
VALUES (${apiKeyId}, ${newerId}, ${affectedCompanyId}, 'dupe-key', 'hash')
`;
// Re-run the migration: it should clean up the duplicates, then recreate the index.
await applyPendingMigrations(database.connectionString);
const agentRows = await sql<{ id: string; status: string }[]>`
SELECT "id", "status" FROM "agents"
WHERE "company_id" IN (${affectedCompanyId}, ${cleanCompanyId})
`;
const statusById = new Map(agentRows.map((row) => [row.id, row.status]));
// Oldest active row kept; newer duplicate terminated; the clean company untouched.
expect(statusById.get(olderId)).toBe("idle");
expect(statusById.get(newerId)).toBe("terminated");
expect(statusById.get(cleanSummarizerId)).toBe("idle");
const activeMarked = await sql<{ count: string }[]>`
SELECT count(*)::text AS count FROM "agents"
WHERE "company_id" = ${affectedCompanyId}
AND "status" <> 'terminated'
AND ("metadata" -> 'paperclipBuiltInAgent' ->> 'key') = 'summarizer'
`;
expect(activeMarked[0]!.count).toBe("1");
// The newer duplicate's orphan approval was cancelled and its API key revoked.
const [approval] = await sql<{ status: string }[]>`
SELECT "status" FROM "approvals" WHERE "id" = ${approvalId}
`;
expect(approval!.status).toBe("cancelled");
const [apiKey] = await sql<{ revoked_at: Date | null }[]>`
SELECT "revoked_at" FROM "agent_api_keys" WHERE "id" = ${apiKeyId}
`;
expect(apiKey!.revoked_at).not.toBeNull();
// The partial unique index exists again and now rejects a second active row.
const indexes = await sql<{ indexname: string }[]>`
SELECT "indexname" FROM "pg_indexes"
WHERE "tablename" = 'agents' AND "indexname" = ${UNIQUE_INDEX}
`;
expect(indexes).toHaveLength(1);
await expect(
sql`
INSERT INTO "agents" ("id", "company_id", "name", "status", "metadata")
VALUES (${randomUUID()}, ${affectedCompanyId}, 'Summarizer Dupe', 'idle', ${sql.json(markerObj("summarizer"))})
`,
).rejects.toMatchObject({ code: "23505", constraint_name: UNIQUE_INDEX });
}, 30_000);
});
packages/db/src/migrations/0193_document_memberships.sql
+15
View File
@@ -0,0 +1,15 @@
CREATE TABLE IF NOT EXISTS "document_memberships" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
"company_id" uuid NOT NULL REFERENCES "companies"("id") ON DELETE CASCADE,
"document_id" uuid NOT NULL REFERENCES "documents"("id") ON DELETE CASCADE,
"user_id" text NOT NULL,
"starred_at" timestamp with time zone,
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
);--> statement-breakpoint
-- paperclip:migration-safety-ignore large-create-index-not-concurrently: Drizzle migrations run transactionally, so CONCURRENTLY is unavailable; this index is created with the new empty membership table for user-scoped starred ordering.
CREATE INDEX IF NOT EXISTS "document_memberships_company_user_starred_idx"
ON "document_memberships" USING btree ("company_id", "user_id", "starred_at");--> statement-breakpoint
-- paperclip:migration-safety-ignore large-create-index-not-concurrently: Drizzle migrations run transactionally, so CONCURRENTLY is unavailable; this uniqueness index is created with the new empty membership table to make star upserts idempotent.
CREATE UNIQUE INDEX IF NOT EXISTS "document_memberships_company_user_document_uq"
ON "document_memberships" USING btree ("company_id", "user_id", "document_id");
packages/db/src/migrations/0194_company_skill_releases.sql
+4
View File
@@ -0,0 +1,4 @@
ALTER TABLE "company_skill_versions" ADD COLUMN IF NOT EXISTS "release_id" text;--> statement-breakpoint
ALTER TABLE "company_skill_versions" ADD COLUMN IF NOT EXISTS "release_name" text;--> statement-breakpoint
ALTER TABLE "company_skill_versions" ADD COLUMN IF NOT EXISTS "released_at" timestamp with time zone;--> statement-breakpoint
CREATE UNIQUE INDEX IF NOT EXISTS "company_skill_versions_skill_release_idx" ON "company_skill_versions" USING btree ("company_skill_id","release_id") WHERE "company_skill_versions"."release_id" is not null;
packages/db/src/migrations/0195_built_in_agent_unique_marker.sql
+53
View File
@@ -0,0 +1,53 @@
-- Enforce one active built-in agent per (company, marker key).
--
-- Provisioning used to check-then-insert with no database guard, so two
-- concurrent server processes could each see "no summarizer exists" and both
-- insert, leaving a company with duplicate built-in agents plus paired orphan
-- pending hire_agent approvals.
--
-- Step 1: resolve any pre-existing duplicates so the unique index below can be
-- created. Keep the oldest marked row per (company, key), terminate the newer
-- duplicates, cancel their orphan pending hire_agent approvals, and revoke any
-- API keys those terminated rows held.
WITH ranked AS (
SELECT
id,
row_number() OVER (
PARTITION BY company_id, (metadata -> 'paperclipBuiltInAgent' ->> 'key')
ORDER BY created_at ASC, id ASC
) AS rn
FROM agents
WHERE (metadata -> 'paperclipBuiltInAgent' ->> 'key') IS NOT NULL
AND status <> 'terminated'
),
duplicates AS (
SELECT id FROM ranked WHERE rn > 1
),
terminated AS (
UPDATE agents
SET status = 'terminated', updated_at = now()
WHERE id IN (SELECT id FROM duplicates)
RETURNING id
),
cancelled_approvals AS (
UPDATE approvals
SET status = 'cancelled', updated_at = now()
WHERE type = 'hire_agent'
AND status IN ('pending', 'revision_requested')
AND (payload ->> 'agentId') IN (SELECT id::text FROM terminated)
RETURNING id
)
UPDATE agent_api_keys
SET revoked_at = now()
WHERE revoked_at IS NULL
AND agent_id IN (SELECT id FROM terminated);
--> statement-breakpoint
-- Step 2: enforce uniqueness going forward. A partial unique index lets the
-- provisioning race lose cleanly (the losing INSERT raises 23505, which
-- provision()/ensure() catch and re-resolve to the winner's row). Terminated
-- rows are excluded so re-provisioning after a termination stays possible.
CREATE UNIQUE INDEX IF NOT EXISTS "agents_company_built_in_agent_key_unique_idx"
ON "agents" ("company_id", ((metadata -> 'paperclipBuiltInAgent' ->> 'key')))
WHERE (metadata -> 'paperclipBuiltInAgent' ->> 'key') IS NOT NULL
AND status <> 'terminated';
packages/db/src/migrations/0196_drop_cloud_upstream_tables.sql
+9
View File
@@ -0,0 +1,9 @@
-- Drop the cloud sync (cloud upstream) tables. The host-to-host transport
-- has been removed in favor of the Import/Export flow, and nothing reads or
-- writes these tables any more. The sender tables (connections, runs) came
-- from 0089; the receiver-side tables never shipped, so these two are all
-- that exist. The feature was experimental and flag-gated off by default,
-- so its run history is intentionally discarded.
DROP TABLE IF EXISTS "cloud_upstream_runs";
--> statement-breakpoint
DROP TABLE IF EXISTS "cloud_upstream_connections";
packages/db/src/migrations/meta/_journal.json
+28
View File
@@ -1338,6 +1338,34 @@
"when": 1784916886226,
"tag": "0192_task_watchdog_stop_snapshots",
"breakpoints": true
},
{
"idx": 193,
"version": "7",
"when": 1785170000000,
"tag": "0193_document_memberships",
"breakpoints": true
},
{
"idx": 194,
"version": "7",
"when": 1784920485226,
"tag": "0194_company_skill_releases",
"breakpoints": true
},
{
"idx": 195,
"version": "7",
"when": 1785170000001,
"tag": "0195_built_in_agent_unique_marker",
"breakpoints": true
},
{
"idx": 196,
"version": "7",
"when": 1785170001001,
"tag": "0196_drop_cloud_upstream_tables",
"breakpoints": true
}
]
}
packages/db/src/schema/cloud_upstreams.ts
-75
View File
@@ -1,75 +0,0 @@
import { boolean, index, integer, jsonb, pgTable, text, timestamp, uuid } from "drizzle-orm/pg-core";
import { companies } from "./companies.js";
export const cloudUpstreamConnections = pgTable(
"cloud_upstream_connections",
{
id: uuid("id").primaryKey().defaultRandom(),
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
remoteUrl: text("remote_url").notNull(),
sourceInstanceId: text("source_instance_id").notNull(),
sourceInstanceFingerprint: text("source_instance_fingerprint").notNull(),
sourcePublicKey: text("source_public_key").notNull(),
// Stored through the Cloud Upstream service as an encrypted credential envelope.
privateKeyPem: text("private_key_pem").notNull(),
tokenStatus: text("token_status").notNull(),
scopes: text("scopes").array().notNull().default([]),
authorizedGlobalUserId: text("authorized_global_user_id"),
// Stored through the Cloud Upstream service as an encrypted credential envelope.
accessToken: text("access_token"),
tokenId: text("token_id"),
tokenExpiresAt: timestamp("token_expires_at", { withTimezone: true }),
targetStackId: text("target_stack_id").notNull(),
targetStackSlug: text("target_stack_slug"),
targetStackDisplayName: text("target_stack_display_name"),
targetCompanyId: text("target_company_id").notNull(),
targetOrigin: text("target_origin").notNull(),
targetPrimaryHost: text("target_primary_host").notNull(),
targetProduct: text("target_product").notNull(),
targetSchemaMajor: integer("target_schema_major").notNull(),
targetMaxChunkBytes: integer("target_max_chunk_bytes").notNull(),
pendingState: text("pending_state"),
pendingCodeVerifier: text("pending_code_verifier"),
pendingRedirectUri: text("pending_redirect_uri"),
pendingTokenUrl: text("pending_token_url"),
lastRunId: uuid("last_run_id"),
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
},
(table) => [
index("cloud_upstream_connections_company_idx").on(table.companyId),
],
);
export const cloudUpstreamRuns = pgTable(
"cloud_upstream_runs",
{
id: uuid("id").primaryKey().defaultRandom(),
connectionId: uuid("connection_id").notNull().references(() => cloudUpstreamConnections.id, { onDelete: "cascade" }),
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
remoteRunId: text("remote_run_id"),
status: text("status").notNull(),
activeStep: text("active_step").notNull(),
progressPercent: integer("progress_percent").notNull().default(0),
dryRun: boolean("dry_run").notNull().default(false),
retryOfRunId: uuid("retry_of_run_id"),
summary: jsonb("summary").$type<import("@paperclipai/shared").CloudUpstreamSummaryCount[]>().notNull().default([]),
warnings: jsonb("warnings").$type<import("@paperclipai/shared").CloudUpstreamWarning[]>().notNull().default([]),
conflicts: jsonb("conflicts").$type<import("@paperclipai/shared").CloudUpstreamConflict[]>().notNull().default([]),
events: jsonb("events").$type<import("@paperclipai/shared").CloudUpstreamRunEvent[]>().notNull().default([]),
report: jsonb("report").$type<Record<string, unknown>>().notNull().default({}),
idempotencyKey: text("idempotency_key").notNull(),
manifestHash: text("manifest_hash").notNull(),
targetUrl: text("target_url"),
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
completedAt: timestamp("completed_at", { withTimezone: true }),
},
(table) => [
index("cloud_upstream_runs_company_created_idx").on(table.companyId, table.createdAt),
index("cloud_upstream_runs_connection_idx").on(table.connectionId),
],
);
packages/db/src/schema/company_skills.ts
+7
View File
@@ -9,6 +9,7 @@ import {
integer,
uniqueIndex,
} from "drizzle-orm/pg-core";
import { sql } from "drizzle-orm";
import type { CompanySkillFileInventoryEntry, CompanySkillSharingScope } from "@paperclipai/shared";
import { agents } from "./agents.js";
import { companies } from "./companies.js";
@@ -73,6 +74,9 @@ export const companySkillVersions = pgTable(
companySkillId: uuid("company_skill_id").notNull().references(() => companySkills.id, { onDelete: "cascade" }),
revisionNumber: integer("revision_number").notNull(),
label: text("label"),
releaseId: text("release_id"),
releaseName: text("release_name"),
releasedAt: timestamp("released_at", { withTimezone: true }),
fileInventory: jsonb("file_inventory").$type<CompanySkillVersionFileInventoryEntry[]>().notNull().default([]),
authorAgentId: uuid("author_agent_id").references(() => agents.id, { onDelete: "set null" }),
authorUserId: text("author_user_id"),
@@ -83,6 +87,9 @@ export const companySkillVersions = pgTable(
table.companySkillId,
table.revisionNumber,
),
companySkillReleaseUniqueIdx: uniqueIndex("company_skill_versions_skill_release_idx")
.on(table.companySkillId, table.releaseId)
.where(sql`${table.releaseId} is not null`),
companySkillCreatedIdx: index("company_skill_versions_company_skill_created_idx").on(
table.companyId,
table.companySkillId,
packages/db/src/schema/document_memberships.ts
+28
View File
@@ -0,0 +1,28 @@
import { pgTable, uuid, text, timestamp, uniqueIndex, index } from "drizzle-orm/pg-core";
import { companies } from "./companies.js";
import { documents } from "./documents.js";
export const documentMemberships = pgTable(
"document_memberships",
{
id: uuid("id").primaryKey().defaultRandom(),
companyId: uuid("company_id").notNull().references(() => companies.id, { onDelete: "cascade" }),
documentId: uuid("document_id").notNull().references(() => documents.id, { onDelete: "cascade" }),
userId: text("user_id").notNull(),
starredAt: timestamp("starred_at", { withTimezone: true }),
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
},
(table) => ({
companyUserStarredIdx: index("document_memberships_company_user_starred_idx").on(
table.companyId,
table.userId,
table.starredAt,
),
companyUserDocumentUq: uniqueIndex("document_memberships_company_user_document_uq").on(
table.companyId,
table.userId,
table.documentId,
),
}),
);
packages/db/src/schema/index.ts
+1 -1
View File
@@ -2,7 +2,6 @@ export { companies } from "./companies.js";
export { companyLogos } from "./company_logos.js";
export { authUsers, authSessions, authAccounts, authVerifications } from "./auth.js";
export { instanceSettings } from "./instance_settings.js";
export { cloudUpstreamConnections, cloudUpstreamRuns } from "./cloud_upstreams.js";
export { instanceUserRoles } from "./instance_user_roles.js";
export { userSidebarPreferences } from "./user_sidebar_preferences.js";
export { agents } from "./agents.js";
@@ -25,6 +24,7 @@ export { agentTaskSessions } from "./agent_task_sessions.js";
export { agentWakeupRequests } from "./agent_wakeup_requests.js";
export { projects } from "./projects.js";
export { projectMemberships } from "./project_memberships.js";
export { documentMemberships } from "./document_memberships.js";
export { projectWorkspaces } from "./project_workspaces.js";
export { executionWorkspaces } from "./execution_workspaces.js";
export { environments } from "./environments.js";
packages/db/src/table-size-estimates.ts
-2
View File
@@ -92,8 +92,6 @@ export const LOCAL_TABLE_ROW_COUNTS = [
{ table: "budget_incidents", localRows: 0 },
{ table: "budget_policies", localRows: 0 },
{ table: "cli_auth_challenges", localRows: 0 },
{ table: "cloud_upstream_connections", localRows: 0 },
{ table: "cloud_upstream_runs", localRows: 0 },
{ table: "company_logos", localRows: 0 },
{ table: "company_secret_provider_configs", localRows: 0 },
{ table: "company_skill_comments", localRows: 0 },
packages/db/src/test-embedded-postgres.test.ts
+113
View File
@@ -0,0 +1,113 @@
import fs from "node:fs";
import { afterEach, describe, expect, it } from "vitest";
import {
__embeddedPostgresStartMaxAttemptsForTests as MAX_ATTEMPTS,
__setEmbeddedPostgresCtorProviderForTests,
__startEmbeddedPostgresWithRetryForTests as startWithRetry,
} from "./test-embedded-postgres.js";
// A fake embedded-postgres constructor. It records every constructed instance so
// the test can assert the retry uses a fresh port and a fresh data directory each
// attempt. `start()` emits the same output the real cluster writes for a port
// conflict, then rejects with an empty message (the real rejection shape). The
// option type matches the real constructor so no type cast is needed.
type FakeOptions = {
databaseDir: string;
user: string;
password: string;
port: number;
persistent: boolean;
initdbFlags?: string[];
onLog?: (message: unknown) => void;
onError?: (message: unknown) => void;
};
const BIND_CONFLICT_LOG = 'could not bind IPv4 address "127.0.0.1": Address already in use';
function makeFakeCtor(failFirst: number) {
const constructed: FakeOptions[] = [];
let started = 0;
class FakeEmbeddedPostgres {
private readonly options: FakeOptions;
constructor(options: FakeOptions) {
this.options = options;
constructed.push(options);
}
async initialise(): Promise<void> {}
async start(): Promise<void> {
started += 1;
if (started <= failFirst) {
// Mirror the real failure: Postgres logs the reason, then `start()`
// rejects with an Error whose message is empty.
this.options.onLog?.(BIND_CONFLICT_LOG);
throw new Error();
}
}
async stop(): Promise<void> {}
}
return { ctor: FakeEmbeddedPostgres, constructed };
}
describe("startEmbeddedPostgresWithRetry", () => {
afterEach(() => {
__setEmbeddedPostgresCtorProviderForTests(null);
});
it("recovers from a transient port conflict and returns on a later attempt", async () => {
const { ctor, constructed } = makeFakeCtor(2);
__setEmbeddedPostgresCtorProviderForTests(async () => ctor);
const started = await startWithRetry("paperclip-retry-recover-");
// The first two attempts fail, the third succeeds.
expect(constructed).toHaveLength(3);
// Each attempt uses a fresh data directory. The two failed directories are
// removed. The returned directory still exists.
const dataDirs = constructed.map((options) => options.databaseDir);
expect(new Set(dataDirs).size).toBe(3);
expect(fs.existsSync(dataDirs[0])).toBe(false);
expect(fs.existsSync(dataDirs[1])).toBe(false);
expect(started.dataDir).toBe(dataDirs[2]);
expect(fs.existsSync(started.dataDir)).toBe(true);
// Each attempt allocates a port.
for (const options of constructed) {
expect(Number.isInteger(options.port)).toBe(true);
expect(options.port).toBeGreaterThan(0);
}
// Clean up the returned attempt.
await started.instance.stop();
fs.rmSync(started.dataDir, { recursive: true, force: true });
});
it("throws with the real Postgres output after the attempt bound", async () => {
const { ctor, constructed } = makeFakeCtor(Number.POSITIVE_INFINITY);
__setEmbeddedPostgresCtorProviderForTests(async () => ctor);
await expect(startWithRetry("paperclip-retry-fail-")).rejects.toThrow(/after \d+ attempts/);
// The retry stops at the bound and does not loop forever.
expect(constructed).toHaveLength(MAX_ATTEMPTS);
// Every failed attempt removes its data directory.
for (const options of constructed) {
expect(fs.existsSync(options.databaseDir)).toBe(false);
}
});
it("keeps the real failure reason instead of a generic fallback", async () => {
const { ctor } = makeFakeCtor(Number.POSITIVE_INFINITY);
__setEmbeddedPostgresCtorProviderForTests(async () => ctor);
const error = await startWithRetry("paperclip-retry-reason-").catch((caught: unknown) => caught);
expect(error).toBeInstanceOf(Error);
// The thrown message carries the captured Postgres output, not only the
// generic "embedded Postgres startup failed" text.
expect((error as Error).message).toContain("Address already in use");
});
});
packages/db/src/test-embedded-postgres.ts
+95 -37
View File
@@ -3,6 +3,10 @@ import net from "node:net";
import os from "node:os";
import path from "node:path";
import { applyPendingMigrations, ensurePostgresDatabase } from "./client.js";
import {
createEmbeddedPostgresLogBuffer,
formatEmbeddedPostgresError,
} from "./embedded-postgres-error.js";
import { prepareEmbeddedPostgresNativeRuntime } from "./embedded-postgres-native.js";
type EmbeddedPostgresInstance = {
@@ -47,12 +51,29 @@ function getReservedTestPorts(): Set<number> {
return new Set(configuredPorts.filter((port) => Number.isInteger(port) && port > 0 && port <= 65535));
}
async function getEmbeddedPostgresCtor(): Promise<EmbeddedPostgresCtor> {
type EmbeddedPostgresCtorProvider = () => Promise<EmbeddedPostgresCtor>;
async function loadEmbeddedPostgresCtor(): Promise<EmbeddedPostgresCtor> {
const mod = await import("embedded-postgres");
await prepareEmbeddedPostgresNativeRuntime();
return mod.default as EmbeddedPostgresCtor;
}
let embeddedPostgresCtorProvider: EmbeddedPostgresCtorProvider = loadEmbeddedPostgresCtor;
// Test seam. Replace the embedded-postgres constructor provider so a test can
// simulate a failed start without the native runtime. Pass `null` to restore
// the default provider. This module is test support only, so the seam is safe.
export function __setEmbeddedPostgresCtorProviderForTests(
provider: EmbeddedPostgresCtorProvider | null,
): void {
embeddedPostgresCtorProvider = provider ?? loadEmbeddedPostgresCtor;
}
async function getEmbeddedPostgresCtor(): Promise<EmbeddedPostgresCtor> {
return await embeddedPostgresCtorProvider();
}
async function getAvailablePort(): Promise<number> {
const reservedPorts = getReservedTestPorts();
for (let attempt = 0; attempt < 20; attempt += 1) {
@@ -88,6 +109,11 @@ async function createEmbeddedPostgresTestInstance(tempDirPrefix: string) {
const dataDir = fs.mkdtempSync(path.join(os.tmpdir(), tempDirPrefix));
const port = await getAvailablePort();
const EmbeddedPostgres = await getEmbeddedPostgresCtor();
// Postgres writes the true reason for a failed start to its output, for
// example `could not bind IPv4 address "127.0.0.1": Address already in use`.
// The `start()` rejection carries an empty message, so we capture the output
// in a bounded buffer and surface it in the thrown error.
const logBuffer = createEmbeddedPostgresLogBuffer();
const instance = new EmbeddedPostgres({
databaseDir: dataDir,
user: "paperclip",
@@ -95,11 +121,11 @@ async function createEmbeddedPostgresTestInstance(tempDirPrefix: string) {
port,
persistent: true,
initdbFlags: ["--encoding=UTF8", "--locale=C", "--lc-messages=C"],
onLog: () => {},
onError: () => {},
onLog: (message) => logBuffer.append(message),
onError: (message) => logBuffer.append(message),
});
return { dataDir, port, instance };
return { dataDir, port, instance, getRecentLogs: () => logBuffer.getRecentLogs() };
}
function cleanupEmbeddedPostgresTestDirs(dataDir: string) {
@@ -161,34 +187,72 @@ async function stopEmbeddedPostgresBounded(
}
}
function formatEmbeddedPostgresError(error: unknown): string {
if (error instanceof Error && error.message.length > 0) return error.message;
if (typeof error === "string" && error.length > 0) return error;
return "embedded Postgres startup failed";
// Upper bound on start attempts. `getAvailablePort` uses a check-then-use probe:
// it binds port 0, reads the assigned port, closes the probe, then Postgres binds
// that port. Under load another process can take the port in that window, so the
// bind fails with "Address already in use" and `start()` rejects. Each retry uses
// a fresh port and a fresh data directory, so a transient collision clears.
const EMBEDDED_POSTGRES_START_MAX_ATTEMPTS = 5;
// Start one embedded Postgres cluster with a bounded retry. Each attempt gets a
// fresh port and a fresh data directory. On a failed attempt we stop the cluster
// and remove its data directory before the next attempt. After the last attempt
// we throw with the real Postgres output so the failure is loud and diagnosable.
async function startEmbeddedPostgresWithRetry(tempDirPrefix: string): Promise<{
port: number;
dataDir: string;
instance: EmbeddedPostgresInstance;
}> {
let lastError = new Error("embedded Postgres startup failed");
for (let attempt = 1; attempt <= EMBEDDED_POSTGRES_START_MAX_ATTEMPTS; attempt += 1) {
const created = await createEmbeddedPostgresTestInstance(tempDirPrefix);
try {
await created.instance.initialise();
await created.instance.start();
return { port: created.port, dataDir: created.dataDir, instance: created.instance };
} catch (error) {
lastError = formatEmbeddedPostgresError(error, {
fallbackMessage: "embedded Postgres startup failed",
recentLogs: created.getRecentLogs(),
});
// Stop the failed cluster and remove its data directory. The next attempt
// allocates a fresh port and a fresh data directory.
await stopEmbeddedPostgresBounded(created.instance, () =>
cleanupEmbeddedPostgresTestDirs(created.dataDir),
);
}
}
throw new Error(
`Failed to start embedded PostgreSQL test database after ${EMBEDDED_POSTGRES_START_MAX_ATTEMPTS} attempts: ${lastError.message}`,
);
}
// Test-only accessors. Production callers use `startEmbeddedPostgresTestDatabase`
// or `getEmbeddedPostgresTestSupport`. A test drives the bounded retry directly
// so it does not need a real Postgres connection.
export const __startEmbeddedPostgresWithRetryForTests = startEmbeddedPostgresWithRetry;
export const __embeddedPostgresStartMaxAttemptsForTests = EMBEDDED_POSTGRES_START_MAX_ATTEMPTS;
async function probeEmbeddedPostgresSupport(): Promise<EmbeddedPostgresTestSupport> {
let dataDir: string | null = null;
let instance: EmbeddedPostgresInstance | null = null;
let started: { dataDir: string; instance: EmbeddedPostgresInstance } | null = null;
try {
const created = await createEmbeddedPostgresTestInstance(
"paperclip-embedded-postgres-probe-",
);
dataDir = created.dataDir;
instance = created.instance;
await instance.initialise();
await instance.start();
started = await startEmbeddedPostgresWithRetry("paperclip-embedded-postgres-probe-");
return { supported: true };
} catch (error) {
return {
supported: false,
reason: formatEmbeddedPostgresError(error),
reason: formatEmbeddedPostgresError(error, {
fallbackMessage: "embedded Postgres startup failed",
}).message,
};
} finally {
await stopEmbeddedPostgresBounded(instance, () => {
if (dataDir) cleanupEmbeddedPostgresTestDirs(dataDir);
});
if (started) {
const { dataDir, instance } = started;
await stopEmbeddedPostgresBounded(instance, () => cleanupEmbeddedPostgresTestDirs(dataDir));
}
}
}
@@ -202,17 +266,11 @@ export async function getEmbeddedPostgresTestSupport(): Promise<EmbeddedPostgres
export async function startEmbeddedPostgresTestDatabase(
tempDirPrefix: string,
): Promise<EmbeddedPostgresTestDatabase> {
let dataDir: string | null = null;
let instance: EmbeddedPostgresInstance | null = null;
// The bounded retry hardens the cluster start against the port race. It throws
// with the real Postgres output if every attempt fails.
const { port, dataDir, instance } = await startEmbeddedPostgresWithRetry(tempDirPrefix);
try {
const created = await createEmbeddedPostgresTestInstance(tempDirPrefix);
dataDir = created.dataDir;
instance = created.instance;
const { port } = created;
await instance.initialise();
await instance.start();
const adminConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${port}/postgres`;
await ensurePostgresDatabase(adminConnectionString, "paperclip");
const connectionString = `postgres://paperclip:paperclip@127.0.0.1:${port}/paperclip`;
@@ -221,17 +279,17 @@ export async function startEmbeddedPostgresTestDatabase(
return {
connectionString,
cleanup: async () => {
await stopEmbeddedPostgresBounded(instance, () => {
if (dataDir) cleanupEmbeddedPostgresTestDirs(dataDir);
});
await stopEmbeddedPostgresBounded(instance, () => cleanupEmbeddedPostgresTestDirs(dataDir));
},
};
} catch (error) {
await stopEmbeddedPostgresBounded(instance, () => {
if (dataDir) cleanupEmbeddedPostgresTestDirs(dataDir);
});
await stopEmbeddedPostgresBounded(instance, () => cleanupEmbeddedPostgresTestDirs(dataDir));
throw new Error(
`Failed to start embedded PostgreSQL test database: ${formatEmbeddedPostgresError(error)}`,
`Failed to start embedded PostgreSQL test database: ${
formatEmbeddedPostgresError(error, {
fallbackMessage: "embedded Postgres startup failed",
}).message
}`,
);
}
}
packages/google-sheets-mcp-server/package.json
+1 -1
View File
@@ -50,7 +50,7 @@
"zod": "^3.24.2"
},
"devDependencies": {
"@types/node": "^24.6.0",
"@types/node": "^22.20.1",
"typescript": "^5.7.3",
"vitest": "^4.1.10"
}
packages/kv-demo-mcp-server/package.json
+1 -1
View File
@@ -48,7 +48,7 @@
"zod": "^3.24.2"
},
"devDependencies": {
"@types/node": "^24.6.0",
"@types/node": "^22.20.1",
"typescript": "^5.7.3",
"vitest": "^4.1.10"
}
Loaded 100 of 417 files, more files were not shown because too many files have changed in this diff. Show more