mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-21 00:05:49 +02:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents run tasks in isolated execution workspaces that are provisioned as git worktrees by `scripts/provision-worktree.sh` > - The script runs the base workspace's CLI (`cli/src/index.ts` via the base `tsx` install) to seed each new worktree, and it only checked that those files exist > - pnpm links each package's `node_modules` into a hash-versioned virtual store; a lockfile change followed by a partial or filtered install prunes old hashed dirs without relinking every package, leaving dangling symlinks > - A CLI with dangling symlinks fails ESM resolution (`ERR_MODULE_NOT_FOUND`) at boot, so provisioning aborts with `setup_failed` — deterministically, on every retry, with no self-heal path > - This pull request makes provisioning health-check the CLI by actually booting it, repair the base install when the check fails, and degrade to the no-CLI fallback config instead of failing the run > - The benefit is that a class of permanent `setup_failed` loops becomes self-healing, and workspace provisioning survives a broken base CLI ## Linked Issues or Issue Description No public GitHub issue exists; the underlying bug is described here per `bug_report.yml`. Related PR: #10578 self-heals the sibling workspace-validation failure loop uncovered by the same incident diagnosis. **What happened?** Isolated-workspace runs failed at provision time with `setup_failed`. Every retry failed identically. One observed incident burned 4 runs across two adapters before the task was stranded. **Expected behavior** Provisioning either succeeds or degrades gracefully; a broken base CLI install repairs itself instead of permanently blocking all new worktrees. **Steps to reproduce** In the base workspace, cause a lockfile-affecting dependency bump plus a partial/filtered `pnpm install` so a package symlink (e.g. `cli/node_modules/drizzle-orm`) dangles into a pruned virtual-store dir. Start any isolated-workspace run. Provision fails with `ERR_MODULE_NOT_FOUND` and the run ends `setup_failed`; retries never recover. **Paperclip version or commit** master as of the branch point of this PR. **Deployment mode** Local trusted deployment with git-worktree isolated workspaces. ## What Changed - `base_cli_healthy` now boots the base CLI (`--help`) instead of only testing file existence, which exercises the top-level import graph. - New `repair_base_workspace_install`: when the health check fails, run a non-interactive `pnpm install --prod=false --force --frozen-lockfile` in the base workspace. `--force` guarantees relinking when pnpm's up-to-date heuristics would skip dangling symlinks; `--frozen-lockfile` keeps the repair from mutating the shared lockfile. - The repair install is serialized with `flock` on a lock file inside the resolved git dir (`git rev-parse --absolute-git-dir`), so locking also covers base workspaces that are linked worktrees, where `.git` is a file. - If every CLI candidate is unusable (including a base CLI the repair could not fix), provisioning falls back to the existing no-CLI fallback config writer (loudly, on stderr) instead of failing the run. A CLI that runs and fails `worktree init` still fails provisioning with its real exit code — that deliberate fail-closed policy is unchanged and covered by an existing server regression test. - Fixed a latent bug: `run_isolated_worktree_init` returned 0 unconditionally after the init subshell, so callers treated a failed init as success. Exit codes now propagate. ## Verification - Reproduced the incident state (dangling `cli/node_modules/drizzle-orm` symlink); the base CLI failed with the exact `ERR_MODULE_NOT_FOUND` seen in the incident run logs. - Ran the patched script against a fresh scratch worktree: health check failed → locked repair install ran (~26 s warm) → symlink relinked → `worktree init` completed → exit 0 with `.paperclip/config.json` and `.env` written. - Happy path (healthy base CLI): provisioning behavior unchanged, exit 0. - Verified `git rev-parse --absolute-git-dir` resolves a real directory for both a normal checkout and a linked worktree. - New hermetic tests: `node --test ./scripts/__tests__/provision-worktree-self-heal.test.mjs` (4 tests: healthy CLI used, broken CLI degrades, locked repair end-to-end with a fake pnpm, init failure propagates). Not yet wired into a CI workflow. - `server`: the existing `realizeExecutionWorkspace` fail-closed regression test ("fails instead of writing an unseeded fallback config when worktree init errors after CLI detection succeeds") passes against the new script. - `bash -n scripts/provision-worktree.sh` is clean. ## Risks - Low risk overall: the script only adds recovery paths; the happy path is unchanged. - The repair install runs in the shared base workspace. It is bounded by `--frozen-lockfile` (no lockfile mutation) and serialized by `flock`, but it can add ~30 s to the first provision after a base install breaks. - If the repair cannot fix the CLI and no other CLI candidate exists, runs now continue with an unseeded fallback config instead of failing; that is intentional, and the fallback path already existed. Genuine `worktree init` failures from a working CLI still fail the run. ## Model Used Claude Fable 5 (Anthropic, model ID `claude-fable-5`), extended thinking, agentic tool use (Claude Code harness). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing>
205 lines
8.0 KiB
JavaScript
205 lines
8.0 KiB
JavaScript
import assert from "node:assert/strict";
|
|
import { spawnSync } from "node:child_process";
|
|
import test from "node:test";
|
|
import fs from "node:fs";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
|
|
const script = new URL("../provision-worktree.sh", import.meta.url).pathname;
|
|
|
|
// Keep the PATH minimal so the fallback ladder is deterministic: node must be
|
|
// reachable, but a globally installed `paperclipai` must not shadow the paths
|
|
// under test.
|
|
const testPath = [path.dirname(process.execPath), "/usr/bin", "/bin"].join(":");
|
|
|
|
const cleanupDirs = [];
|
|
|
|
function makeTempDir(prefix) {
|
|
const dir = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
|
|
cleanupDirs.push(dir);
|
|
return dir;
|
|
}
|
|
|
|
test.after(() => {
|
|
for (const dir of cleanupDirs) {
|
|
fs.rmSync(dir, { recursive: true, force: true });
|
|
}
|
|
});
|
|
|
|
/**
|
|
* Writes a fake base workspace whose "tsx runner" is a plain node script, so
|
|
* the provision script's health check and init call can be steered per test.
|
|
*
|
|
* helpExit: exit code for `... index.ts --help` (the health check boot).
|
|
* initExit: exit code for `... index.ts worktree init ...`; on 0 the fake CLI
|
|
* writes a marker config so tests can tell CLI init from fallback.
|
|
*/
|
|
function makeBaseWorkspace({ helpExit, initExit }) {
|
|
const baseCwd = makeTempDir("paperclip-provision-base-");
|
|
const runnerPath = path.join(baseCwd, "cli", "node_modules", "tsx", "dist", "cli.mjs");
|
|
const entryPath = path.join(baseCwd, "cli", "src", "index.ts");
|
|
fs.mkdirSync(path.dirname(runnerPath), { recursive: true });
|
|
fs.mkdirSync(path.dirname(entryPath), { recursive: true });
|
|
fs.writeFileSync(entryPath, "// fake CLI entry\n");
|
|
fs.writeFileSync(
|
|
runnerPath,
|
|
`
|
|
import fs from "node:fs";
|
|
const cliArgs = process.argv.slice(3);
|
|
if (cliArgs.includes("--help")) {
|
|
if (${helpExit} !== 0) console.error("ERR_MODULE_NOT_FOUND: drizzle-orm");
|
|
process.exit(${helpExit});
|
|
}
|
|
if (cliArgs[0] === "worktree" && cliArgs[1] === "init") {
|
|
if (${initExit} !== 0) {
|
|
console.error("fake worktree init failure");
|
|
process.exit(${initExit});
|
|
}
|
|
fs.mkdirSync(".paperclip", { recursive: true });
|
|
fs.writeFileSync(".paperclip/config.json", JSON.stringify({ $meta: { source: "fake-cli" } }));
|
|
fs.writeFileSync(".paperclip/.env", "PAPERCLIP_IN_WORKTREE=true\\n");
|
|
process.exit(0);
|
|
}
|
|
process.exit(0);
|
|
`,
|
|
);
|
|
return baseCwd;
|
|
}
|
|
|
|
function runProvision(baseCwd, { pathPrefix } = {}) {
|
|
const worktreeCwd = makeTempDir("paperclip-provision-worktree-");
|
|
const worktreesHome = makeTempDir("paperclip-provision-home-");
|
|
const result = spawnSync("bash", [script], {
|
|
cwd: worktreeCwd,
|
|
encoding: "utf8",
|
|
env: {
|
|
PATH: pathPrefix ? `${pathPrefix}:${testPath}` : testPath,
|
|
HOME: os.homedir(),
|
|
PAPERCLIP_WORKSPACE_BASE_CWD: baseCwd,
|
|
PAPERCLIP_WORKSPACE_CWD: worktreeCwd,
|
|
PAPERCLIP_WORKSPACE_BRANCH: "feature/provision-test",
|
|
PAPERCLIP_WORKTREES_DIR: worktreesHome,
|
|
PAPERCLIP_HOME: path.join(worktreesHome, "no-such-instance-home"),
|
|
},
|
|
});
|
|
return { result, worktreeCwd, worktreesHome };
|
|
}
|
|
|
|
function readWorktreeConfig(worktreeCwd) {
|
|
const configPath = path.join(worktreeCwd, ".paperclip", "config.json");
|
|
assert.ok(fs.existsSync(configPath), `expected ${configPath} to exist`);
|
|
return JSON.parse(fs.readFileSync(configPath, "utf8"));
|
|
}
|
|
|
|
test("uses the base CLI when its import graph boots", () => {
|
|
const baseCwd = makeBaseWorkspace({ helpExit: 0, initExit: 0 });
|
|
const { result, worktreeCwd } = runProvision(baseCwd);
|
|
|
|
assert.equal(result.status, 0, result.stderr);
|
|
const config = readWorktreeConfig(worktreeCwd);
|
|
assert.equal(config.$meta.source, "fake-cli");
|
|
});
|
|
|
|
test("falls back to an isolated config when the base CLI cannot boot", () => {
|
|
// Simulates the dangling pnpm symlink incident: the runner and entry files
|
|
// exist, but booting the CLI fails ESM resolution. The base has no
|
|
// package.json/pnpm-lock.yaml, so the repair install is not possible and the
|
|
// script must degrade to the no-CLI fallback config instead of failing.
|
|
const baseCwd = makeBaseWorkspace({ helpExit: 1, initExit: 0 });
|
|
const { result, worktreeCwd, worktreesHome } = runProvision(baseCwd);
|
|
|
|
assert.equal(result.status, 0, result.stderr);
|
|
assert.match(result.stderr, /writing isolated fallback config/);
|
|
const config = readWorktreeConfig(worktreeCwd);
|
|
assert.equal(config.$meta.source, "configure");
|
|
const dataDir = config.database.embeddedPostgresDataDir;
|
|
assert.ok(
|
|
!path.relative(worktreesHome, dataDir).startsWith(".."),
|
|
`expected ${dataDir} to live under ${worktreesHome}`,
|
|
);
|
|
const env = fs.readFileSync(path.join(worktreeCwd, ".paperclip", ".env"), "utf8");
|
|
assert.match(env, /PAPERCLIP_IN_WORKTREE=true/);
|
|
});
|
|
|
|
test("repairs an unhealthy base install under the lock and then uses the CLI", (t) => {
|
|
const hasTools = ["flock", "git"].every(
|
|
(tool) => spawnSync("bash", ["-lc", `command -v ${tool}`], { env: { PATH: testPath } }).status === 0,
|
|
);
|
|
if (!hasTools) {
|
|
t.skip("flock or git not available on this host");
|
|
return;
|
|
}
|
|
|
|
// The CLI's health is controlled by a flag file, and a fake `pnpm install`
|
|
// creates that flag — modeling a forced reinstall that relinks the store.
|
|
const baseCwd = makeTempDir("paperclip-provision-repair-base-");
|
|
const healthFlag = path.join(baseCwd, "cli-healthy.flag");
|
|
const runnerPath = path.join(baseCwd, "cli", "node_modules", "tsx", "dist", "cli.mjs");
|
|
const entryPath = path.join(baseCwd, "cli", "src", "index.ts");
|
|
fs.mkdirSync(path.dirname(runnerPath), { recursive: true });
|
|
fs.mkdirSync(path.dirname(entryPath), { recursive: true });
|
|
fs.writeFileSync(entryPath, "// fake CLI entry\n");
|
|
fs.writeFileSync(
|
|
runnerPath,
|
|
`
|
|
import fs from "node:fs";
|
|
const cliArgs = process.argv.slice(3);
|
|
if (cliArgs.includes("--help")) {
|
|
process.exit(fs.existsSync(${JSON.stringify(healthFlag)}) ? 0 : 1);
|
|
}
|
|
if (cliArgs[0] === "worktree" && cliArgs[1] === "init") {
|
|
fs.mkdirSync(".paperclip", { recursive: true });
|
|
fs.writeFileSync(".paperclip/config.json", JSON.stringify({ $meta: { source: "fake-cli" } }));
|
|
fs.writeFileSync(".paperclip/.env", "PAPERCLIP_IN_WORKTREE=true\\n");
|
|
process.exit(0);
|
|
}
|
|
process.exit(0);
|
|
`,
|
|
);
|
|
fs.writeFileSync(path.join(baseCwd, "package.json"), "{}\n");
|
|
fs.writeFileSync(path.join(baseCwd, "pnpm-lock.yaml"), "lockfileVersion: '9.0'\n");
|
|
spawnSync("git", ["init", "-q", baseCwd], { env: { PATH: testPath } });
|
|
|
|
const fakeBin = makeTempDir("paperclip-provision-fakebin-");
|
|
const installLog = path.join(baseCwd, "pnpm-invocations.log");
|
|
fs.writeFileSync(
|
|
path.join(fakeBin, "pnpm"),
|
|
`#!/usr/bin/env bash
|
|
if [[ "$1" == "install" ]]; then
|
|
echo "$@" >> ${JSON.stringify(installLog)}
|
|
touch ${JSON.stringify(healthFlag)}
|
|
exit 0
|
|
fi
|
|
exit 1
|
|
`,
|
|
{ mode: 0o755 },
|
|
);
|
|
|
|
const { result, worktreeCwd } = runProvision(baseCwd, { pathPrefix: fakeBin });
|
|
|
|
assert.equal(result.status, 0, result.stderr);
|
|
const config = readWorktreeConfig(worktreeCwd);
|
|
assert.equal(config.$meta.source, "fake-cli");
|
|
const installs = fs.readFileSync(installLog, "utf8").trim().split("\n");
|
|
assert.equal(installs.length, 1, `expected exactly one repair install, got: ${installs.join(" | ")}`);
|
|
assert.match(installs[0], /--force/);
|
|
assert.match(installs[0], /--frozen-lockfile/);
|
|
assert.ok(
|
|
fs.existsSync(path.join(baseCwd, ".git", "paperclip-provision-repair.lock")),
|
|
"expected the repair lock file inside the resolved git dir",
|
|
);
|
|
});
|
|
|
|
test("a failed CLI init fails provisioning instead of being masked as success", () => {
|
|
// Regression test for the masked `return 0` after the init subshell: a CLI
|
|
// that passes the health check but fails `worktree init` signals a real
|
|
// problem, so the script must propagate the failure rather than report
|
|
// success or write an unseeded fallback config over it.
|
|
const baseCwd = makeBaseWorkspace({ helpExit: 0, initExit: 3 });
|
|
const { result, worktreeCwd } = runProvision(baseCwd);
|
|
|
|
assert.equal(result.status, 3, result.stderr);
|
|
assert.match(result.stderr, /fake worktree init failure/);
|
|
assert.ok(!fs.existsSync(path.join(worktreeCwd, ".paperclip", "config.json")));
|
|
});
|