fix(workspaces): make managed runtimes reliable across restarts (#11740)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Execution workspaces need isolated databases, ports, and runtime
services
> - Concurrent workspaces could reuse ports or lose service ownership
after a restart
> - A markerless worktree also needed seed recovery, but normal
markerless instances still needed to boot
> - This pull request makes seed, port, and service ownership state
explicit and recoverable
> - It also checks live process and listener identity before it reclaims
shared resources
> - The benefit is reliable workspace startup, restart, adoption, and
concurrent provisioning

## Linked Issues or Issue Description

**What happened?**

Managed workspaces could lose runtime service ownership after a
control-plane restart. Concurrent worktrees could also reuse a port when
their parent paths differed. A seed recovery change made every
markerless instance resolve a worktree seed source, so normal instances
without a source could not start.

**Expected behavior**

Paperclip must preserve healthy managed services across restarts. It
must reserve unique ports across worktree parents. It must provision a
registered markerless worktree, but it must skip seed work for a normal
markerless instance.

**Steps to reproduce**

1. Start two managed worktrees under different parent paths at the same
time.
2. Restart the control plane while a managed service stays alive.
3. Start Paperclip with a config that has no seed markers and no
registered worktree source.
4. Observe duplicate port selection, lost service adoption, or a
seed-source startup error.

**Paperclip version or commit**

Current `master` plus the workspace runtime reliability changes in this
pull request.

**Deployment mode**

Local development with managed execution workspaces and embedded
Postgres.

## What Changed

- Added a shared port registry with lease heartbeats, process identity
checks, and live listener probes.
- Reserved worktree ports across custom parent paths and repaired
duplicate legacy assignments.
- Preserved and adopted healthy managed services across control-plane
restarts.
- Reconciled guest bind modes and verified listener ownership before
termination or reuse.
- Provisioned registered markerless worktree databases and kept normal
markerless instance startup as a no-op.
- Added CLI, shared, server, and shell regression tests for seed, port,
listener, restart, and adoption behavior.
- Updated the worktree development documentation.

## Verification

- `pnpm exec vitest run cli/src/__tests__/worktree.test.ts
--reporter=verbose` — 63 tests passed.
- `pnpm exec vitest run
packages/shared/src/worktree-port-registry.test.ts --reporter=verbose` —
5 tests passed.
- Focused runtime Vitest set — 199 tests passed across 37 suites.
- `node --test scripts/__tests__/provision-worktree-self-heal.test.mjs`
— 10 tests passed.
- `git diff --check` passed.

## Risks

- Port reservation now depends on lease and process identity data. The
fallback listener probe prevents early reclamation when process metadata
is incomplete.
- Runtime adoption is stricter about bind and owner identity. The tests
cover healthy adoption, stale records, PID reuse, and unrelated
listeners.
- Markerless seed detection now separates registered worktrees from
normal instances. The tests cover both paths.
- There are no database schema migrations.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex with the `gpt-5` model family. The serving snapshot and
context-window size are not exposed. The agent used reasoning,
repository tools, code execution, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Dev Agent <dev@paperclip.ing>
This commit is contained in:
authored and GitHub committed 2026-08-19 14:55:16 -05:00
1 parent 433b1eb099
commit bd059a073d
21 files changed
+2406 -345

No files matched your search

@@ -0,0 +1,155 @@
import { spawn } from "node:child_process";
import { randomUUID } from "node:crypto";
import fs from "node:fs/promises";
import net from "node:net";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import {
resetRuntimeServicesForTests,
startRuntimeServicesForWorkspaceControl,
} from "../services/workspace-runtime.js";
import {
doesLocalServiceCommandLineMatch,
listLocalServiceRegistryRecords,
readLocalServicePortOwner,
resolveLocalServiceLogPath,
terminateLocalService,
} from "../services/local-service-supervisor.js";
describe("local service supervision", () => {
afterEach(async () => {
await resetRuntimeServicesForTests({ terminateProcesses: true });
});
it("keeps request-logging runtime stdio usable after the supervisor side closes", async () => {
const workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-service-stdio-"));
const paperclipHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-service-home-"));
const previousPaperclipHome = process.env.PAPERCLIP_HOME;
const previousInstanceId = process.env.PAPERCLIP_INSTANCE_ID;
process.env.PAPERCLIP_HOME = paperclipHome;
process.env.PAPERCLIP_INSTANCE_ID = `service-stdio-${randomUUID()}`;
let registryRecord: Awaited<ReturnType<typeof listLocalServiceRegistryRecords>>[number] | null = null;
try {
const [service] = await startRuntimeServicesForWorkspaceControl({
actor: { id: null, name: "Board", companyId: randomUUID() },
issue: null,
workspace: {
baseCwd: workspaceRoot,
source: "agent_home",
projectId: null,
workspaceId: null,
repoUrl: null,
repoRef: null,
strategy: "project_primary",
cwd: workspaceRoot,
branchName: null,
worktreePath: null,
warnings: [],
created: false,
},
config: {
workspaceRuntime: {
services: [{
name: "web",
command: "node -e \"const http=require('node:http'); process.on('SIGTERM',()=>{}); http.createServer((req,res)=>{ process.stdout.write('request '+req.url+'\\\\n',(error)=>{ if (!error) res.end('ok'); }); }).listen(Number(process.env.PORT), '127.0.0.1')\"",
port: { type: "auto" },
readiness: {
type: "http",
urlTemplate: "http://127.0.0.1:{{port}}",
timeoutSec: 10,
intervalMs: 100,
},
lifecycle: "shared",
reuseScope: "agent",
stopPolicy: { type: "manual" },
}],
},
},
adapterEnv: {},
});
expect(service?.url).toMatch(/^http:\/\/127\.0\.0\.1:\d+$/);
expect(service?.port).toBeTypeOf("number");
registryRecord = (await listLocalServiceRegistryRecords({ profileKind: "workspace-runtime" }))
.find((record) => record.runtimeServiceId === service!.id) ?? null;
expect(registryRecord).not.toBeNull();
await resetRuntimeServicesForTests({ simulateSupervisorExit: true });
await expect(fetch(`${service!.url}/after-restart`)).resolves.toMatchObject({ ok: true });
const log = await fs.readFile(resolveLocalServiceLogPath(registryRecord!.serviceKey), "utf8");
expect(log).toContain("request /after-restart");
await terminateLocalService(registryRecord!);
expect(await readLocalServicePortOwner(service!.port!)).toBeNull();
await expect(fetch(service!.url!)).rejects.toThrow();
registryRecord = null;
} finally {
if (registryRecord) await terminateLocalService(registryRecord).catch(() => undefined);
if (previousPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME;
else process.env.PAPERCLIP_HOME = previousPaperclipHome;
if (previousInstanceId === undefined) delete process.env.PAPERCLIP_INSTANCE_ID;
else process.env.PAPERCLIP_INSTANCE_ID = previousInstanceId;
await fs.rm(paperclipHome, { recursive: true, force: true });
await fs.rm(workspaceRoot, { recursive: true, force: true });
}
}, 15_000);
it("does not wait for an unrelated process that takes over the service port", async () => {
if (process.platform === "win32") return;
const unrelatedListener = net.createServer();
await new Promise<void>((resolve) => unrelatedListener.listen(0, "127.0.0.1", resolve));
const address = unrelatedListener.address();
const port = typeof address === "object" && address ? address.port : null;
if (!port) throw new Error("Failed to allocate unrelated listener port");
const child = spawn(process.execPath, ["-e", "setInterval(()=>{},1000)"], {
detached: true,
stdio: "ignore",
});
child.unref();
try {
await terminateLocalService({
pid: child.pid!,
processGroupId: child.pid!,
port,
});
expect(await readLocalServicePortOwner(port)).toBe(process.pid);
} finally {
try {
process.kill(-child.pid!, "SIGKILL");
} catch {
// The target process group was already terminated.
}
await new Promise<void>((resolve, reject) => {
unrelatedListener.close((error) => error ? reject(error) : resolve());
});
}
});
it("recognizes a pnpm command after the launcher becomes pnpm.cjs", () => {
expect(doesLocalServiceCommandLineMatch({
commandLine: "/usr/bin/node /opt/pnpm/pnpm.cjs dev -- --bind custom --bind-host 127.0.0.1",
recordedCommand: "pnpm dev -- --bind custom --bind-host 127.0.0.1",
serviceName: "paperclip-dev",
})).toBe(true);
});
it("does not accept a different command merely because it uses node", () => {
expect(doesLocalServiceCommandLineMatch({
commandLine: "/usr/bin/node /workspace/server/dist/index.js",
recordedCommand: "pnpm dev -- --bind custom --bind-host 127.0.0.1",
serviceName: "paperclip-dev",
})).toBe(false);
});
it("does not collapse different script paths to the same basename", () => {
expect(doesLocalServiceCommandLineMatch({
commandLine: "/usr/bin/node /other-workspace/server.js",
recordedCommand: "node ./server.js",
serviceName: "web",
})).toBe(false);
});
});
+291 -15
View File
@@ -448,8 +448,6 @@ describe("resolveRuntimeProvisionCommand", () => {
path.join(baseCwd, "scripts", "provision-worktree-runtime.sh"),
"#!/usr/bin/env bash\n",
);
await fs.mkdir(path.join(cwd, ".paperclip"), { recursive: true });
await fs.writeFile(path.join(cwd, ".paperclip", "seed-pending"), "{}\n");
const workspace = {
...buildWorkspace(cwd),
baseCwd,
@@ -457,6 +455,17 @@ describe("resolveRuntimeProvisionCommand", () => {
worktreePath: cwd,
};
expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe(
"bash ./scripts/provision-worktree-runtime.sh",
);
await fs.mkdir(path.join(cwd, ".paperclip"), { recursive: true });
await fs.writeFile(path.join(cwd, ".paperclip", "config.json"), "{}\n");
expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe(
"bash ./scripts/provision-worktree-runtime.sh",
);
await fs.writeFile(path.join(cwd, ".paperclip", "seed-pending"), "{}\n");
expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe(
"bash ./scripts/provision-worktree-runtime.sh",
);
@@ -466,7 +475,9 @@ describe("resolveRuntimeProvisionCommand", () => {
})).toBe("./custom-provision.sh");
await fs.writeFile(path.join(cwd, ".paperclip", "seed-complete"), "{}\n");
expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe("");
expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe(
"bash ./scripts/provision-worktree-runtime.sh",
);
await fs.writeFile(
path.join(cwd, ".paperclip", "seed-manifest.json"),
@@ -772,6 +783,55 @@ describe("realizeExecutionWorkspace", () => {
expect(second.branchName).toBe(first.branchName);
});
it("defaults the repo-provided worktree provisioner for git worktree strategies", async () => {
const repoRoot = await createTempRepo();
await fs.mkdir(path.join(repoRoot, "scripts"), { recursive: true });
await fs.writeFile(
path.join(repoRoot, "scripts", "provision-worktree.sh"),
[
"#!/usr/bin/env bash",
"set -euo pipefail",
"mkdir -p .paperclip",
"printf 'provisioned\\n' > .paperclip/default-provision-ran",
"",
].join("\n"),
"utf8",
);
await runGit(repoRoot, ["add", "scripts/provision-worktree.sh"]);
await runGit(repoRoot, ["commit", "-m", "Add repository worktree provisioner"]);
const workspace = await realizeExecutionWorkspace({
base: {
baseCwd: repoRoot,
source: "project_primary",
projectId: "project-1",
workspaceId: "workspace-1",
repoUrl: null,
repoRef: "HEAD",
},
config: {
workspaceStrategy: {
type: "git_worktree",
branchTemplate: "{{issue.identifier}}-{{slug}}",
},
},
issue: {
id: "issue-default-provision",
identifier: "PAP-17684",
title: "Default worktree provisioning",
},
agent: {
id: "agent-1",
name: "Codex Coder",
companyId: "company-1",
},
});
await expect(
fs.readFile(path.join(workspace.cwd, ".paperclip", "default-provision-ran"), "utf8"),
).resolves.toBe("provisioned\n");
});
it("warns when reusing a git worktree whose base ref has advanced", async () => {
const repoRoot = await createTempRepo();
@@ -7230,7 +7290,7 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
}
}, 40_000);
it("adopts a live auto-port shared service after runtime state is reset", async () => {
it("re-adopts a request-logging service on the same auto port after supervisor stdio closes", async () => {
const workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-reconcile-"));
const paperclipHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-home-"));
process.env.PAPERCLIP_HOME = paperclipHome;
@@ -7239,7 +7299,6 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
const companyId = randomUUID();
const agentId = randomUUID();
const runId = randomUUID();
const executionWorkspaceId = randomUUID();
await db.insert(companies).values({
id: companyId,
@@ -7273,6 +7332,16 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
projectId: null,
workspaceId: null,
};
await fs.writeFile(
path.join(workspaceRoot, "request-logger.cjs"),
[
"const http = require('node:http');",
"http.createServer((req, res) => {",
" process.stdout.write(`request ${req.url}\\n`, (error) => { if (!error) res.end('ok'); });",
"}).listen(Number(process.env.PORT), '127.0.0.1');",
].join("\n"),
"utf8",
);
leasedRunIds.add(runId);
const services = await ensureRuntimeServicesForRun({
@@ -7290,8 +7359,7 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
services: [
{
name: "web",
command:
"node -e \"require('node:http').createServer((req,res)=>res.end('ok')).listen(Number(process.env.PORT), '127.0.0.1')\"",
command: "node request-logger.cjs",
port: { type: "auto" },
readiness: {
type: "http",
@@ -7315,9 +7383,13 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
const service = services[0];
expect(service?.url).toMatch(/^http:\/\/127\.0\.0\.1:\d+$/);
await expect(fetch(service!.url!)).resolves.toMatchObject({ ok: true });
const originalPort = service!.port;
const originalProviderRef = service!.providerRef;
await fs.rm(paperclipHome, { recursive: true, force: true });
await resetRuntimeServicesForTests();
// Closing the parent's pipe endpoints reproduces a real control-plane exit.
// A service whose per-request logger still targets those pipes wedges (or
// exits) on the reconciliation health probe instead of answering it.
await resetRuntimeServicesForTests({ simulateSupervisorExit: true });
const result = await reconcilePersistedRuntimeServicesOnStartup(db);
expect(result).toMatchObject({ reconciled: 1, adopted: 1, stopped: 0 });
@@ -7328,17 +7400,221 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
.where(eq(workspaceRuntimeServices.id, service!.id))
.then((rows) => rows[0] ?? null);
expect(persisted?.status).toBe("running");
expect(persisted?.providerRef).toMatch(/^\d+$/);
expect(persisted?.port).toBe(originalPort);
expect(persisted?.providerRef).toBe(originalProviderRef);
await expect(fetch(service!.url!)).resolves.toMatchObject({ ok: true });
await stopRuntimeServicesForExecutionWorkspace({
db,
executionWorkspaceId,
workspaceCwd: workspace.cwd,
});
await resetRuntimeServicesForTests({ terminateProcesses: true });
leasedRunIds.delete(runId);
await fs.rm(paperclipHome, { recursive: true, force: true });
await fs.rm(workspaceRoot, { recursive: true, force: true });
await expect(fetch(service!.url!)).rejects.toThrow();
});
it("re-adopts a desired service when pnpm is represented as the pnpm.cjs launcher", async () => {
const workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-pnpm-reconcile-"));
const paperclipHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-home-"));
const previousPaperclipHome = process.env.PAPERCLIP_HOME;
const previousInstanceId = process.env.PAPERCLIP_INSTANCE_ID;
process.env.PAPERCLIP_HOME = paperclipHome;
process.env.PAPERCLIP_INSTANCE_ID = `runtime-pnpm-reconcile-${randomUUID()}`;
const portProbe = net.createServer();
await new Promise<void>((resolve) => portProbe.listen(0, "127.0.0.1", resolve));
const address = portProbe.address();
const port = typeof address === "object" && address ? address.port : null;
await new Promise<void>((resolve, reject) => {
portProbe.close((error) => error ? reject(error) : resolve());
});
if (!port) throw new Error("Failed to reserve pnpm reconciliation test port");
const companyId = randomUUID();
const projectId = randomUUID();
const executionWorkspaceId = randomUUID();
const runtimeServiceId = randomUUID();
const command = "pnpm dev";
const service = {
name: "web",
command,
port,
readiness: {
type: "http",
urlTemplate: "http://127.0.0.1:{{port}}",
timeoutSec: 10,
intervalMs: 50,
},
expose: null,
lifecycle: "shared",
reuseScope: "execution_workspace",
stopPolicy: { type: "manual" },
};
const reuseKey = createHash("sha256")
.update(stableStringifyForTest({
scopeType: "execution_workspace",
scopeId: executionWorkspaceId,
serviceName: service.name,
command,
cwd: workspaceRoot,
port,
env: {},
expose: null,
}))
.digest("hex");
const wrapperPath = path.join(workspaceRoot, "pnpm.cjs");
await fs.writeFile(
wrapperPath,
[
"const http = require('node:http');",
"const port = Number(process.argv[3]);",
"http.createServer((_req, res) => res.end('ok')).listen(port, '127.0.0.1');",
].join("\n"),
"utf8",
);
const child = spawn(process.execPath, [wrapperPath, "dev", String(port)], {
cwd: workspaceRoot,
detached: process.platform !== "win32",
stdio: "ignore",
});
child.unref();
try {
for (let attempt = 0; attempt < 50; attempt += 1) {
try {
if ((await fetch(`http://127.0.0.1:${port}`)).ok) break;
} catch {
// Wait for the simulated package-manager launcher to bind.
}
await new Promise((resolve) => setTimeout(resolve, 50));
}
await expect(fetch(`http://127.0.0.1:${port}`)).resolves.toMatchObject({ ok: true });
await db.insert(companies).values({
id: companyId,
name: "Paperclip",
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
requireBoardApprovalForNewAgents: false,
});
await db.insert(projects).values({
id: projectId,
companyId,
name: "Runtime pnpm reconciliation",
status: "in_progress",
});
await db.insert(executionWorkspaces).values({
id: executionWorkspaceId,
companyId,
projectId,
projectWorkspaceId: null,
mode: "isolated_workspace",
strategyType: "git_worktree",
name: "Runtime pnpm reconciliation workspace",
status: "active",
cwd: workspaceRoot,
providerType: "local_fs",
providerRef: workspaceRoot,
metadata: {
config: {
workspaceRuntime: { services: [service] },
desiredState: "running",
serviceStates: { "0": "running" },
},
},
});
await db.insert(workspaceRuntimeServices).values({
id: runtimeServiceId,
companyId,
projectId,
projectWorkspaceId: null,
executionWorkspaceId,
issueId: null,
scopeType: "execution_workspace",
scopeId: executionWorkspaceId,
serviceName: service.name,
status: "running",
lifecycle: "shared",
reuseKey,
command,
cwd: workspaceRoot,
port,
url: `http://127.0.0.1:${port}`,
provider: "local_process",
providerRef: String(child.pid),
ownerAgentId: null,
startedByRunId: null,
lastUsedAt: new Date(),
startedAt: new Date(),
stoppedAt: null,
stopPolicy: { type: "manual" },
healthStatus: "healthy",
});
await writeLocalServiceRegistryRecord({
version: 1,
serviceKey: `workspace-runtime-web-${randomUUID()}`,
profileKind: "workspace-runtime",
serviceName: service.name,
command,
cwd: workspaceRoot,
envFingerprint: reuseKey,
port,
url: `http://127.0.0.1:${port}`,
pid: child.pid!,
processGroupId: child.pid ?? null,
provider: "local_process",
runtimeServiceId,
reuseKey,
startedAt: new Date().toISOString(),
lastSeenAt: new Date().toISOString(),
metadata: { executionWorkspaceId },
});
const result = await reconcilePersistedRuntimeServicesOnStartup(db);
expect(result).toMatchObject({
reconciled: 1,
adopted: 1,
stopped: 0,
restarted: 0,
restartFailed: 0,
});
const rows = await db
.select()
.from(workspaceRuntimeServices)
.where(eq(workspaceRuntimeServices.executionWorkspaceId, executionWorkspaceId));
expect(rows).toHaveLength(1);
expect(rows[0]).toMatchObject({
id: runtimeServiceId,
status: "running",
healthStatus: "healthy",
port,
providerRef: String(child.pid),
});
expect(await readLocalServicePortOwner(port)).toBe(child.pid);
await expect(fetch(`http://127.0.0.1:${port}`)).resolves.toMatchObject({ ok: true });
} finally {
await stopRuntimeServicesForExecutionWorkspace({
db,
executionWorkspaceId,
workspaceCwd: workspaceRoot,
}).catch(() => undefined);
if (child.pid) {
try {
if (process.platform === "win32") process.kill(child.pid, "SIGKILL");
else process.kill(-child.pid, "SIGKILL");
} catch {
// The adopted runtime was already stopped through the managed path.
}
}
await resetRuntimeServicesForTests();
if (previousPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME;
else process.env.PAPERCLIP_HOME = previousPaperclipHome;
if (previousInstanceId === undefined) delete process.env.PAPERCLIP_INSTANCE_ID;
else process.env.PAPERCLIP_INSTANCE_ID = previousInstanceId;
await fs.rm(paperclipHome, { recursive: true, force: true });
await fs.rm(workspaceRoot, { recursive: true, force: true });
}
}, 20_000);
it("does not reuse a stopped auto-port service port while another process owns it", async () => {
const workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-unhealthy-adopt-"));
const paperclipHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-home-"));
+17 -3
View File
@@ -444,13 +444,27 @@ describe("worktree config repair", () => {
const firstConfigPath = path.join(firstWorktreeRoot, ".paperclip", "config.json");
const secondConfigPath = path.join(secondWorktreeRoot, ".paperclip", "config.json");
const writeWorktree = async (worktreeRoot: string, name: string) => {
const writeWorktree = async (
worktreeRoot: string,
name: string,
databaseMode: "embedded-postgres" | "postgres" = "embedded-postgres",
) => {
const paperclipDir = path.join(worktreeRoot, ".paperclip");
const instanceRoot = path.join(isolatedHome, "instances", name.toLowerCase());
const config = buildIsolatedConfig(instanceRoot, 45439, 55439);
await fs.mkdir(paperclipDir, { recursive: true });
await fs.writeFile(
path.join(paperclipDir, "config.json"),
`${JSON.stringify(buildIsolatedConfig(instanceRoot, 45439, 55439), null, 2)}\n`,
`${JSON.stringify({
...config,
database: {
...config.database,
mode: databaseMode,
...(databaseMode === "postgres"
? { connectionString: "postgres://paperclip:paperclip@127.0.0.1:55439/paperclip" }
: {}),
},
}, null, 2)}\n`,
"utf8",
);
await fs.writeFile(
@@ -480,7 +494,7 @@ describe("worktree config repair", () => {
delete process.env.DATABASE_URL;
};
await writeWorktree(firstWorktreeRoot, "PAP-14013-import-bulk-skills");
await writeWorktree(firstWorktreeRoot, "PAP-14013-import-bulk-skills", "postgres");
await writeWorktree(secondWorktreeRoot, "PAP-14069-port-conflicts");
const staleLockPath = path.join(isolatedHome, ".worktree-port-reservations.lock");
await fs.mkdir(staleLockPath, { recursive: true });
@@ -0,0 +1,229 @@
import { execFile } from "node:child_process";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { promisify } from "node:util";
import { afterEach, describe, expect, it } from "vitest";
import {
ensureWorktreeSeeded,
readWorktreeSeedManifest,
} from "../../../cli/src/commands/worktree.ts";
import { realizeExecutionWorkspace } from "../services/workspace-runtime.ts";
const execFileAsync = promisify(execFile);
const cleanup: string[] = [];
const originalConfig = process.env.PAPERCLIP_CONFIG;
const originalWorktreesDir = process.env.PAPERCLIP_WORKTREES_DIR;
async function runGit(cwd: string, args: string[]) {
await execFileAsync("git", args, { cwd });
}
async function writeConfig(configPath: string, instanceId: string) {
const instanceRoot = path.dirname(configPath);
await fs.mkdir(path.dirname(configPath), { recursive: true });
await fs.writeFile(configPath, `${JSON.stringify({
$meta: { version: 1, updatedAt: "2026-08-19T00:00:00.000Z", source: "configure" },
database: {
mode: "embedded-postgres",
embeddedPostgresDataDir: path.join(instanceRoot, "db"),
embeddedPostgresPort: 54329,
backup: { enabled: false, intervalMinutes: 60, retentionDays: 30, dir: path.join(instanceRoot, "backups") },
},
logging: { mode: "file", logDir: path.join(instanceRoot, "logs") },
server: {
deploymentMode: "local_trusted",
exposure: "private",
host: "127.0.0.1",
port: 3100,
allowedHostnames: [],
serveUi: true,
},
}, null, 2)}\n`, "utf8");
await fs.writeFile(
path.join(path.dirname(configPath), ".env"),
`PAPERCLIP_INSTANCE_ID=${instanceId}\n`,
"utf8",
);
}
function verifiedSeedResult() {
return {
backupSummary: "snapshot.sql",
snapshotAt: "2026-08-19T00:00:00.000Z",
migrationRevision: "0223_test.sql",
pausedScheduledRoutines: 0,
executionQuarantine: {
disabledTimerHeartbeats: 0,
resetRunningAgents: 0,
quarantinedInProgressIssues: 0,
unassignedTodoIssues: 0,
unassignedReviewIssues: 0,
stoppedProjectWorkspaceRuntimes: 0,
stoppedExecutionWorkspaceRuntimes: 0,
stoppedRuntimeServices: 0,
},
reboundWorkspaces: [],
validation: {
authUserCount: 1,
credentialAccountCount: 1,
instanceAdminCount: 1,
activeMembershipCount: 1,
companyCount: 1,
issueCount: 1,
representativeCompanyId: "00000000-0000-4000-8000-000000000001",
representativeIssueId: "00000000-0000-4000-8000-000000000002",
migrationRevision: "0223_test.sql",
},
};
}
afterEach(async () => {
if (originalConfig === undefined) delete process.env.PAPERCLIP_CONFIG;
else process.env.PAPERCLIP_CONFIG = originalConfig;
if (originalWorktreesDir === undefined) delete process.env.PAPERCLIP_WORKTREES_DIR;
else process.env.PAPERCLIP_WORKTREES_DIR = originalWorktreesDir;
for (const dir of cleanup.splice(0)) {
await fs.rm(dir, { recursive: true, force: true });
}
});
describe("managed worktree seed source through the server spawn path", () => {
it("re-derives an ambient-instance manifest written before provisioning", async () => {
const tempRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-server-seed-source-"));
cleanup.push(tempRoot);
const repoRoot = path.join(tempRoot, "repo");
const hooksDir = path.join(tempRoot, "hooks");
const ambientConfigPath = path.join(tempRoot, "ambient", "config.json");
const registeredConfigPath = path.join(repoRoot, ".paperclip", "config.json");
const worktreeHome = path.join(tempRoot, "worktree-home");
await fs.mkdir(repoRoot, { recursive: true });
await runGit(repoRoot, ["init", "-q"]);
await runGit(repoRoot, ["config", "user.email", "paperclip@example.com"]);
await runGit(repoRoot, ["config", "user.name", "Paperclip Test"]);
await fs.mkdir(path.join(repoRoot, "scripts"), { recursive: true });
await fs.writeFile(path.join(repoRoot, "README.md"), "server spawn regression\n", "utf8");
await fs.copyFile(
fileURLToPath(new URL("../../../scripts/provision-worktree.sh", import.meta.url)),
path.join(repoRoot, "scripts", "provision-worktree.sh"),
);
await fs.chmod(path.join(repoRoot, "scripts", "provision-worktree.sh"), 0o755);
await runGit(repoRoot, ["add", "README.md", "scripts/provision-worktree.sh"]);
await runGit(repoRoot, ["commit", "-qm", "Add managed provision script"]);
await writeConfig(registeredConfigPath, "registered-source");
await writeConfig(ambientConfigPath, "ambient-instance");
await fs.mkdir(worktreeHome, { recursive: true });
// Reproduce the integrated lane's ordering with a checkout hook: an earlier
// worktree-init-style writer inherits the server environment and leaves a valid
// target config whose manifest diagnostic points at the ambient instance.
await fs.mkdir(hooksDir, { recursive: true });
const hookPath = path.join(hooksDir, "post-checkout");
await fs.writeFile(
hookPath,
`#!/usr/bin/env node
const crypto = require("node:crypto");
const fs = require("node:fs");
const path = require("node:path");
const cwd = process.cwd();
if (cwd === ${JSON.stringify(repoRoot)}) process.exit(0);
const stateDir = path.join(cwd, ".paperclip");
const normalized = path.basename(cwd).trim().toLowerCase().replace(/[^a-z0-9_-]+/g, "-").replace(/-+/g, "-").replace(/^[-_]+|[-_]+$/g, "");
const instanceId = \`${"${(normalized || \"worktree\").slice(0, 48)}"}-\${crypto.createHash("sha256").update(path.resolve(cwd)).digest("hex").slice(0, 12)}\`;
const targetConfigPath = path.join(stateDir, "config.json");
const instanceRoot = path.join(process.env.PAPERCLIP_WORKTREES_DIR, "instances", instanceId);
fs.mkdirSync(stateDir, { recursive: true });
fs.mkdirSync(instanceRoot, { recursive: true });
fs.writeFileSync(targetConfigPath, JSON.stringify({
$meta: { version: 1, updatedAt: "2026-08-19T00:00:00.000Z", source: "configure" },
database: {
mode: "embedded-postgres",
embeddedPostgresDataDir: path.join(instanceRoot, "db"),
embeddedPostgresPort: 54330,
backup: { enabled: false, intervalMinutes: 60, retentionDays: 30, dir: path.join(instanceRoot, "backups") },
},
logging: { mode: "file", logDir: path.join(instanceRoot, "logs") },
server: {
deploymentMode: "local_trusted",
exposure: "private",
host: "127.0.0.1",
port: 3101,
allowedHostnames: [],
serveUi: true,
},
}, null, 2) + "\\n");
fs.writeFileSync(path.join(stateDir, ".env"), [
"PAPERCLIP_HOME=" + JSON.stringify(process.env.PAPERCLIP_WORKTREES_DIR),
"PAPERCLIP_INSTANCE_ID=" + JSON.stringify(instanceId),
"PAPERCLIP_CONFIG=" + JSON.stringify(targetConfigPath),
"",
].join("\\n"));
fs.writeFileSync(path.join(stateDir, "seed-manifest.json"), JSON.stringify({
version: 2,
source: { instanceId: "ambient-instance", configPath: process.env.PAPERCLIP_CONFIG },
snapshotAt: null,
seedMode: "minimal",
migrationRevision: null,
targetInstanceId: instanceId,
phase: "pending",
state: "pending",
attemptId: "ambient-writer",
startedAt: null,
finishedAt: null,
diagnostics: [{ phase: "pending", status: "succeeded", at: new Date().toISOString() }],
}, null, 2) + "\\n");
`,
"utf8",
);
await fs.chmod(hookPath, 0o755);
await runGit(repoRoot, ["config", "core.hooksPath", hooksDir]);
process.env.PAPERCLIP_CONFIG = ambientConfigPath;
process.env.PAPERCLIP_WORKTREES_DIR = worktreeHome;
const workspace = await realizeExecutionWorkspace({
base: {
baseCwd: repoRoot,
source: "project_primary",
projectId: "project-1",
workspaceId: "project-workspace-1",
repoUrl: null,
repoRef: "HEAD",
},
config: {
workspaceStrategy: {
type: "git_worktree",
branchTemplate: "{{issue.identifier}}-{{slug}}",
provisionCommand: "bash ./scripts/provision-worktree.sh",
},
},
issue: {
id: "issue-1",
identifier: "PAP-17681",
title: "Recover ambient seed source",
},
agent: { id: "agent-1", name: "Coder", companyId: "company-1" },
});
expect(readWorktreeSeedManifest(path.join(workspace.cwd, ".paperclip", "config.json"))).toMatchObject({
source: { instanceId: "ambient-instance", configPath: ambientConfigPath },
state: "pending",
});
await expect(ensureWorktreeSeeded({
config: path.join(workspace.cwd, ".paperclip", "config.json"),
registeredBaseWorkspaceCwd: repoRoot,
registeredProjectWorkspaceId: "project-workspace-1",
expectedCompanyId: "company-1",
}, {
seedDatabase: async () => verifiedSeedResult(),
})).resolves.toMatchObject({ seeded: true, reason: "seeded" });
expect(readWorktreeSeedManifest(path.join(workspace.cwd, ".paperclip", "config.json"))).toMatchObject({
source: { instanceId: "registered-source", configPath: registeredConfigPath },
state: "verified",
phase: "complete",
});
}, 20_000);
});
+149 -23
View File
@@ -1,5 +1,6 @@
import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import fsSync from "node:fs";
import fs from "node:fs/promises";
import path from "node:path";
import { setTimeout as delay } from "node:timers/promises";
@@ -63,6 +64,32 @@ function getRuntimeServicesDir() {
return path.resolve(resolvePaperclipInstanceRoot(), "runtime-services");
}
function getRuntimeServiceLogsDir() {
return path.resolve(resolvePaperclipInstanceRoot(), "runtime-service-logs");
}
export function resolveLocalServiceLogPath(serviceKey: string) {
if (!/^[a-z0-9._-]+$/.test(serviceKey)) {
throw new Error("Invalid local service key for log path");
}
return path.resolve(getRuntimeServiceLogsDir(), `${serviceKey}.log`);
}
/**
* Open a managed service's durable append-only output file.
*
* The returned descriptor is intended to be passed directly to spawn(). The
* child receives its own duplicate, so the caller can close this handle as soon
* as spawn returns without tying the service's stdio lifetime to Paperclip's.
*/
export async function openLocalServiceLogFile(serviceKey: string) {
await fs.mkdir(getRuntimeServiceLogsDir(), { recursive: true });
const logPath = resolveLocalServiceLogPath(serviceKey);
const handle = await fs.open(logPath, "a+", 0o600);
const startOffset = (await handle.stat()).size;
return { handle, logPath, startOffset };
}
function getRuntimeServiceRegistryPath(serviceKey: string) {
return path.resolve(getRuntimeServicesDir(), `${serviceKey}.json`);
}
@@ -230,10 +257,91 @@ export function isProcessGroupAlive(processGroupId: number | null | undefined) {
if (typeof processGroupId !== "number" || !Number.isInteger(processGroupId) || processGroupId <= 0) return false;
try {
process.kill(-processGroupId, 0);
return true;
} catch {
return false;
}
if (process.platform === "linux") {
const liveMember = readLinuxProcessGroupActivity(processGroupId);
if (liveMember !== null) return liveMember;
}
return true;
}
function readLinuxProcessGroupActivity(processGroupId: number): boolean | null {
let entries: fsSync.Dirent[];
try {
entries = fsSync.readdirSync("/proc", { withFileTypes: true });
} catch {
return null;
}
let foundMember = false;
for (const entry of entries) {
if (!entry.isDirectory() || !/^\d+$/.test(entry.name)) continue;
try {
const stat = fsSync.readFileSync(`/proc/${entry.name}/stat`, "utf8");
const commandEnd = stat.lastIndexOf(")");
if (commandEnd < 0) continue;
const fields = stat.slice(commandEnd + 1).trim().split(/\s+/);
const state = fields[0];
const memberProcessGroupId = Number.parseInt(fields[2] ?? "", 10);
if (memberProcessGroupId !== processGroupId) continue;
foundMember = true;
if (state !== "Z" && state !== "X") return true;
} catch {
// The process can exit while /proc is scanned.
}
}
// kill(-pgid, 0) also succeeds for a group that contains only zombies. Such
// processes cannot run or own a listener and are waiting only for their
// parent to reap them, so termination is complete for service-control use.
return foundMember ? false : null;
}
function tokenizeCommandLine(value: string) {
return value.match(/"(?:\\.|[^"\\])*"|'[^']*'|\S+/g) ?? [];
}
function normalizeCommandToken(value: string) {
const unquoted = value.replace(/^["']|["']$/g, "");
const basename = path.basename(unquoted.replace(/\\/g, "/"));
const launcher = basename.replace(/\.(?:cjs|mjs|js|cmd|exe)$/i, "");
return /^(?:bun|node|nodejs|npm|npx|pnpm|yarn)$/i.test(launcher) ? launcher : unquoted;
}
/**
* Compare a configured service command with the argv exposed by the OS.
*
* Package-manager launchers commonly replace `pnpm dev` with
* `node /path/to/pnpm.cjs dev` after the shell starts. A literal substring
* check rejects that surviving process even though the executable and all
* configured arguments are still present. Normalize executable paths and
* script extensions, then require the configured argv to remain contiguous.
*/
export function doesLocalServiceCommandLineMatch(input: {
commandLine: string;
recordedCommand: string;
serviceName: string;
}) {
const normalize = (value: string) => value.replace(/["']/g, "").replace(/\s+/g, " ").trim();
const normalizedCommandLine = normalize(input.commandLine);
const normalizedRecordedCommand = normalize(input.recordedCommand);
if (
normalizedCommandLine.includes(normalizedRecordedCommand)
|| normalizedCommandLine.includes(input.serviceName)
) {
return true;
}
const actualTokens = tokenizeCommandLine(input.commandLine).map(normalizeCommandToken);
const recordedTokens = tokenizeCommandLine(input.recordedCommand).map(normalizeCommandToken);
if (recordedTokens.length === 0 || recordedTokens.length > actualTokens.length) return false;
return actualTokens.some((_, start) => recordedTokens.every(
(token, offset) => actualTokens[start + offset] === token,
));
}
async function isLikelyMatchingCommand(record: LocalServiceRegistryRecord) {
@@ -242,10 +350,11 @@ async function isLikelyMatchingCommand(record: LocalServiceRegistryRecord) {
const { stdout } = await execFileAsync("ps", ["-o", "command=", "-p", String(record.pid)]);
const commandLine = stdout.trim();
if (!commandLine) return false;
const normalize = (value: string) => value.replace(/["']/g, "").replace(/\s+/g, " ").trim();
const normalizedCommandLine = normalize(commandLine);
const normalizedRecordedCommand = normalize(record.command);
return normalizedCommandLine.includes(normalizedRecordedCommand) || normalizedCommandLine.includes(record.serviceName);
return doesLocalServiceCommandLineMatch({
commandLine,
recordedCommand: record.command,
serviceName: record.serviceName,
});
} catch {
return true;
}
@@ -396,11 +505,34 @@ export async function touchLocalServiceRegistryRecord(
}
export async function terminateLocalService(
record: Pick<LocalServiceRegistryRecord, "pid" | "processGroupId">,
opts?: { signal?: NodeJS.Signals; forceAfterMs?: number },
record: Pick<LocalServiceRegistryRecord, "pid" | "processGroupId"> &
Partial<Pick<LocalServiceRegistryRecord, "port">>,
opts?: { signal?: NodeJS.Signals; forceAfterMs?: number; verifyAfterMs?: number },
) {
const signal = opts?.signal ?? "SIGTERM";
const targetProcessGroup = process.platform !== "win32" && record.processGroupId && record.processGroupId > 0;
const targetIsGone = async () => {
const targetAlive = targetProcessGroup
? isProcessGroupAlive(record.processGroupId)
: isPidAlive(record.pid);
if (targetAlive) return false;
if (!record.port) return true;
const portOwnerPid = await readLocalServicePortOwner(record.port);
if (!portOwnerPid) return true;
const ownerProcessId = targetProcessGroup ? record.processGroupId! : record.pid;
return !(await isLocalServiceProcessOwnedBy(portOwnerPid, ownerProcessId));
};
const waitUntilGone = async (timeoutMs: number) => {
const deadline = Date.now() + timeoutMs;
do {
if (await targetIsGone()) return true;
await delay(100);
} while (Date.now() < deadline);
return await targetIsGone();
};
try {
if (targetProcessGroup) {
process.kill(-record.processGroupId!, signal);
@@ -408,24 +540,10 @@ export async function terminateLocalService(
process.kill(record.pid, signal);
}
} catch {
return;
if (await targetIsGone()) return;
}
const deadline = Date.now() + (opts?.forceAfterMs ?? 2_000);
while (Date.now() < deadline) {
const targetAlive = targetProcessGroup
? isProcessGroupAlive(record.processGroupId)
: isPidAlive(record.pid);
if (!targetAlive) {
return;
}
await delay(100);
}
const stillAlive = targetProcessGroup
? isProcessGroupAlive(record.processGroupId)
: isPidAlive(record.pid);
if (!stillAlive) return;
if (await waitUntilGone(opts?.forceAfterMs ?? 2_000)) return;
try {
if (targetProcessGroup) {
process.kill(-record.processGroupId!, "SIGKILL");
@@ -435,6 +553,14 @@ export async function terminateLocalService(
} catch {
// Ignore cleanup races.
}
if (await waitUntilGone(opts?.verifyAfterMs ?? 2_000)) return;
const target = targetProcessGroup
? `process group ${record.processGroupId}`
: `process ${record.pid}`;
const listener = record.port ? ` and listener on port ${record.port}` : "";
throw new Error(`Failed to terminate local service ${target}${listener}`);
}
export async function readLocalServicePortOwner(port: number) {
@@ -162,7 +162,7 @@ describe("diagnoseRuntimeListenerBinds against live listeners", () => {
expect(diagnosis).toContain(`port ${appPort}`);
// Node's hostless listen is dual-stack, so /proc shows :: and/or 0.0.0.0.
expect(diagnosis).toMatch(/0\.0\.0\.0|::/);
expect(diagnosis).toContain("--bind custom --bind-host 127.0.0.1");
expect(diagnosis).toContain("--bind loopback");
});
});
@@ -160,6 +160,6 @@ export async function diagnoseRuntimeListenerBinds(ports: number[]): Promise<str
`${violations.join("; ")} instead of loopback only. The broker will not expose a listener `
+ "reachable off-loopback. This means the workspace checkout's dev server ignored the managed "
+ "loopback bind — a checkout that predates managed HTTPS exposure overwrites PAPERCLIP_BIND "
+ "from its own --bind argv, so the start command must pass --bind custom --bind-host 127.0.0.1."
+ "from its own --bind argv, so the start command must pass --bind loopback."
);
}
@@ -120,6 +120,10 @@ beforeAll(async () => {
await fs.writeFile(path.join(guestDir, "dev-runner.mjs"), PRE_MANAGED_EXPOSURE_GUEST);
await fs.writeFile(path.join(guestDir, "dev-runner-legacy.mjs"), ALWAYS_WILDCARD_GUEST);
await fs.writeFile(path.join(guestDir, "dev-runner-wildcard-hmr.mjs"), WILDCARD_HMR_GUEST);
await fs.writeFile(
path.join(guestDir, "dev-runner-bind-conflict.mjs"),
'process.stderr.write("local_trusted requires server.bind=loopback\\n"); process.exit(1);\n',
);
});
afterAll(async () => {
@@ -474,7 +478,7 @@ describe("loopback bind is forced on the guest, not merely requested (PAP-17256)
}));
// The launched command carries the loopback bind, replacing `--bind lan`.
expect(runtime.command).toContain("--bind custom --bind-host 127.0.0.1");
expect(runtime.command).toContain("--bind loopback");
expect(runtime.command).not.toContain("--bind lan");
// And because the guest honoured it, both listeners are loopback-only, so
@@ -530,7 +534,7 @@ describe("loopback bind is forced on the guest, not merely requested (PAP-17256)
// The bare code alone is what cost PAP-17254 three diagnostic cycles.
expect(error!.message).toContain("listener_ownership_mismatch");
expect(error!.message).toContain("loopback");
expect(error!.message).toContain("--bind custom --bind-host 127.0.0.1");
expect(error!.message).toContain("--bind loopback");
}, 20_000);
it("leaves a non-Paperclip service's --bind argument alone", async () => {
@@ -547,7 +551,7 @@ describe("loopback bind is forced on the guest, not merely requested (PAP-17256)
}));
expect(runtime.command).toBe(declared);
expect(runtime.command).not.toContain("--bind custom");
expect(runtime.command).not.toContain("--bind loopback");
expect(calls.slice(0, 2)).toEqual(["reserve", "expose"]);
expect(runtime.exposure?.state).toBe("ready");
}, 20_000);
@@ -567,6 +571,20 @@ describe("loopback bind is forced on the guest, not merely requested (PAP-17256)
expect(calls).toEqual([]);
expect(runtime.command).toBe(declared);
}, 20_000);
it("surfaces a deployment/bind conflict from a guest that exits during startup", async () => {
const { broker } = createBroker();
installDeps({ broker });
await expect(startRuntimeServicesForWorkspaceControl(startInput({
serviceName: "paperclip-dev",
command: guestCommand("dev-runner-bind-conflict.mjs"),
expose: LEGACY_HTTP_EXPOSE,
port: { type: "auto", envKey: "PORT" },
}))).rejects.toThrow(
/deployment\/bind conflict: local_trusted requires server\.bind=loopback.*output: local_trusted requires server\.bind=loopback/s,
);
}, 20_000);
});
describe("readiness probes loopback for an exposed runtime (PAP-17256)", () => {
+94 -55
View File
@@ -52,6 +52,7 @@ import {
findAdoptableLocalService,
isLocalServiceProcessOwnedBy,
isLocalServiceProcessInWorkspace,
openLocalServiceLogFile,
readLocalServiceProcessCwd,
readLocalServicePortOwner,
removeLocalServiceRegistryRecord,
@@ -475,7 +476,7 @@ type ProcessOutputAccumulator = {
* broker fake handles the removal rather than the real host broker.
*/
export async function resetRuntimeServicesForTests(
opts: { terminateProcesses?: boolean } = {},
opts: { terminateProcesses?: boolean; simulateSupervisorExit?: boolean } = {},
) {
if (opts.terminateProcesses) {
for (const serviceId of [...runtimeServicesById.keys()]) {
@@ -484,6 +485,13 @@ export async function resetRuntimeServicesForTests(
}
for (const record of runtimeServicesById.values()) {
clearIdleTimer(record);
if (opts.simulateSupervisorExit) {
// A real supervisor exit closes its side of every inherited pipe. Tests
// use this to prove surviving request-logging services do not depend on
// Paperclip keeping an anonymous stdio peer alive.
record.child?.stdout?.destroy();
record.child?.stderr?.destroy();
}
}
runtimeServicesById.clear();
runtimeServicesByReuseKey.clear();
@@ -2757,6 +2765,20 @@ function quoteShellArg(value: string) {
return `'${value.replace(/'/g, `'\\''`)}'`;
}
const BUILTIN_WORKSPACE_PROVISION_COMMAND = "bash ./scripts/provision-worktree.sh";
function resolveWorkspaceProvisionCommand(
strategy: Record<string, unknown>,
repoRoot: string,
) {
const configuredCommand = asString(strategy.provisionCommand, "").trim();
if (configuredCommand) return configuredCommand;
return existsSync(path.join(repoRoot, "scripts", "provision-worktree.sh"))
? BUILTIN_WORKSPACE_PROVISION_COMMAND
: "";
}
function resolveRepoManagedWorkspaceCommand(command: string, repoRoot: string) {
const patterns = [
/^(?<prefix>(?:bash|sh|zsh)\s+)(?<quote>["']?)(?<relative>\.\/[^"'\s]+)\k<quote>(?<suffix>(?:\s.*)?)$/s,
@@ -2953,7 +2975,7 @@ async function provisionExecutionWorktree(input: {
created: boolean;
recorder?: WorkspaceOperationRecorder | null;
}) {
const provisionCommand = asString(input.strategy.provisionCommand, "").trim();
const provisionCommand = resolveWorkspaceProvisionCommand(input.strategy, input.repoRoot);
if (!provisionCommand) return;
const resolvedProvisionCommand = resolveRepoManagedWorkspaceCommand(provisionCommand, input.repoRoot);
@@ -3434,22 +3456,20 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
});
realized.warnings = [...repairWarnings, ...baseRefreshWarnings, ...baseDrift.warnings];
realized.baseRefSha = refresh.baseRefSha ?? recordedBaseRefSha ?? baseDrift.branchBaseRefSha ?? baseDrift.currentBaseRefSha;
if (provisionCommand) {
await provisionExecutionWorktree({
strategy: {
type: "git_worktree",
provisionCommand,
},
base: input.base,
repoRoot,
worktreePath: realized.worktreePath ?? cwd,
branchName: realized.branchName ?? "",
issue: input.issue,
agent: input.agent,
created: false,
recorder: input.recorder ?? null,
});
}
await provisionExecutionWorktree({
strategy: {
type: "git_worktree",
...(provisionCommand ? { provisionCommand } : {}),
},
base: input.base,
repoRoot,
worktreePath: realized.worktreePath ?? cwd,
branchName: realized.branchName ?? "",
issue: input.issue,
agent: input.agent,
created: false,
recorder: input.recorder ?? null,
});
return realized;
}
@@ -5139,17 +5159,12 @@ export function resolveRuntimeProvisionCommand(input: {
const stateDir = path.join(input.workspace.cwd, ".paperclip");
const manifestPath = path.join(stateDir, "seed-manifest.json");
const pendingMarker = path.join(stateDir, "seed-pending");
const completeMarker = path.join(stateDir, "seed-complete");
const provisionScript = path.join(
input.workspace.baseCwd,
"scripts",
"provision-worktree-runtime.sh",
);
let needsSeed = existsSync(pendingMarker) && !existsSync(completeMarker);
if (existsSync(manifestPath)) {
needsSeed = !hasVerifiedWorktreeSeedManifest(manifestPath);
}
const needsSeed = !existsSync(manifestPath) || !hasVerifiedWorktreeSeedManifest(manifestPath);
if (!needsSeed || !existsSync(provisionScript)) {
return "";
}
@@ -5535,7 +5550,7 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
// still rejected by the broker unless /proc proves loopback-only listeners.
//
// Three independent layers force the loopback bind, because a guest checkout
// can be arbitrarily old (PAP-17256): the `--bind custom --bind-host` argv
// can be arbitrarily old (PAP-17256): the `--bind loopback` argv
// added above, these env vars for a runner that reads them, and HOST for one
// old enough to ignore both and infer its bind mode from HOST alone.
env.PAPERCLIP_BIND = RUNTIME_EXPOSURE_BIND_MODE;
@@ -5748,12 +5763,21 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
}
const shell = resolveShell();
const child = spawn(shell, ["-lc", command], {
cwd: serviceCwd,
env,
detached: process.platform !== "win32",
stdio: ["ignore", "pipe", "pipe"],
});
const serviceLog = await openLocalServiceLogFile(serviceKey);
let child: ChildProcess;
try {
child = spawn(shell, ["-lc", command], {
cwd: serviceCwd,
env,
detached: process.platform !== "win32",
// The service receives duplicate append-only file descriptors. Closing
// Paperclip (or this parent handle below) cannot strand a request logger
// on an orphaned socketpair during startup reconciliation.
stdio: ["ignore", serviceLog.handle.fd, serviceLog.handle.fd],
});
} finally {
await serviceLog.handle.close();
}
record.child = child;
record.providerRef = child.pid ? String(child.pid) : null;
record.processGroupId = child.pid ?? null;
@@ -5763,24 +5787,23 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
});
});
const earlyExitPromise = new Promise<never>((_, reject) => {
child.once("exit", (code, signal) => {
// `close` follows `exit` after the child's inherited stdout/stderr file
// descriptors are closed. Waiting for it makes the startup log excerpt
// deterministic instead of racing the final validation line.
child.once("close", (code, signal) => {
reject(new Error(
`service process exited before readiness (code ${code ?? "unknown"}, signal ${signal ?? "none"})`,
));
});
});
let stderrExcerpt = "";
let stdoutExcerpt = "";
child.stdout?.on("data", async (chunk) => {
const text = String(chunk);
stdoutExcerpt = (stdoutExcerpt + text).slice(-4096);
if (input.onLog) await input.onLog("stdout", `[service:${serviceName}] ${text}`);
});
child.stderr?.on("data", async (chunk) => {
const text = String(chunk);
stderrExcerpt = (stderrExcerpt + text).slice(-4096);
if (input.onLog) await input.onLog("stderr", `[service:${serviceName}] ${text}`);
});
const readServiceOutputExcerpt = async () => {
try {
const contents = await fs.readFile(serviceLog.logPath);
return contents.subarray(Math.max(serviceLog.startOffset, contents.length - 4096)).toString("utf8");
} catch {
return "";
}
};
if (child.pid) {
await writeLocalServiceRegistryRecord({
@@ -5907,6 +5930,10 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
record.healthStatus = "healthy";
record.lastUsedAt = new Date().toISOString();
record.stoppedAt = null;
const serviceOutputExcerpt = await readServiceOutputExcerpt();
if (serviceOutputExcerpt && input.onLog) {
await input.onLog("stdout", `[service:${serviceName}] ${serviceOutputExcerpt}`);
}
await touchLocalServiceRegistryRecord(record.serviceKey, {
runtimeServiceId: record.id,
lastSeenAt: record.lastUsedAt,
@@ -5914,18 +5941,20 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
}).catch(async (err) => {
releasePortReservation(reservedPort);
releasePortReservation(claimedIdentityPort);
const failureMessage = err instanceof Error ? err.message : String(err);
const failureMessage = err instanceof Error ? err.message : String(err);
const serviceOutputExcerpt = await readServiceOutputExcerpt();
const bindCollision = !exposureConfig && (
err instanceof RuntimeServicePortBindCollision || Boolean(
port
&& (input.allowFixedPortFallback || portType === "auto")
&& /(?:EADDRINUSE|address already in use)/i.test(`${failureMessage}\n${stderrExcerpt}`),
&& /(?:EADDRINUSE|address already in use)/i.test(`${failureMessage}\n${serviceOutputExcerpt}`),
)
);
if (child.pid) {
await terminateLocalService({
pid: child.pid,
processGroupId: child.pid,
port,
});
}
await cleanupRecordExposure(record, { preserveFailure: true });
@@ -5938,8 +5967,14 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
await persistRuntimeServiceRecord(record.db, record).catch(() => undefined);
}
if (bindCollision && port) throw new RuntimeServicePortBindCollision(port);
const deploymentBindConflict = /local_trusted requires server\.bind=loopback/i.test(
`${failureMessage}\n${serviceOutputExcerpt}`,
);
const actionableFailure = deploymentBindConflict
? `${failureMessage} | deployment/bind conflict: local_trusted requires server.bind=loopback; the managed runtime requested an incompatible bind mode`
: failureMessage;
throw new Error(
`Failed to start runtime service "${serviceName}": ${failureMessage}${stderrExcerpt ? ` | stderr: ${stderrExcerpt.trim()}` : ""}`,
`Failed to start runtime service "${serviceName}": ${actionableFailure}${serviceOutputExcerpt ? ` | output: ${serviceOutputExcerpt.trim()}` : ""}`,
);
});
@@ -6125,19 +6160,14 @@ async function stopRuntimeService(serviceId: string) {
const record = runtimeServicesById.get(serviceId);
if (!record) return;
clearIdleTimer(record);
record.status = "stopped";
record.healthStatus = "unknown";
record.lastUsedAt = new Date().toISOString();
record.stoppedAt = new Date().toISOString();
runtimeServicesById.delete(serviceId);
if (record.reuseKey && runtimeServicesByReuseKey.get(record.reuseKey) === record.id) {
runtimeServicesByReuseKey.delete(record.reuseKey);
}
// Remove any public exposure first, but keep the process registered and the
// row non-stopped until verified termination succeeds.
await cleanupRecordExposure(record);
if (record.child && record.child.pid) {
await terminateLocalService({
pid: record.child.pid,
processGroupId: record.processGroupId ?? record.child.pid,
port: record.port,
});
} else if (record.providerRef) {
const pid = Number.parseInt(record.providerRef, 10);
@@ -6145,9 +6175,18 @@ async function stopRuntimeService(serviceId: string) {
await terminateLocalService({
pid,
processGroupId: record.processGroupId,
port: record.port,
});
}
}
record.status = "stopped";
record.healthStatus = "unknown";
record.lastUsedAt = new Date().toISOString();
record.stoppedAt = new Date().toISOString();
runtimeServicesById.delete(serviceId);
if (record.reuseKey && runtimeServicesByReuseKey.get(record.reuseKey) === record.id) {
runtimeServicesByReuseKey.delete(record.reuseKey);
}
await removeLocalServiceRegistryRecord(record.serviceKey);
await persistRuntimeServiceRecord(record.db, record);
}
+6 -86
View File
@@ -8,6 +8,11 @@ import {
type PaperclipConfig,
} from "@paperclipai/shared";
import { updateEnvFileContents, writeEnvFileAtomicallyIfChanged } from "@paperclipai/shared/env-file";
import {
readWorktreePortRegistry,
withWorktreePortRegistryLockSync,
writeWorktreePortRegistry,
} from "@paperclipai/shared/worktree-port-registry";
import { resolvePaperclipConfigPath, resolvePaperclipEnvPath } from "./paths.js";
import { rewriteUrlPort } from "./url-utils.js";
@@ -99,89 +104,6 @@ type WorktreeRuntimeContext = {
secretsKeyFilePath: string;
};
type WorktreePortRegistry = {
version: 1;
configPaths: string[];
};
const WORKTREE_PORT_REGISTRY_FILE = "worktree-port-reservations.json";
const WORKTREE_PORT_REGISTRY_LOCK_DIR = ".worktree-port-reservations.lock";
const WORKTREE_PORT_REGISTRY_LOCK_STALE_MS = 5_000;
const WORKTREE_PORT_REGISTRY_LOCK_TIMEOUT_MS = 10_000;
const sleepSyncBuffer = new Int32Array(new SharedArrayBuffer(4));
function sleepSync(durationMs: number): void {
Atomics.wait(sleepSyncBuffer, 0, 0, durationMs);
}
function withWorktreePortRegistryLock<T>(homeDir: string, run: () => T): T {
fs.mkdirSync(homeDir, { recursive: true });
const lockPath = path.resolve(homeDir, WORKTREE_PORT_REGISTRY_LOCK_DIR);
const deadline = Date.now() + WORKTREE_PORT_REGISTRY_LOCK_TIMEOUT_MS;
while (true) {
try {
fs.mkdirSync(lockPath);
break;
} catch (error) {
const code = error instanceof Error && "code" in error ? error.code : null;
if (code !== "EEXIST") throw error;
try {
const ageMs = Date.now() - fs.statSync(lockPath).mtimeMs;
if (ageMs > WORKTREE_PORT_REGISTRY_LOCK_STALE_MS) {
fs.rmSync(lockPath, { recursive: true, force: true });
continue;
}
} catch {
continue;
}
if (Date.now() >= deadline) {
throw new Error(`Timed out waiting for worktree port reservation lock at ${lockPath}`);
}
sleepSync(25);
}
}
try {
return run();
} finally {
fs.rmSync(lockPath, { recursive: true, force: true });
}
}
function readWorktreePortRegistry(homeDir: string): Set<string> {
const registryPath = path.resolve(homeDir, WORKTREE_PORT_REGISTRY_FILE);
if (!fs.existsSync(registryPath)) return new Set();
try {
const parsed = JSON.parse(fs.readFileSync(registryPath, "utf8")) as Partial<WorktreePortRegistry>;
if (parsed.version !== 1 || !Array.isArray(parsed.configPaths)) return new Set();
return new Set(
parsed.configPaths
.filter((configPath): configPath is string => typeof configPath === "string" && configPath.length > 0)
.map((configPath) => path.resolve(configPath)),
);
} catch {
return new Set();
}
}
function writeWorktreePortRegistry(homeDir: string, configPaths: Iterable<string>): void {
const registryPath = path.resolve(homeDir, WORKTREE_PORT_REGISTRY_FILE);
const persistedPaths = Array.from(new Set(Array.from(configPaths, (configPath) => path.resolve(configPath))))
.filter((configPath) => fs.existsSync(configPath))
.sort();
const registry: WorktreePortRegistry = {
version: 1,
configPaths: persistedPaths,
};
const temporaryPath = `${registryPath}.${process.pid}.tmp`;
fs.writeFileSync(temporaryPath, `${JSON.stringify(registry, null, 2)}\n`, { mode: 0o600 });
fs.renameSync(temporaryPath, registryPath);
}
function resolveWorktreeRuntimeContext(
env: NodeJS.ProcessEnv,
overrideConfigPath?: string,
@@ -354,7 +276,6 @@ function collectSiblingWorktreePorts(
serverPorts.add(siblingConfig.server.port);
}
if (
siblingConfig.database.mode === "embedded-postgres" &&
Number.isInteger(siblingConfig.database.embeddedPostgresPort) &&
siblingConfig.database.embeddedPostgresPort > 0
) {
@@ -542,7 +463,7 @@ export function maybeRepairLegacyWorktreeConfigAndEnvFiles(): {
let repairedConfig = false;
if (fs.existsSync(context.configPath)) {
try {
const runtimeConfig = withWorktreePortRegistryLock(context.homeDir, () => {
const runtimeConfig = withWorktreePortRegistryLockSync(context.homeDir, () => {
const parsed = JSON.parse(fs.readFileSync(context.configPath, "utf8")) as PaperclipConfig;
let selectedConfig = parsed;
const registeredConfigPaths = readWorktreePortRegistry(context.homeDir);
@@ -671,4 +592,3 @@ export function maybePersistWorktreeRuntimePorts(input: {
writeConfigFile(context.configPath, config);
}
}