mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 17:33:51 +02:00
fix(workspaces): make managed runtimes reliable across restarts (#11740)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Execution workspaces need isolated databases, ports, and runtime services > - Concurrent workspaces could reuse ports or lose service ownership after a restart > - A markerless worktree also needed seed recovery, but normal markerless instances still needed to boot > - This pull request makes seed, port, and service ownership state explicit and recoverable > - It also checks live process and listener identity before it reclaims shared resources > - The benefit is reliable workspace startup, restart, adoption, and concurrent provisioning ## Linked Issues or Issue Description **What happened?** Managed workspaces could lose runtime service ownership after a control-plane restart. Concurrent worktrees could also reuse a port when their parent paths differed. A seed recovery change made every markerless instance resolve a worktree seed source, so normal instances without a source could not start. **Expected behavior** Paperclip must preserve healthy managed services across restarts. It must reserve unique ports across worktree parents. It must provision a registered markerless worktree, but it must skip seed work for a normal markerless instance. **Steps to reproduce** 1. Start two managed worktrees under different parent paths at the same time. 2. Restart the control plane while a managed service stays alive. 3. Start Paperclip with a config that has no seed markers and no registered worktree source. 4. Observe duplicate port selection, lost service adoption, or a seed-source startup error. **Paperclip version or commit** Current `master` plus the workspace runtime reliability changes in this pull request. **Deployment mode** Local development with managed execution workspaces and embedded Postgres. ## What Changed - Added a shared port registry with lease heartbeats, process identity checks, and live listener probes. - Reserved worktree ports across custom parent paths and repaired duplicate legacy assignments. - Preserved and adopted healthy managed services across control-plane restarts. - Reconciled guest bind modes and verified listener ownership before termination or reuse. - Provisioned registered markerless worktree databases and kept normal markerless instance startup as a no-op. - Added CLI, shared, server, and shell regression tests for seed, port, listener, restart, and adoption behavior. - Updated the worktree development documentation. ## Verification - `pnpm exec vitest run cli/src/__tests__/worktree.test.ts --reporter=verbose` — 63 tests passed. - `pnpm exec vitest run packages/shared/src/worktree-port-registry.test.ts --reporter=verbose` — 5 tests passed. - Focused runtime Vitest set — 199 tests passed across 37 suites. - `node --test scripts/__tests__/provision-worktree-self-heal.test.mjs` — 10 tests passed. - `git diff --check` passed. ## Risks - Port reservation now depends on lease and process identity data. The fallback listener probe prevents early reclamation when process metadata is incomplete. - Runtime adoption is stricter about bind and owner identity. The tests cover healthy adoption, stale records, PID reuse, and unrelated listeners. - Markerless seed detection now separates registered worktrees from normal instances. The tests cover both paths. - There are no database schema migrations. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex with the `gpt-5` model family. The serving snapshot and context-window size are not exposed. The agent used reasoning, repository tools, code execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Dev Agent <dev@paperclip.ing>
This commit is contained in:
21 files changed
+2406
-345
No files matched your search
@@ -0,0 +1,155 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import fs from "node:fs/promises";
|
||||
import net from "node:net";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
resetRuntimeServicesForTests,
|
||||
startRuntimeServicesForWorkspaceControl,
|
||||
} from "../services/workspace-runtime.js";
|
||||
import {
|
||||
doesLocalServiceCommandLineMatch,
|
||||
listLocalServiceRegistryRecords,
|
||||
readLocalServicePortOwner,
|
||||
resolveLocalServiceLogPath,
|
||||
terminateLocalService,
|
||||
} from "../services/local-service-supervisor.js";
|
||||
|
||||
describe("local service supervision", () => {
|
||||
afterEach(async () => {
|
||||
await resetRuntimeServicesForTests({ terminateProcesses: true });
|
||||
});
|
||||
|
||||
it("keeps request-logging runtime stdio usable after the supervisor side closes", async () => {
|
||||
const workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-service-stdio-"));
|
||||
const paperclipHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-service-home-"));
|
||||
const previousPaperclipHome = process.env.PAPERCLIP_HOME;
|
||||
const previousInstanceId = process.env.PAPERCLIP_INSTANCE_ID;
|
||||
process.env.PAPERCLIP_HOME = paperclipHome;
|
||||
process.env.PAPERCLIP_INSTANCE_ID = `service-stdio-${randomUUID()}`;
|
||||
|
||||
let registryRecord: Awaited<ReturnType<typeof listLocalServiceRegistryRecords>>[number] | null = null;
|
||||
try {
|
||||
const [service] = await startRuntimeServicesForWorkspaceControl({
|
||||
actor: { id: null, name: "Board", companyId: randomUUID() },
|
||||
issue: null,
|
||||
workspace: {
|
||||
baseCwd: workspaceRoot,
|
||||
source: "agent_home",
|
||||
projectId: null,
|
||||
workspaceId: null,
|
||||
repoUrl: null,
|
||||
repoRef: null,
|
||||
strategy: "project_primary",
|
||||
cwd: workspaceRoot,
|
||||
branchName: null,
|
||||
worktreePath: null,
|
||||
warnings: [],
|
||||
created: false,
|
||||
},
|
||||
config: {
|
||||
workspaceRuntime: {
|
||||
services: [{
|
||||
name: "web",
|
||||
command: "node -e \"const http=require('node:http'); process.on('SIGTERM',()=>{}); http.createServer((req,res)=>{ process.stdout.write('request '+req.url+'\\\\n',(error)=>{ if (!error) res.end('ok'); }); }).listen(Number(process.env.PORT), '127.0.0.1')\"",
|
||||
port: { type: "auto" },
|
||||
readiness: {
|
||||
type: "http",
|
||||
urlTemplate: "http://127.0.0.1:{{port}}",
|
||||
timeoutSec: 10,
|
||||
intervalMs: 100,
|
||||
},
|
||||
lifecycle: "shared",
|
||||
reuseScope: "agent",
|
||||
stopPolicy: { type: "manual" },
|
||||
}],
|
||||
},
|
||||
},
|
||||
adapterEnv: {},
|
||||
});
|
||||
|
||||
expect(service?.url).toMatch(/^http:\/\/127\.0\.0\.1:\d+$/);
|
||||
expect(service?.port).toBeTypeOf("number");
|
||||
registryRecord = (await listLocalServiceRegistryRecords({ profileKind: "workspace-runtime" }))
|
||||
.find((record) => record.runtimeServiceId === service!.id) ?? null;
|
||||
expect(registryRecord).not.toBeNull();
|
||||
|
||||
await resetRuntimeServicesForTests({ simulateSupervisorExit: true });
|
||||
|
||||
await expect(fetch(`${service!.url}/after-restart`)).resolves.toMatchObject({ ok: true });
|
||||
const log = await fs.readFile(resolveLocalServiceLogPath(registryRecord!.serviceKey), "utf8");
|
||||
expect(log).toContain("request /after-restart");
|
||||
|
||||
await terminateLocalService(registryRecord!);
|
||||
expect(await readLocalServicePortOwner(service!.port!)).toBeNull();
|
||||
await expect(fetch(service!.url!)).rejects.toThrow();
|
||||
registryRecord = null;
|
||||
} finally {
|
||||
if (registryRecord) await terminateLocalService(registryRecord).catch(() => undefined);
|
||||
if (previousPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME;
|
||||
else process.env.PAPERCLIP_HOME = previousPaperclipHome;
|
||||
if (previousInstanceId === undefined) delete process.env.PAPERCLIP_INSTANCE_ID;
|
||||
else process.env.PAPERCLIP_INSTANCE_ID = previousInstanceId;
|
||||
await fs.rm(paperclipHome, { recursive: true, force: true });
|
||||
await fs.rm(workspaceRoot, { recursive: true, force: true });
|
||||
}
|
||||
}, 15_000);
|
||||
|
||||
it("does not wait for an unrelated process that takes over the service port", async () => {
|
||||
if (process.platform === "win32") return;
|
||||
const unrelatedListener = net.createServer();
|
||||
await new Promise<void>((resolve) => unrelatedListener.listen(0, "127.0.0.1", resolve));
|
||||
const address = unrelatedListener.address();
|
||||
const port = typeof address === "object" && address ? address.port : null;
|
||||
if (!port) throw new Error("Failed to allocate unrelated listener port");
|
||||
const child = spawn(process.execPath, ["-e", "setInterval(()=>{},1000)"], {
|
||||
detached: true,
|
||||
stdio: "ignore",
|
||||
});
|
||||
child.unref();
|
||||
|
||||
try {
|
||||
await terminateLocalService({
|
||||
pid: child.pid!,
|
||||
processGroupId: child.pid!,
|
||||
port,
|
||||
});
|
||||
expect(await readLocalServicePortOwner(port)).toBe(process.pid);
|
||||
} finally {
|
||||
try {
|
||||
process.kill(-child.pid!, "SIGKILL");
|
||||
} catch {
|
||||
// The target process group was already terminated.
|
||||
}
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
unrelatedListener.close((error) => error ? reject(error) : resolve());
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
it("recognizes a pnpm command after the launcher becomes pnpm.cjs", () => {
|
||||
expect(doesLocalServiceCommandLineMatch({
|
||||
commandLine: "/usr/bin/node /opt/pnpm/pnpm.cjs dev -- --bind custom --bind-host 127.0.0.1",
|
||||
recordedCommand: "pnpm dev -- --bind custom --bind-host 127.0.0.1",
|
||||
serviceName: "paperclip-dev",
|
||||
})).toBe(true);
|
||||
});
|
||||
|
||||
it("does not accept a different command merely because it uses node", () => {
|
||||
expect(doesLocalServiceCommandLineMatch({
|
||||
commandLine: "/usr/bin/node /workspace/server/dist/index.js",
|
||||
recordedCommand: "pnpm dev -- --bind custom --bind-host 127.0.0.1",
|
||||
serviceName: "paperclip-dev",
|
||||
})).toBe(false);
|
||||
});
|
||||
|
||||
it("does not collapse different script paths to the same basename", () => {
|
||||
expect(doesLocalServiceCommandLineMatch({
|
||||
commandLine: "/usr/bin/node /other-workspace/server.js",
|
||||
recordedCommand: "node ./server.js",
|
||||
serviceName: "web",
|
||||
})).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -448,8 +448,6 @@ describe("resolveRuntimeProvisionCommand", () => {
|
||||
path.join(baseCwd, "scripts", "provision-worktree-runtime.sh"),
|
||||
"#!/usr/bin/env bash\n",
|
||||
);
|
||||
await fs.mkdir(path.join(cwd, ".paperclip"), { recursive: true });
|
||||
await fs.writeFile(path.join(cwd, ".paperclip", "seed-pending"), "{}\n");
|
||||
const workspace = {
|
||||
...buildWorkspace(cwd),
|
||||
baseCwd,
|
||||
@@ -457,6 +455,17 @@ describe("resolveRuntimeProvisionCommand", () => {
|
||||
worktreePath: cwd,
|
||||
};
|
||||
|
||||
expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe(
|
||||
"bash ./scripts/provision-worktree-runtime.sh",
|
||||
);
|
||||
|
||||
await fs.mkdir(path.join(cwd, ".paperclip"), { recursive: true });
|
||||
await fs.writeFile(path.join(cwd, ".paperclip", "config.json"), "{}\n");
|
||||
expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe(
|
||||
"bash ./scripts/provision-worktree-runtime.sh",
|
||||
);
|
||||
|
||||
await fs.writeFile(path.join(cwd, ".paperclip", "seed-pending"), "{}\n");
|
||||
expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe(
|
||||
"bash ./scripts/provision-worktree-runtime.sh",
|
||||
);
|
||||
@@ -466,7 +475,9 @@ describe("resolveRuntimeProvisionCommand", () => {
|
||||
})).toBe("./custom-provision.sh");
|
||||
|
||||
await fs.writeFile(path.join(cwd, ".paperclip", "seed-complete"), "{}\n");
|
||||
expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe("");
|
||||
expect(resolveRuntimeProvisionCommand({ config: {}, workspace })).toBe(
|
||||
"bash ./scripts/provision-worktree-runtime.sh",
|
||||
);
|
||||
|
||||
await fs.writeFile(
|
||||
path.join(cwd, ".paperclip", "seed-manifest.json"),
|
||||
@@ -772,6 +783,55 @@ describe("realizeExecutionWorkspace", () => {
|
||||
expect(second.branchName).toBe(first.branchName);
|
||||
});
|
||||
|
||||
it("defaults the repo-provided worktree provisioner for git worktree strategies", async () => {
|
||||
const repoRoot = await createTempRepo();
|
||||
await fs.mkdir(path.join(repoRoot, "scripts"), { recursive: true });
|
||||
await fs.writeFile(
|
||||
path.join(repoRoot, "scripts", "provision-worktree.sh"),
|
||||
[
|
||||
"#!/usr/bin/env bash",
|
||||
"set -euo pipefail",
|
||||
"mkdir -p .paperclip",
|
||||
"printf 'provisioned\\n' > .paperclip/default-provision-ran",
|
||||
"",
|
||||
].join("\n"),
|
||||
"utf8",
|
||||
);
|
||||
await runGit(repoRoot, ["add", "scripts/provision-worktree.sh"]);
|
||||
await runGit(repoRoot, ["commit", "-m", "Add repository worktree provisioner"]);
|
||||
|
||||
const workspace = await realizeExecutionWorkspace({
|
||||
base: {
|
||||
baseCwd: repoRoot,
|
||||
source: "project_primary",
|
||||
projectId: "project-1",
|
||||
workspaceId: "workspace-1",
|
||||
repoUrl: null,
|
||||
repoRef: "HEAD",
|
||||
},
|
||||
config: {
|
||||
workspaceStrategy: {
|
||||
type: "git_worktree",
|
||||
branchTemplate: "{{issue.identifier}}-{{slug}}",
|
||||
},
|
||||
},
|
||||
issue: {
|
||||
id: "issue-default-provision",
|
||||
identifier: "PAP-17684",
|
||||
title: "Default worktree provisioning",
|
||||
},
|
||||
agent: {
|
||||
id: "agent-1",
|
||||
name: "Codex Coder",
|
||||
companyId: "company-1",
|
||||
},
|
||||
});
|
||||
|
||||
await expect(
|
||||
fs.readFile(path.join(workspace.cwd, ".paperclip", "default-provision-ran"), "utf8"),
|
||||
).resolves.toBe("provisioned\n");
|
||||
});
|
||||
|
||||
it("warns when reusing a git worktree whose base ref has advanced", async () => {
|
||||
const repoRoot = await createTempRepo();
|
||||
|
||||
@@ -7230,7 +7290,7 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
|
||||
}
|
||||
}, 40_000);
|
||||
|
||||
it("adopts a live auto-port shared service after runtime state is reset", async () => {
|
||||
it("re-adopts a request-logging service on the same auto port after supervisor stdio closes", async () => {
|
||||
const workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-reconcile-"));
|
||||
const paperclipHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-home-"));
|
||||
process.env.PAPERCLIP_HOME = paperclipHome;
|
||||
@@ -7239,7 +7299,6 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
|
||||
const companyId = randomUUID();
|
||||
const agentId = randomUUID();
|
||||
const runId = randomUUID();
|
||||
const executionWorkspaceId = randomUUID();
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
@@ -7273,6 +7332,16 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
|
||||
projectId: null,
|
||||
workspaceId: null,
|
||||
};
|
||||
await fs.writeFile(
|
||||
path.join(workspaceRoot, "request-logger.cjs"),
|
||||
[
|
||||
"const http = require('node:http');",
|
||||
"http.createServer((req, res) => {",
|
||||
" process.stdout.write(`request ${req.url}\\n`, (error) => { if (!error) res.end('ok'); });",
|
||||
"}).listen(Number(process.env.PORT), '127.0.0.1');",
|
||||
].join("\n"),
|
||||
"utf8",
|
||||
);
|
||||
leasedRunIds.add(runId);
|
||||
|
||||
const services = await ensureRuntimeServicesForRun({
|
||||
@@ -7290,8 +7359,7 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
|
||||
services: [
|
||||
{
|
||||
name: "web",
|
||||
command:
|
||||
"node -e \"require('node:http').createServer((req,res)=>res.end('ok')).listen(Number(process.env.PORT), '127.0.0.1')\"",
|
||||
command: "node request-logger.cjs",
|
||||
port: { type: "auto" },
|
||||
readiness: {
|
||||
type: "http",
|
||||
@@ -7315,9 +7383,13 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
|
||||
const service = services[0];
|
||||
expect(service?.url).toMatch(/^http:\/\/127\.0\.0\.1:\d+$/);
|
||||
await expect(fetch(service!.url!)).resolves.toMatchObject({ ok: true });
|
||||
const originalPort = service!.port;
|
||||
const originalProviderRef = service!.providerRef;
|
||||
|
||||
await fs.rm(paperclipHome, { recursive: true, force: true });
|
||||
await resetRuntimeServicesForTests();
|
||||
// Closing the parent's pipe endpoints reproduces a real control-plane exit.
|
||||
// A service whose per-request logger still targets those pipes wedges (or
|
||||
// exits) on the reconciliation health probe instead of answering it.
|
||||
await resetRuntimeServicesForTests({ simulateSupervisorExit: true });
|
||||
|
||||
const result = await reconcilePersistedRuntimeServicesOnStartup(db);
|
||||
expect(result).toMatchObject({ reconciled: 1, adopted: 1, stopped: 0 });
|
||||
@@ -7328,17 +7400,221 @@ describeEmbeddedPostgres("workspace runtime startup reconciliation", () => {
|
||||
.where(eq(workspaceRuntimeServices.id, service!.id))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(persisted?.status).toBe("running");
|
||||
expect(persisted?.providerRef).toMatch(/^\d+$/);
|
||||
expect(persisted?.port).toBe(originalPort);
|
||||
expect(persisted?.providerRef).toBe(originalProviderRef);
|
||||
await expect(fetch(service!.url!)).resolves.toMatchObject({ ok: true });
|
||||
|
||||
await stopRuntimeServicesForExecutionWorkspace({
|
||||
db,
|
||||
executionWorkspaceId,
|
||||
workspaceCwd: workspace.cwd,
|
||||
});
|
||||
await resetRuntimeServicesForTests({ terminateProcesses: true });
|
||||
leasedRunIds.delete(runId);
|
||||
await fs.rm(paperclipHome, { recursive: true, force: true });
|
||||
await fs.rm(workspaceRoot, { recursive: true, force: true });
|
||||
|
||||
await expect(fetch(service!.url!)).rejects.toThrow();
|
||||
});
|
||||
|
||||
it("re-adopts a desired service when pnpm is represented as the pnpm.cjs launcher", async () => {
|
||||
const workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-pnpm-reconcile-"));
|
||||
const paperclipHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-home-"));
|
||||
const previousPaperclipHome = process.env.PAPERCLIP_HOME;
|
||||
const previousInstanceId = process.env.PAPERCLIP_INSTANCE_ID;
|
||||
process.env.PAPERCLIP_HOME = paperclipHome;
|
||||
process.env.PAPERCLIP_INSTANCE_ID = `runtime-pnpm-reconcile-${randomUUID()}`;
|
||||
|
||||
const portProbe = net.createServer();
|
||||
await new Promise<void>((resolve) => portProbe.listen(0, "127.0.0.1", resolve));
|
||||
const address = portProbe.address();
|
||||
const port = typeof address === "object" && address ? address.port : null;
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
portProbe.close((error) => error ? reject(error) : resolve());
|
||||
});
|
||||
if (!port) throw new Error("Failed to reserve pnpm reconciliation test port");
|
||||
|
||||
const companyId = randomUUID();
|
||||
const projectId = randomUUID();
|
||||
const executionWorkspaceId = randomUUID();
|
||||
const runtimeServiceId = randomUUID();
|
||||
const command = "pnpm dev";
|
||||
const service = {
|
||||
name: "web",
|
||||
command,
|
||||
port,
|
||||
readiness: {
|
||||
type: "http",
|
||||
urlTemplate: "http://127.0.0.1:{{port}}",
|
||||
timeoutSec: 10,
|
||||
intervalMs: 50,
|
||||
},
|
||||
expose: null,
|
||||
lifecycle: "shared",
|
||||
reuseScope: "execution_workspace",
|
||||
stopPolicy: { type: "manual" },
|
||||
};
|
||||
const reuseKey = createHash("sha256")
|
||||
.update(stableStringifyForTest({
|
||||
scopeType: "execution_workspace",
|
||||
scopeId: executionWorkspaceId,
|
||||
serviceName: service.name,
|
||||
command,
|
||||
cwd: workspaceRoot,
|
||||
port,
|
||||
env: {},
|
||||
expose: null,
|
||||
}))
|
||||
.digest("hex");
|
||||
const wrapperPath = path.join(workspaceRoot, "pnpm.cjs");
|
||||
await fs.writeFile(
|
||||
wrapperPath,
|
||||
[
|
||||
"const http = require('node:http');",
|
||||
"const port = Number(process.argv[3]);",
|
||||
"http.createServer((_req, res) => res.end('ok')).listen(port, '127.0.0.1');",
|
||||
].join("\n"),
|
||||
"utf8",
|
||||
);
|
||||
const child = spawn(process.execPath, [wrapperPath, "dev", String(port)], {
|
||||
cwd: workspaceRoot,
|
||||
detached: process.platform !== "win32",
|
||||
stdio: "ignore",
|
||||
});
|
||||
child.unref();
|
||||
|
||||
try {
|
||||
for (let attempt = 0; attempt < 50; attempt += 1) {
|
||||
try {
|
||||
if ((await fetch(`http://127.0.0.1:${port}`)).ok) break;
|
||||
} catch {
|
||||
// Wait for the simulated package-manager launcher to bind.
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, 50));
|
||||
}
|
||||
await expect(fetch(`http://127.0.0.1:${port}`)).resolves.toMatchObject({ ok: true });
|
||||
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Paperclip",
|
||||
issuePrefix: `T${companyId.replace(/-/g, "").slice(0, 6).toUpperCase()}`,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
await db.insert(projects).values({
|
||||
id: projectId,
|
||||
companyId,
|
||||
name: "Runtime pnpm reconciliation",
|
||||
status: "in_progress",
|
||||
});
|
||||
await db.insert(executionWorkspaces).values({
|
||||
id: executionWorkspaceId,
|
||||
companyId,
|
||||
projectId,
|
||||
projectWorkspaceId: null,
|
||||
mode: "isolated_workspace",
|
||||
strategyType: "git_worktree",
|
||||
name: "Runtime pnpm reconciliation workspace",
|
||||
status: "active",
|
||||
cwd: workspaceRoot,
|
||||
providerType: "local_fs",
|
||||
providerRef: workspaceRoot,
|
||||
metadata: {
|
||||
config: {
|
||||
workspaceRuntime: { services: [service] },
|
||||
desiredState: "running",
|
||||
serviceStates: { "0": "running" },
|
||||
},
|
||||
},
|
||||
});
|
||||
await db.insert(workspaceRuntimeServices).values({
|
||||
id: runtimeServiceId,
|
||||
companyId,
|
||||
projectId,
|
||||
projectWorkspaceId: null,
|
||||
executionWorkspaceId,
|
||||
issueId: null,
|
||||
scopeType: "execution_workspace",
|
||||
scopeId: executionWorkspaceId,
|
||||
serviceName: service.name,
|
||||
status: "running",
|
||||
lifecycle: "shared",
|
||||
reuseKey,
|
||||
command,
|
||||
cwd: workspaceRoot,
|
||||
port,
|
||||
url: `http://127.0.0.1:${port}`,
|
||||
provider: "local_process",
|
||||
providerRef: String(child.pid),
|
||||
ownerAgentId: null,
|
||||
startedByRunId: null,
|
||||
lastUsedAt: new Date(),
|
||||
startedAt: new Date(),
|
||||
stoppedAt: null,
|
||||
stopPolicy: { type: "manual" },
|
||||
healthStatus: "healthy",
|
||||
});
|
||||
await writeLocalServiceRegistryRecord({
|
||||
version: 1,
|
||||
serviceKey: `workspace-runtime-web-${randomUUID()}`,
|
||||
profileKind: "workspace-runtime",
|
||||
serviceName: service.name,
|
||||
command,
|
||||
cwd: workspaceRoot,
|
||||
envFingerprint: reuseKey,
|
||||
port,
|
||||
url: `http://127.0.0.1:${port}`,
|
||||
pid: child.pid!,
|
||||
processGroupId: child.pid ?? null,
|
||||
provider: "local_process",
|
||||
runtimeServiceId,
|
||||
reuseKey,
|
||||
startedAt: new Date().toISOString(),
|
||||
lastSeenAt: new Date().toISOString(),
|
||||
metadata: { executionWorkspaceId },
|
||||
});
|
||||
|
||||
const result = await reconcilePersistedRuntimeServicesOnStartup(db);
|
||||
|
||||
expect(result).toMatchObject({
|
||||
reconciled: 1,
|
||||
adopted: 1,
|
||||
stopped: 0,
|
||||
restarted: 0,
|
||||
restartFailed: 0,
|
||||
});
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(workspaceRuntimeServices)
|
||||
.where(eq(workspaceRuntimeServices.executionWorkspaceId, executionWorkspaceId));
|
||||
expect(rows).toHaveLength(1);
|
||||
expect(rows[0]).toMatchObject({
|
||||
id: runtimeServiceId,
|
||||
status: "running",
|
||||
healthStatus: "healthy",
|
||||
port,
|
||||
providerRef: String(child.pid),
|
||||
});
|
||||
expect(await readLocalServicePortOwner(port)).toBe(child.pid);
|
||||
await expect(fetch(`http://127.0.0.1:${port}`)).resolves.toMatchObject({ ok: true });
|
||||
} finally {
|
||||
await stopRuntimeServicesForExecutionWorkspace({
|
||||
db,
|
||||
executionWorkspaceId,
|
||||
workspaceCwd: workspaceRoot,
|
||||
}).catch(() => undefined);
|
||||
if (child.pid) {
|
||||
try {
|
||||
if (process.platform === "win32") process.kill(child.pid, "SIGKILL");
|
||||
else process.kill(-child.pid, "SIGKILL");
|
||||
} catch {
|
||||
// The adopted runtime was already stopped through the managed path.
|
||||
}
|
||||
}
|
||||
await resetRuntimeServicesForTests();
|
||||
if (previousPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME;
|
||||
else process.env.PAPERCLIP_HOME = previousPaperclipHome;
|
||||
if (previousInstanceId === undefined) delete process.env.PAPERCLIP_INSTANCE_ID;
|
||||
else process.env.PAPERCLIP_INSTANCE_ID = previousInstanceId;
|
||||
await fs.rm(paperclipHome, { recursive: true, force: true });
|
||||
await fs.rm(workspaceRoot, { recursive: true, force: true });
|
||||
}
|
||||
}, 20_000);
|
||||
|
||||
it("does not reuse a stopped auto-port service port while another process owns it", async () => {
|
||||
const workspaceRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-unhealthy-adopt-"));
|
||||
const paperclipHome = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-runtime-home-"));
|
||||
|
||||
@@ -444,13 +444,27 @@ describe("worktree config repair", () => {
|
||||
const firstConfigPath = path.join(firstWorktreeRoot, ".paperclip", "config.json");
|
||||
const secondConfigPath = path.join(secondWorktreeRoot, ".paperclip", "config.json");
|
||||
|
||||
const writeWorktree = async (worktreeRoot: string, name: string) => {
|
||||
const writeWorktree = async (
|
||||
worktreeRoot: string,
|
||||
name: string,
|
||||
databaseMode: "embedded-postgres" | "postgres" = "embedded-postgres",
|
||||
) => {
|
||||
const paperclipDir = path.join(worktreeRoot, ".paperclip");
|
||||
const instanceRoot = path.join(isolatedHome, "instances", name.toLowerCase());
|
||||
const config = buildIsolatedConfig(instanceRoot, 45439, 55439);
|
||||
await fs.mkdir(paperclipDir, { recursive: true });
|
||||
await fs.writeFile(
|
||||
path.join(paperclipDir, "config.json"),
|
||||
`${JSON.stringify(buildIsolatedConfig(instanceRoot, 45439, 55439), null, 2)}\n`,
|
||||
`${JSON.stringify({
|
||||
...config,
|
||||
database: {
|
||||
...config.database,
|
||||
mode: databaseMode,
|
||||
...(databaseMode === "postgres"
|
||||
? { connectionString: "postgres://paperclip:paperclip@127.0.0.1:55439/paperclip" }
|
||||
: {}),
|
||||
},
|
||||
}, null, 2)}\n`,
|
||||
"utf8",
|
||||
);
|
||||
await fs.writeFile(
|
||||
@@ -480,7 +494,7 @@ describe("worktree config repair", () => {
|
||||
delete process.env.DATABASE_URL;
|
||||
};
|
||||
|
||||
await writeWorktree(firstWorktreeRoot, "PAP-14013-import-bulk-skills");
|
||||
await writeWorktree(firstWorktreeRoot, "PAP-14013-import-bulk-skills", "postgres");
|
||||
await writeWorktree(secondWorktreeRoot, "PAP-14069-port-conflicts");
|
||||
const staleLockPath = path.join(isolatedHome, ".worktree-port-reservations.lock");
|
||||
await fs.mkdir(staleLockPath, { recursive: true });
|
||||
|
||||
@@ -0,0 +1,229 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { promisify } from "node:util";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
ensureWorktreeSeeded,
|
||||
readWorktreeSeedManifest,
|
||||
} from "../../../cli/src/commands/worktree.ts";
|
||||
import { realizeExecutionWorkspace } from "../services/workspace-runtime.ts";
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
const cleanup: string[] = [];
|
||||
const originalConfig = process.env.PAPERCLIP_CONFIG;
|
||||
const originalWorktreesDir = process.env.PAPERCLIP_WORKTREES_DIR;
|
||||
|
||||
async function runGit(cwd: string, args: string[]) {
|
||||
await execFileAsync("git", args, { cwd });
|
||||
}
|
||||
|
||||
async function writeConfig(configPath: string, instanceId: string) {
|
||||
const instanceRoot = path.dirname(configPath);
|
||||
await fs.mkdir(path.dirname(configPath), { recursive: true });
|
||||
await fs.writeFile(configPath, `${JSON.stringify({
|
||||
$meta: { version: 1, updatedAt: "2026-08-19T00:00:00.000Z", source: "configure" },
|
||||
database: {
|
||||
mode: "embedded-postgres",
|
||||
embeddedPostgresDataDir: path.join(instanceRoot, "db"),
|
||||
embeddedPostgresPort: 54329,
|
||||
backup: { enabled: false, intervalMinutes: 60, retentionDays: 30, dir: path.join(instanceRoot, "backups") },
|
||||
},
|
||||
logging: { mode: "file", logDir: path.join(instanceRoot, "logs") },
|
||||
server: {
|
||||
deploymentMode: "local_trusted",
|
||||
exposure: "private",
|
||||
host: "127.0.0.1",
|
||||
port: 3100,
|
||||
allowedHostnames: [],
|
||||
serveUi: true,
|
||||
},
|
||||
}, null, 2)}\n`, "utf8");
|
||||
await fs.writeFile(
|
||||
path.join(path.dirname(configPath), ".env"),
|
||||
`PAPERCLIP_INSTANCE_ID=${instanceId}\n`,
|
||||
"utf8",
|
||||
);
|
||||
}
|
||||
|
||||
function verifiedSeedResult() {
|
||||
return {
|
||||
backupSummary: "snapshot.sql",
|
||||
snapshotAt: "2026-08-19T00:00:00.000Z",
|
||||
migrationRevision: "0223_test.sql",
|
||||
pausedScheduledRoutines: 0,
|
||||
executionQuarantine: {
|
||||
disabledTimerHeartbeats: 0,
|
||||
resetRunningAgents: 0,
|
||||
quarantinedInProgressIssues: 0,
|
||||
unassignedTodoIssues: 0,
|
||||
unassignedReviewIssues: 0,
|
||||
stoppedProjectWorkspaceRuntimes: 0,
|
||||
stoppedExecutionWorkspaceRuntimes: 0,
|
||||
stoppedRuntimeServices: 0,
|
||||
},
|
||||
reboundWorkspaces: [],
|
||||
validation: {
|
||||
authUserCount: 1,
|
||||
credentialAccountCount: 1,
|
||||
instanceAdminCount: 1,
|
||||
activeMembershipCount: 1,
|
||||
companyCount: 1,
|
||||
issueCount: 1,
|
||||
representativeCompanyId: "00000000-0000-4000-8000-000000000001",
|
||||
representativeIssueId: "00000000-0000-4000-8000-000000000002",
|
||||
migrationRevision: "0223_test.sql",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
if (originalConfig === undefined) delete process.env.PAPERCLIP_CONFIG;
|
||||
else process.env.PAPERCLIP_CONFIG = originalConfig;
|
||||
if (originalWorktreesDir === undefined) delete process.env.PAPERCLIP_WORKTREES_DIR;
|
||||
else process.env.PAPERCLIP_WORKTREES_DIR = originalWorktreesDir;
|
||||
for (const dir of cleanup.splice(0)) {
|
||||
await fs.rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
describe("managed worktree seed source through the server spawn path", () => {
|
||||
it("re-derives an ambient-instance manifest written before provisioning", async () => {
|
||||
const tempRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-server-seed-source-"));
|
||||
cleanup.push(tempRoot);
|
||||
const repoRoot = path.join(tempRoot, "repo");
|
||||
const hooksDir = path.join(tempRoot, "hooks");
|
||||
const ambientConfigPath = path.join(tempRoot, "ambient", "config.json");
|
||||
const registeredConfigPath = path.join(repoRoot, ".paperclip", "config.json");
|
||||
const worktreeHome = path.join(tempRoot, "worktree-home");
|
||||
|
||||
await fs.mkdir(repoRoot, { recursive: true });
|
||||
await runGit(repoRoot, ["init", "-q"]);
|
||||
await runGit(repoRoot, ["config", "user.email", "paperclip@example.com"]);
|
||||
await runGit(repoRoot, ["config", "user.name", "Paperclip Test"]);
|
||||
await fs.mkdir(path.join(repoRoot, "scripts"), { recursive: true });
|
||||
await fs.writeFile(path.join(repoRoot, "README.md"), "server spawn regression\n", "utf8");
|
||||
await fs.copyFile(
|
||||
fileURLToPath(new URL("../../../scripts/provision-worktree.sh", import.meta.url)),
|
||||
path.join(repoRoot, "scripts", "provision-worktree.sh"),
|
||||
);
|
||||
await fs.chmod(path.join(repoRoot, "scripts", "provision-worktree.sh"), 0o755);
|
||||
await runGit(repoRoot, ["add", "README.md", "scripts/provision-worktree.sh"]);
|
||||
await runGit(repoRoot, ["commit", "-qm", "Add managed provision script"]);
|
||||
await writeConfig(registeredConfigPath, "registered-source");
|
||||
await writeConfig(ambientConfigPath, "ambient-instance");
|
||||
await fs.mkdir(worktreeHome, { recursive: true });
|
||||
|
||||
// Reproduce the integrated lane's ordering with a checkout hook: an earlier
|
||||
// worktree-init-style writer inherits the server environment and leaves a valid
|
||||
// target config whose manifest diagnostic points at the ambient instance.
|
||||
await fs.mkdir(hooksDir, { recursive: true });
|
||||
const hookPath = path.join(hooksDir, "post-checkout");
|
||||
await fs.writeFile(
|
||||
hookPath,
|
||||
`#!/usr/bin/env node
|
||||
const crypto = require("node:crypto");
|
||||
const fs = require("node:fs");
|
||||
const path = require("node:path");
|
||||
const cwd = process.cwd();
|
||||
if (cwd === ${JSON.stringify(repoRoot)}) process.exit(0);
|
||||
const stateDir = path.join(cwd, ".paperclip");
|
||||
const normalized = path.basename(cwd).trim().toLowerCase().replace(/[^a-z0-9_-]+/g, "-").replace(/-+/g, "-").replace(/^[-_]+|[-_]+$/g, "");
|
||||
const instanceId = \`${"${(normalized || \"worktree\").slice(0, 48)}"}-\${crypto.createHash("sha256").update(path.resolve(cwd)).digest("hex").slice(0, 12)}\`;
|
||||
const targetConfigPath = path.join(stateDir, "config.json");
|
||||
const instanceRoot = path.join(process.env.PAPERCLIP_WORKTREES_DIR, "instances", instanceId);
|
||||
fs.mkdirSync(stateDir, { recursive: true });
|
||||
fs.mkdirSync(instanceRoot, { recursive: true });
|
||||
fs.writeFileSync(targetConfigPath, JSON.stringify({
|
||||
$meta: { version: 1, updatedAt: "2026-08-19T00:00:00.000Z", source: "configure" },
|
||||
database: {
|
||||
mode: "embedded-postgres",
|
||||
embeddedPostgresDataDir: path.join(instanceRoot, "db"),
|
||||
embeddedPostgresPort: 54330,
|
||||
backup: { enabled: false, intervalMinutes: 60, retentionDays: 30, dir: path.join(instanceRoot, "backups") },
|
||||
},
|
||||
logging: { mode: "file", logDir: path.join(instanceRoot, "logs") },
|
||||
server: {
|
||||
deploymentMode: "local_trusted",
|
||||
exposure: "private",
|
||||
host: "127.0.0.1",
|
||||
port: 3101,
|
||||
allowedHostnames: [],
|
||||
serveUi: true,
|
||||
},
|
||||
}, null, 2) + "\\n");
|
||||
fs.writeFileSync(path.join(stateDir, ".env"), [
|
||||
"PAPERCLIP_HOME=" + JSON.stringify(process.env.PAPERCLIP_WORKTREES_DIR),
|
||||
"PAPERCLIP_INSTANCE_ID=" + JSON.stringify(instanceId),
|
||||
"PAPERCLIP_CONFIG=" + JSON.stringify(targetConfigPath),
|
||||
"",
|
||||
].join("\\n"));
|
||||
fs.writeFileSync(path.join(stateDir, "seed-manifest.json"), JSON.stringify({
|
||||
version: 2,
|
||||
source: { instanceId: "ambient-instance", configPath: process.env.PAPERCLIP_CONFIG },
|
||||
snapshotAt: null,
|
||||
seedMode: "minimal",
|
||||
migrationRevision: null,
|
||||
targetInstanceId: instanceId,
|
||||
phase: "pending",
|
||||
state: "pending",
|
||||
attemptId: "ambient-writer",
|
||||
startedAt: null,
|
||||
finishedAt: null,
|
||||
diagnostics: [{ phase: "pending", status: "succeeded", at: new Date().toISOString() }],
|
||||
}, null, 2) + "\\n");
|
||||
`,
|
||||
"utf8",
|
||||
);
|
||||
await fs.chmod(hookPath, 0o755);
|
||||
await runGit(repoRoot, ["config", "core.hooksPath", hooksDir]);
|
||||
|
||||
process.env.PAPERCLIP_CONFIG = ambientConfigPath;
|
||||
process.env.PAPERCLIP_WORKTREES_DIR = worktreeHome;
|
||||
const workspace = await realizeExecutionWorkspace({
|
||||
base: {
|
||||
baseCwd: repoRoot,
|
||||
source: "project_primary",
|
||||
projectId: "project-1",
|
||||
workspaceId: "project-workspace-1",
|
||||
repoUrl: null,
|
||||
repoRef: "HEAD",
|
||||
},
|
||||
config: {
|
||||
workspaceStrategy: {
|
||||
type: "git_worktree",
|
||||
branchTemplate: "{{issue.identifier}}-{{slug}}",
|
||||
provisionCommand: "bash ./scripts/provision-worktree.sh",
|
||||
},
|
||||
},
|
||||
issue: {
|
||||
id: "issue-1",
|
||||
identifier: "PAP-17681",
|
||||
title: "Recover ambient seed source",
|
||||
},
|
||||
agent: { id: "agent-1", name: "Coder", companyId: "company-1" },
|
||||
});
|
||||
|
||||
expect(readWorktreeSeedManifest(path.join(workspace.cwd, ".paperclip", "config.json"))).toMatchObject({
|
||||
source: { instanceId: "ambient-instance", configPath: ambientConfigPath },
|
||||
state: "pending",
|
||||
});
|
||||
|
||||
await expect(ensureWorktreeSeeded({
|
||||
config: path.join(workspace.cwd, ".paperclip", "config.json"),
|
||||
registeredBaseWorkspaceCwd: repoRoot,
|
||||
registeredProjectWorkspaceId: "project-workspace-1",
|
||||
expectedCompanyId: "company-1",
|
||||
}, {
|
||||
seedDatabase: async () => verifiedSeedResult(),
|
||||
})).resolves.toMatchObject({ seeded: true, reason: "seeded" });
|
||||
|
||||
expect(readWorktreeSeedManifest(path.join(workspace.cwd, ".paperclip", "config.json"))).toMatchObject({
|
||||
source: { instanceId: "registered-source", configPath: registeredConfigPath },
|
||||
state: "verified",
|
||||
phase: "complete",
|
||||
});
|
||||
}, 20_000);
|
||||
});
|
||||
@@ -1,5 +1,6 @@
|
||||
import { execFile } from "node:child_process";
|
||||
import { createHash } from "node:crypto";
|
||||
import fsSync from "node:fs";
|
||||
import fs from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
@@ -63,6 +64,32 @@ function getRuntimeServicesDir() {
|
||||
return path.resolve(resolvePaperclipInstanceRoot(), "runtime-services");
|
||||
}
|
||||
|
||||
function getRuntimeServiceLogsDir() {
|
||||
return path.resolve(resolvePaperclipInstanceRoot(), "runtime-service-logs");
|
||||
}
|
||||
|
||||
export function resolveLocalServiceLogPath(serviceKey: string) {
|
||||
if (!/^[a-z0-9._-]+$/.test(serviceKey)) {
|
||||
throw new Error("Invalid local service key for log path");
|
||||
}
|
||||
return path.resolve(getRuntimeServiceLogsDir(), `${serviceKey}.log`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Open a managed service's durable append-only output file.
|
||||
*
|
||||
* The returned descriptor is intended to be passed directly to spawn(). The
|
||||
* child receives its own duplicate, so the caller can close this handle as soon
|
||||
* as spawn returns without tying the service's stdio lifetime to Paperclip's.
|
||||
*/
|
||||
export async function openLocalServiceLogFile(serviceKey: string) {
|
||||
await fs.mkdir(getRuntimeServiceLogsDir(), { recursive: true });
|
||||
const logPath = resolveLocalServiceLogPath(serviceKey);
|
||||
const handle = await fs.open(logPath, "a+", 0o600);
|
||||
const startOffset = (await handle.stat()).size;
|
||||
return { handle, logPath, startOffset };
|
||||
}
|
||||
|
||||
function getRuntimeServiceRegistryPath(serviceKey: string) {
|
||||
return path.resolve(getRuntimeServicesDir(), `${serviceKey}.json`);
|
||||
}
|
||||
@@ -230,10 +257,91 @@ export function isProcessGroupAlive(processGroupId: number | null | undefined) {
|
||||
if (typeof processGroupId !== "number" || !Number.isInteger(processGroupId) || processGroupId <= 0) return false;
|
||||
try {
|
||||
process.kill(-processGroupId, 0);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (process.platform === "linux") {
|
||||
const liveMember = readLinuxProcessGroupActivity(processGroupId);
|
||||
if (liveMember !== null) return liveMember;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
function readLinuxProcessGroupActivity(processGroupId: number): boolean | null {
|
||||
let entries: fsSync.Dirent[];
|
||||
try {
|
||||
entries = fsSync.readdirSync("/proc", { withFileTypes: true });
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
|
||||
let foundMember = false;
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory() || !/^\d+$/.test(entry.name)) continue;
|
||||
try {
|
||||
const stat = fsSync.readFileSync(`/proc/${entry.name}/stat`, "utf8");
|
||||
const commandEnd = stat.lastIndexOf(")");
|
||||
if (commandEnd < 0) continue;
|
||||
const fields = stat.slice(commandEnd + 1).trim().split(/\s+/);
|
||||
const state = fields[0];
|
||||
const memberProcessGroupId = Number.parseInt(fields[2] ?? "", 10);
|
||||
if (memberProcessGroupId !== processGroupId) continue;
|
||||
foundMember = true;
|
||||
if (state !== "Z" && state !== "X") return true;
|
||||
} catch {
|
||||
// The process can exit while /proc is scanned.
|
||||
}
|
||||
}
|
||||
|
||||
// kill(-pgid, 0) also succeeds for a group that contains only zombies. Such
|
||||
// processes cannot run or own a listener and are waiting only for their
|
||||
// parent to reap them, so termination is complete for service-control use.
|
||||
return foundMember ? false : null;
|
||||
}
|
||||
|
||||
function tokenizeCommandLine(value: string) {
|
||||
return value.match(/"(?:\\.|[^"\\])*"|'[^']*'|\S+/g) ?? [];
|
||||
}
|
||||
|
||||
function normalizeCommandToken(value: string) {
|
||||
const unquoted = value.replace(/^["']|["']$/g, "");
|
||||
const basename = path.basename(unquoted.replace(/\\/g, "/"));
|
||||
const launcher = basename.replace(/\.(?:cjs|mjs|js|cmd|exe)$/i, "");
|
||||
return /^(?:bun|node|nodejs|npm|npx|pnpm|yarn)$/i.test(launcher) ? launcher : unquoted;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare a configured service command with the argv exposed by the OS.
|
||||
*
|
||||
* Package-manager launchers commonly replace `pnpm dev` with
|
||||
* `node /path/to/pnpm.cjs dev` after the shell starts. A literal substring
|
||||
* check rejects that surviving process even though the executable and all
|
||||
* configured arguments are still present. Normalize executable paths and
|
||||
* script extensions, then require the configured argv to remain contiguous.
|
||||
*/
|
||||
export function doesLocalServiceCommandLineMatch(input: {
|
||||
commandLine: string;
|
||||
recordedCommand: string;
|
||||
serviceName: string;
|
||||
}) {
|
||||
const normalize = (value: string) => value.replace(/["']/g, "").replace(/\s+/g, " ").trim();
|
||||
const normalizedCommandLine = normalize(input.commandLine);
|
||||
const normalizedRecordedCommand = normalize(input.recordedCommand);
|
||||
if (
|
||||
normalizedCommandLine.includes(normalizedRecordedCommand)
|
||||
|| normalizedCommandLine.includes(input.serviceName)
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
|
||||
const actualTokens = tokenizeCommandLine(input.commandLine).map(normalizeCommandToken);
|
||||
const recordedTokens = tokenizeCommandLine(input.recordedCommand).map(normalizeCommandToken);
|
||||
if (recordedTokens.length === 0 || recordedTokens.length > actualTokens.length) return false;
|
||||
|
||||
return actualTokens.some((_, start) => recordedTokens.every(
|
||||
(token, offset) => actualTokens[start + offset] === token,
|
||||
));
|
||||
}
|
||||
|
||||
async function isLikelyMatchingCommand(record: LocalServiceRegistryRecord) {
|
||||
@@ -242,10 +350,11 @@ async function isLikelyMatchingCommand(record: LocalServiceRegistryRecord) {
|
||||
const { stdout } = await execFileAsync("ps", ["-o", "command=", "-p", String(record.pid)]);
|
||||
const commandLine = stdout.trim();
|
||||
if (!commandLine) return false;
|
||||
const normalize = (value: string) => value.replace(/["']/g, "").replace(/\s+/g, " ").trim();
|
||||
const normalizedCommandLine = normalize(commandLine);
|
||||
const normalizedRecordedCommand = normalize(record.command);
|
||||
return normalizedCommandLine.includes(normalizedRecordedCommand) || normalizedCommandLine.includes(record.serviceName);
|
||||
return doesLocalServiceCommandLineMatch({
|
||||
commandLine,
|
||||
recordedCommand: record.command,
|
||||
serviceName: record.serviceName,
|
||||
});
|
||||
} catch {
|
||||
return true;
|
||||
}
|
||||
@@ -396,11 +505,34 @@ export async function touchLocalServiceRegistryRecord(
|
||||
}
|
||||
|
||||
export async function terminateLocalService(
|
||||
record: Pick<LocalServiceRegistryRecord, "pid" | "processGroupId">,
|
||||
opts?: { signal?: NodeJS.Signals; forceAfterMs?: number },
|
||||
record: Pick<LocalServiceRegistryRecord, "pid" | "processGroupId"> &
|
||||
Partial<Pick<LocalServiceRegistryRecord, "port">>,
|
||||
opts?: { signal?: NodeJS.Signals; forceAfterMs?: number; verifyAfterMs?: number },
|
||||
) {
|
||||
const signal = opts?.signal ?? "SIGTERM";
|
||||
const targetProcessGroup = process.platform !== "win32" && record.processGroupId && record.processGroupId > 0;
|
||||
|
||||
const targetIsGone = async () => {
|
||||
const targetAlive = targetProcessGroup
|
||||
? isProcessGroupAlive(record.processGroupId)
|
||||
: isPidAlive(record.pid);
|
||||
if (targetAlive) return false;
|
||||
if (!record.port) return true;
|
||||
const portOwnerPid = await readLocalServicePortOwner(record.port);
|
||||
if (!portOwnerPid) return true;
|
||||
const ownerProcessId = targetProcessGroup ? record.processGroupId! : record.pid;
|
||||
return !(await isLocalServiceProcessOwnedBy(portOwnerPid, ownerProcessId));
|
||||
};
|
||||
|
||||
const waitUntilGone = async (timeoutMs: number) => {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
do {
|
||||
if (await targetIsGone()) return true;
|
||||
await delay(100);
|
||||
} while (Date.now() < deadline);
|
||||
return await targetIsGone();
|
||||
};
|
||||
|
||||
try {
|
||||
if (targetProcessGroup) {
|
||||
process.kill(-record.processGroupId!, signal);
|
||||
@@ -408,24 +540,10 @@ export async function terminateLocalService(
|
||||
process.kill(record.pid, signal);
|
||||
}
|
||||
} catch {
|
||||
return;
|
||||
if (await targetIsGone()) return;
|
||||
}
|
||||
|
||||
const deadline = Date.now() + (opts?.forceAfterMs ?? 2_000);
|
||||
while (Date.now() < deadline) {
|
||||
const targetAlive = targetProcessGroup
|
||||
? isProcessGroupAlive(record.processGroupId)
|
||||
: isPidAlive(record.pid);
|
||||
if (!targetAlive) {
|
||||
return;
|
||||
}
|
||||
await delay(100);
|
||||
}
|
||||
|
||||
const stillAlive = targetProcessGroup
|
||||
? isProcessGroupAlive(record.processGroupId)
|
||||
: isPidAlive(record.pid);
|
||||
if (!stillAlive) return;
|
||||
if (await waitUntilGone(opts?.forceAfterMs ?? 2_000)) return;
|
||||
try {
|
||||
if (targetProcessGroup) {
|
||||
process.kill(-record.processGroupId!, "SIGKILL");
|
||||
@@ -435,6 +553,14 @@ export async function terminateLocalService(
|
||||
} catch {
|
||||
// Ignore cleanup races.
|
||||
}
|
||||
|
||||
if (await waitUntilGone(opts?.verifyAfterMs ?? 2_000)) return;
|
||||
|
||||
const target = targetProcessGroup
|
||||
? `process group ${record.processGroupId}`
|
||||
: `process ${record.pid}`;
|
||||
const listener = record.port ? ` and listener on port ${record.port}` : "";
|
||||
throw new Error(`Failed to terminate local service ${target}${listener}`);
|
||||
}
|
||||
|
||||
export async function readLocalServicePortOwner(port: number) {
|
||||
|
||||
@@ -162,7 +162,7 @@ describe("diagnoseRuntimeListenerBinds against live listeners", () => {
|
||||
expect(diagnosis).toContain(`port ${appPort}`);
|
||||
// Node's hostless listen is dual-stack, so /proc shows :: and/or 0.0.0.0.
|
||||
expect(diagnosis).toMatch(/0\.0\.0\.0|::/);
|
||||
expect(diagnosis).toContain("--bind custom --bind-host 127.0.0.1");
|
||||
expect(diagnosis).toContain("--bind loopback");
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -160,6 +160,6 @@ export async function diagnoseRuntimeListenerBinds(ports: number[]): Promise<str
|
||||
`${violations.join("; ")} instead of loopback only. The broker will not expose a listener `
|
||||
+ "reachable off-loopback. This means the workspace checkout's dev server ignored the managed "
|
||||
+ "loopback bind — a checkout that predates managed HTTPS exposure overwrites PAPERCLIP_BIND "
|
||||
+ "from its own --bind argv, so the start command must pass --bind custom --bind-host 127.0.0.1."
|
||||
+ "from its own --bind argv, so the start command must pass --bind loopback."
|
||||
);
|
||||
}
|
||||
@@ -120,6 +120,10 @@ beforeAll(async () => {
|
||||
await fs.writeFile(path.join(guestDir, "dev-runner.mjs"), PRE_MANAGED_EXPOSURE_GUEST);
|
||||
await fs.writeFile(path.join(guestDir, "dev-runner-legacy.mjs"), ALWAYS_WILDCARD_GUEST);
|
||||
await fs.writeFile(path.join(guestDir, "dev-runner-wildcard-hmr.mjs"), WILDCARD_HMR_GUEST);
|
||||
await fs.writeFile(
|
||||
path.join(guestDir, "dev-runner-bind-conflict.mjs"),
|
||||
'process.stderr.write("local_trusted requires server.bind=loopback\\n"); process.exit(1);\n',
|
||||
);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
@@ -474,7 +478,7 @@ describe("loopback bind is forced on the guest, not merely requested (PAP-17256)
|
||||
}));
|
||||
|
||||
// The launched command carries the loopback bind, replacing `--bind lan`.
|
||||
expect(runtime.command).toContain("--bind custom --bind-host 127.0.0.1");
|
||||
expect(runtime.command).toContain("--bind loopback");
|
||||
expect(runtime.command).not.toContain("--bind lan");
|
||||
|
||||
// And because the guest honoured it, both listeners are loopback-only, so
|
||||
@@ -530,7 +534,7 @@ describe("loopback bind is forced on the guest, not merely requested (PAP-17256)
|
||||
// The bare code alone is what cost PAP-17254 three diagnostic cycles.
|
||||
expect(error!.message).toContain("listener_ownership_mismatch");
|
||||
expect(error!.message).toContain("loopback");
|
||||
expect(error!.message).toContain("--bind custom --bind-host 127.0.0.1");
|
||||
expect(error!.message).toContain("--bind loopback");
|
||||
}, 20_000);
|
||||
|
||||
it("leaves a non-Paperclip service's --bind argument alone", async () => {
|
||||
@@ -547,7 +551,7 @@ describe("loopback bind is forced on the guest, not merely requested (PAP-17256)
|
||||
}));
|
||||
|
||||
expect(runtime.command).toBe(declared);
|
||||
expect(runtime.command).not.toContain("--bind custom");
|
||||
expect(runtime.command).not.toContain("--bind loopback");
|
||||
expect(calls.slice(0, 2)).toEqual(["reserve", "expose"]);
|
||||
expect(runtime.exposure?.state).toBe("ready");
|
||||
}, 20_000);
|
||||
@@ -567,6 +571,20 @@ describe("loopback bind is forced on the guest, not merely requested (PAP-17256)
|
||||
expect(calls).toEqual([]);
|
||||
expect(runtime.command).toBe(declared);
|
||||
}, 20_000);
|
||||
|
||||
it("surfaces a deployment/bind conflict from a guest that exits during startup", async () => {
|
||||
const { broker } = createBroker();
|
||||
installDeps({ broker });
|
||||
|
||||
await expect(startRuntimeServicesForWorkspaceControl(startInput({
|
||||
serviceName: "paperclip-dev",
|
||||
command: guestCommand("dev-runner-bind-conflict.mjs"),
|
||||
expose: LEGACY_HTTP_EXPOSE,
|
||||
port: { type: "auto", envKey: "PORT" },
|
||||
}))).rejects.toThrow(
|
||||
/deployment\/bind conflict: local_trusted requires server\.bind=loopback.*output: local_trusted requires server\.bind=loopback/s,
|
||||
);
|
||||
}, 20_000);
|
||||
});
|
||||
|
||||
describe("readiness probes loopback for an exposed runtime (PAP-17256)", () => {
|
||||
|
||||
@@ -52,6 +52,7 @@ import {
|
||||
findAdoptableLocalService,
|
||||
isLocalServiceProcessOwnedBy,
|
||||
isLocalServiceProcessInWorkspace,
|
||||
openLocalServiceLogFile,
|
||||
readLocalServiceProcessCwd,
|
||||
readLocalServicePortOwner,
|
||||
removeLocalServiceRegistryRecord,
|
||||
@@ -475,7 +476,7 @@ type ProcessOutputAccumulator = {
|
||||
* broker fake handles the removal rather than the real host broker.
|
||||
*/
|
||||
export async function resetRuntimeServicesForTests(
|
||||
opts: { terminateProcesses?: boolean } = {},
|
||||
opts: { terminateProcesses?: boolean; simulateSupervisorExit?: boolean } = {},
|
||||
) {
|
||||
if (opts.terminateProcesses) {
|
||||
for (const serviceId of [...runtimeServicesById.keys()]) {
|
||||
@@ -484,6 +485,13 @@ export async function resetRuntimeServicesForTests(
|
||||
}
|
||||
for (const record of runtimeServicesById.values()) {
|
||||
clearIdleTimer(record);
|
||||
if (opts.simulateSupervisorExit) {
|
||||
// A real supervisor exit closes its side of every inherited pipe. Tests
|
||||
// use this to prove surviving request-logging services do not depend on
|
||||
// Paperclip keeping an anonymous stdio peer alive.
|
||||
record.child?.stdout?.destroy();
|
||||
record.child?.stderr?.destroy();
|
||||
}
|
||||
}
|
||||
runtimeServicesById.clear();
|
||||
runtimeServicesByReuseKey.clear();
|
||||
@@ -2757,6 +2765,20 @@ function quoteShellArg(value: string) {
|
||||
return `'${value.replace(/'/g, `'\\''`)}'`;
|
||||
}
|
||||
|
||||
const BUILTIN_WORKSPACE_PROVISION_COMMAND = "bash ./scripts/provision-worktree.sh";
|
||||
|
||||
function resolveWorkspaceProvisionCommand(
|
||||
strategy: Record<string, unknown>,
|
||||
repoRoot: string,
|
||||
) {
|
||||
const configuredCommand = asString(strategy.provisionCommand, "").trim();
|
||||
if (configuredCommand) return configuredCommand;
|
||||
|
||||
return existsSync(path.join(repoRoot, "scripts", "provision-worktree.sh"))
|
||||
? BUILTIN_WORKSPACE_PROVISION_COMMAND
|
||||
: "";
|
||||
}
|
||||
|
||||
function resolveRepoManagedWorkspaceCommand(command: string, repoRoot: string) {
|
||||
const patterns = [
|
||||
/^(?<prefix>(?:bash|sh|zsh)\s+)(?<quote>["']?)(?<relative>\.\/[^"'\s]+)\k<quote>(?<suffix>(?:\s.*)?)$/s,
|
||||
@@ -2953,7 +2975,7 @@ async function provisionExecutionWorktree(input: {
|
||||
created: boolean;
|
||||
recorder?: WorkspaceOperationRecorder | null;
|
||||
}) {
|
||||
const provisionCommand = asString(input.strategy.provisionCommand, "").trim();
|
||||
const provisionCommand = resolveWorkspaceProvisionCommand(input.strategy, input.repoRoot);
|
||||
if (!provisionCommand) return;
|
||||
const resolvedProvisionCommand = resolveRepoManagedWorkspaceCommand(provisionCommand, input.repoRoot);
|
||||
|
||||
@@ -3434,22 +3456,20 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
|
||||
});
|
||||
realized.warnings = [...repairWarnings, ...baseRefreshWarnings, ...baseDrift.warnings];
|
||||
realized.baseRefSha = refresh.baseRefSha ?? recordedBaseRefSha ?? baseDrift.branchBaseRefSha ?? baseDrift.currentBaseRefSha;
|
||||
if (provisionCommand) {
|
||||
await provisionExecutionWorktree({
|
||||
strategy: {
|
||||
type: "git_worktree",
|
||||
provisionCommand,
|
||||
},
|
||||
base: input.base,
|
||||
repoRoot,
|
||||
worktreePath: realized.worktreePath ?? cwd,
|
||||
branchName: realized.branchName ?? "",
|
||||
issue: input.issue,
|
||||
agent: input.agent,
|
||||
created: false,
|
||||
recorder: input.recorder ?? null,
|
||||
});
|
||||
}
|
||||
await provisionExecutionWorktree({
|
||||
strategy: {
|
||||
type: "git_worktree",
|
||||
...(provisionCommand ? { provisionCommand } : {}),
|
||||
},
|
||||
base: input.base,
|
||||
repoRoot,
|
||||
worktreePath: realized.worktreePath ?? cwd,
|
||||
branchName: realized.branchName ?? "",
|
||||
issue: input.issue,
|
||||
agent: input.agent,
|
||||
created: false,
|
||||
recorder: input.recorder ?? null,
|
||||
});
|
||||
return realized;
|
||||
}
|
||||
|
||||
@@ -5139,17 +5159,12 @@ export function resolveRuntimeProvisionCommand(input: {
|
||||
|
||||
const stateDir = path.join(input.workspace.cwd, ".paperclip");
|
||||
const manifestPath = path.join(stateDir, "seed-manifest.json");
|
||||
const pendingMarker = path.join(stateDir, "seed-pending");
|
||||
const completeMarker = path.join(stateDir, "seed-complete");
|
||||
const provisionScript = path.join(
|
||||
input.workspace.baseCwd,
|
||||
"scripts",
|
||||
"provision-worktree-runtime.sh",
|
||||
);
|
||||
let needsSeed = existsSync(pendingMarker) && !existsSync(completeMarker);
|
||||
if (existsSync(manifestPath)) {
|
||||
needsSeed = !hasVerifiedWorktreeSeedManifest(manifestPath);
|
||||
}
|
||||
const needsSeed = !existsSync(manifestPath) || !hasVerifiedWorktreeSeedManifest(manifestPath);
|
||||
if (!needsSeed || !existsSync(provisionScript)) {
|
||||
return "";
|
||||
}
|
||||
@@ -5535,7 +5550,7 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
// still rejected by the broker unless /proc proves loopback-only listeners.
|
||||
//
|
||||
// Three independent layers force the loopback bind, because a guest checkout
|
||||
// can be arbitrarily old (PAP-17256): the `--bind custom --bind-host` argv
|
||||
// can be arbitrarily old (PAP-17256): the `--bind loopback` argv
|
||||
// added above, these env vars for a runner that reads them, and HOST for one
|
||||
// old enough to ignore both and infer its bind mode from HOST alone.
|
||||
env.PAPERCLIP_BIND = RUNTIME_EXPOSURE_BIND_MODE;
|
||||
@@ -5748,12 +5763,21 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
}
|
||||
|
||||
const shell = resolveShell();
|
||||
const child = spawn(shell, ["-lc", command], {
|
||||
cwd: serviceCwd,
|
||||
env,
|
||||
detached: process.platform !== "win32",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
const serviceLog = await openLocalServiceLogFile(serviceKey);
|
||||
let child: ChildProcess;
|
||||
try {
|
||||
child = spawn(shell, ["-lc", command], {
|
||||
cwd: serviceCwd,
|
||||
env,
|
||||
detached: process.platform !== "win32",
|
||||
// The service receives duplicate append-only file descriptors. Closing
|
||||
// Paperclip (or this parent handle below) cannot strand a request logger
|
||||
// on an orphaned socketpair during startup reconciliation.
|
||||
stdio: ["ignore", serviceLog.handle.fd, serviceLog.handle.fd],
|
||||
});
|
||||
} finally {
|
||||
await serviceLog.handle.close();
|
||||
}
|
||||
record.child = child;
|
||||
record.providerRef = child.pid ? String(child.pid) : null;
|
||||
record.processGroupId = child.pid ?? null;
|
||||
@@ -5763,24 +5787,23 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
});
|
||||
});
|
||||
const earlyExitPromise = new Promise<never>((_, reject) => {
|
||||
child.once("exit", (code, signal) => {
|
||||
// `close` follows `exit` after the child's inherited stdout/stderr file
|
||||
// descriptors are closed. Waiting for it makes the startup log excerpt
|
||||
// deterministic instead of racing the final validation line.
|
||||
child.once("close", (code, signal) => {
|
||||
reject(new Error(
|
||||
`service process exited before readiness (code ${code ?? "unknown"}, signal ${signal ?? "none"})`,
|
||||
));
|
||||
});
|
||||
});
|
||||
let stderrExcerpt = "";
|
||||
let stdoutExcerpt = "";
|
||||
child.stdout?.on("data", async (chunk) => {
|
||||
const text = String(chunk);
|
||||
stdoutExcerpt = (stdoutExcerpt + text).slice(-4096);
|
||||
if (input.onLog) await input.onLog("stdout", `[service:${serviceName}] ${text}`);
|
||||
});
|
||||
child.stderr?.on("data", async (chunk) => {
|
||||
const text = String(chunk);
|
||||
stderrExcerpt = (stderrExcerpt + text).slice(-4096);
|
||||
if (input.onLog) await input.onLog("stderr", `[service:${serviceName}] ${text}`);
|
||||
});
|
||||
const readServiceOutputExcerpt = async () => {
|
||||
try {
|
||||
const contents = await fs.readFile(serviceLog.logPath);
|
||||
return contents.subarray(Math.max(serviceLog.startOffset, contents.length - 4096)).toString("utf8");
|
||||
} catch {
|
||||
return "";
|
||||
}
|
||||
};
|
||||
|
||||
if (child.pid) {
|
||||
await writeLocalServiceRegistryRecord({
|
||||
@@ -5907,6 +5930,10 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
record.healthStatus = "healthy";
|
||||
record.lastUsedAt = new Date().toISOString();
|
||||
record.stoppedAt = null;
|
||||
const serviceOutputExcerpt = await readServiceOutputExcerpt();
|
||||
if (serviceOutputExcerpt && input.onLog) {
|
||||
await input.onLog("stdout", `[service:${serviceName}] ${serviceOutputExcerpt}`);
|
||||
}
|
||||
await touchLocalServiceRegistryRecord(record.serviceKey, {
|
||||
runtimeServiceId: record.id,
|
||||
lastSeenAt: record.lastUsedAt,
|
||||
@@ -5914,18 +5941,20 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
}).catch(async (err) => {
|
||||
releasePortReservation(reservedPort);
|
||||
releasePortReservation(claimedIdentityPort);
|
||||
const failureMessage = err instanceof Error ? err.message : String(err);
|
||||
const failureMessage = err instanceof Error ? err.message : String(err);
|
||||
const serviceOutputExcerpt = await readServiceOutputExcerpt();
|
||||
const bindCollision = !exposureConfig && (
|
||||
err instanceof RuntimeServicePortBindCollision || Boolean(
|
||||
port
|
||||
&& (input.allowFixedPortFallback || portType === "auto")
|
||||
&& /(?:EADDRINUSE|address already in use)/i.test(`${failureMessage}\n${stderrExcerpt}`),
|
||||
&& /(?:EADDRINUSE|address already in use)/i.test(`${failureMessage}\n${serviceOutputExcerpt}`),
|
||||
)
|
||||
);
|
||||
if (child.pid) {
|
||||
await terminateLocalService({
|
||||
pid: child.pid,
|
||||
processGroupId: child.pid,
|
||||
port,
|
||||
});
|
||||
}
|
||||
await cleanupRecordExposure(record, { preserveFailure: true });
|
||||
@@ -5938,8 +5967,14 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
await persistRuntimeServiceRecord(record.db, record).catch(() => undefined);
|
||||
}
|
||||
if (bindCollision && port) throw new RuntimeServicePortBindCollision(port);
|
||||
const deploymentBindConflict = /local_trusted requires server\.bind=loopback/i.test(
|
||||
`${failureMessage}\n${serviceOutputExcerpt}`,
|
||||
);
|
||||
const actionableFailure = deploymentBindConflict
|
||||
? `${failureMessage} | deployment/bind conflict: local_trusted requires server.bind=loopback; the managed runtime requested an incompatible bind mode`
|
||||
: failureMessage;
|
||||
throw new Error(
|
||||
`Failed to start runtime service "${serviceName}": ${failureMessage}${stderrExcerpt ? ` | stderr: ${stderrExcerpt.trim()}` : ""}`,
|
||||
`Failed to start runtime service "${serviceName}": ${actionableFailure}${serviceOutputExcerpt ? ` | output: ${serviceOutputExcerpt.trim()}` : ""}`,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -6125,19 +6160,14 @@ async function stopRuntimeService(serviceId: string) {
|
||||
const record = runtimeServicesById.get(serviceId);
|
||||
if (!record) return;
|
||||
clearIdleTimer(record);
|
||||
record.status = "stopped";
|
||||
record.healthStatus = "unknown";
|
||||
record.lastUsedAt = new Date().toISOString();
|
||||
record.stoppedAt = new Date().toISOString();
|
||||
runtimeServicesById.delete(serviceId);
|
||||
if (record.reuseKey && runtimeServicesByReuseKey.get(record.reuseKey) === record.id) {
|
||||
runtimeServicesByReuseKey.delete(record.reuseKey);
|
||||
}
|
||||
// Remove any public exposure first, but keep the process registered and the
|
||||
// row non-stopped until verified termination succeeds.
|
||||
await cleanupRecordExposure(record);
|
||||
if (record.child && record.child.pid) {
|
||||
await terminateLocalService({
|
||||
pid: record.child.pid,
|
||||
processGroupId: record.processGroupId ?? record.child.pid,
|
||||
port: record.port,
|
||||
});
|
||||
} else if (record.providerRef) {
|
||||
const pid = Number.parseInt(record.providerRef, 10);
|
||||
@@ -6145,9 +6175,18 @@ async function stopRuntimeService(serviceId: string) {
|
||||
await terminateLocalService({
|
||||
pid,
|
||||
processGroupId: record.processGroupId,
|
||||
port: record.port,
|
||||
});
|
||||
}
|
||||
}
|
||||
record.status = "stopped";
|
||||
record.healthStatus = "unknown";
|
||||
record.lastUsedAt = new Date().toISOString();
|
||||
record.stoppedAt = new Date().toISOString();
|
||||
runtimeServicesById.delete(serviceId);
|
||||
if (record.reuseKey && runtimeServicesByReuseKey.get(record.reuseKey) === record.id) {
|
||||
runtimeServicesByReuseKey.delete(record.reuseKey);
|
||||
}
|
||||
await removeLocalServiceRegistryRecord(record.serviceKey);
|
||||
await persistRuntimeServiceRecord(record.db, record);
|
||||
}
|
||||
|
||||
@@ -8,6 +8,11 @@ import {
|
||||
type PaperclipConfig,
|
||||
} from "@paperclipai/shared";
|
||||
import { updateEnvFileContents, writeEnvFileAtomicallyIfChanged } from "@paperclipai/shared/env-file";
|
||||
import {
|
||||
readWorktreePortRegistry,
|
||||
withWorktreePortRegistryLockSync,
|
||||
writeWorktreePortRegistry,
|
||||
} from "@paperclipai/shared/worktree-port-registry";
|
||||
import { resolvePaperclipConfigPath, resolvePaperclipEnvPath } from "./paths.js";
|
||||
import { rewriteUrlPort } from "./url-utils.js";
|
||||
|
||||
@@ -99,89 +104,6 @@ type WorktreeRuntimeContext = {
|
||||
secretsKeyFilePath: string;
|
||||
};
|
||||
|
||||
type WorktreePortRegistry = {
|
||||
version: 1;
|
||||
configPaths: string[];
|
||||
};
|
||||
|
||||
const WORKTREE_PORT_REGISTRY_FILE = "worktree-port-reservations.json";
|
||||
const WORKTREE_PORT_REGISTRY_LOCK_DIR = ".worktree-port-reservations.lock";
|
||||
const WORKTREE_PORT_REGISTRY_LOCK_STALE_MS = 5_000;
|
||||
const WORKTREE_PORT_REGISTRY_LOCK_TIMEOUT_MS = 10_000;
|
||||
const sleepSyncBuffer = new Int32Array(new SharedArrayBuffer(4));
|
||||
|
||||
function sleepSync(durationMs: number): void {
|
||||
Atomics.wait(sleepSyncBuffer, 0, 0, durationMs);
|
||||
}
|
||||
|
||||
function withWorktreePortRegistryLock<T>(homeDir: string, run: () => T): T {
|
||||
fs.mkdirSync(homeDir, { recursive: true });
|
||||
const lockPath = path.resolve(homeDir, WORKTREE_PORT_REGISTRY_LOCK_DIR);
|
||||
const deadline = Date.now() + WORKTREE_PORT_REGISTRY_LOCK_TIMEOUT_MS;
|
||||
|
||||
while (true) {
|
||||
try {
|
||||
fs.mkdirSync(lockPath);
|
||||
break;
|
||||
} catch (error) {
|
||||
const code = error instanceof Error && "code" in error ? error.code : null;
|
||||
if (code !== "EEXIST") throw error;
|
||||
|
||||
try {
|
||||
const ageMs = Date.now() - fs.statSync(lockPath).mtimeMs;
|
||||
if (ageMs > WORKTREE_PORT_REGISTRY_LOCK_STALE_MS) {
|
||||
fs.rmSync(lockPath, { recursive: true, force: true });
|
||||
continue;
|
||||
}
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (Date.now() >= deadline) {
|
||||
throw new Error(`Timed out waiting for worktree port reservation lock at ${lockPath}`);
|
||||
}
|
||||
sleepSync(25);
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
return run();
|
||||
} finally {
|
||||
fs.rmSync(lockPath, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function readWorktreePortRegistry(homeDir: string): Set<string> {
|
||||
const registryPath = path.resolve(homeDir, WORKTREE_PORT_REGISTRY_FILE);
|
||||
if (!fs.existsSync(registryPath)) return new Set();
|
||||
|
||||
try {
|
||||
const parsed = JSON.parse(fs.readFileSync(registryPath, "utf8")) as Partial<WorktreePortRegistry>;
|
||||
if (parsed.version !== 1 || !Array.isArray(parsed.configPaths)) return new Set();
|
||||
return new Set(
|
||||
parsed.configPaths
|
||||
.filter((configPath): configPath is string => typeof configPath === "string" && configPath.length > 0)
|
||||
.map((configPath) => path.resolve(configPath)),
|
||||
);
|
||||
} catch {
|
||||
return new Set();
|
||||
}
|
||||
}
|
||||
|
||||
function writeWorktreePortRegistry(homeDir: string, configPaths: Iterable<string>): void {
|
||||
const registryPath = path.resolve(homeDir, WORKTREE_PORT_REGISTRY_FILE);
|
||||
const persistedPaths = Array.from(new Set(Array.from(configPaths, (configPath) => path.resolve(configPath))))
|
||||
.filter((configPath) => fs.existsSync(configPath))
|
||||
.sort();
|
||||
const registry: WorktreePortRegistry = {
|
||||
version: 1,
|
||||
configPaths: persistedPaths,
|
||||
};
|
||||
const temporaryPath = `${registryPath}.${process.pid}.tmp`;
|
||||
fs.writeFileSync(temporaryPath, `${JSON.stringify(registry, null, 2)}\n`, { mode: 0o600 });
|
||||
fs.renameSync(temporaryPath, registryPath);
|
||||
}
|
||||
|
||||
function resolveWorktreeRuntimeContext(
|
||||
env: NodeJS.ProcessEnv,
|
||||
overrideConfigPath?: string,
|
||||
@@ -354,7 +276,6 @@ function collectSiblingWorktreePorts(
|
||||
serverPorts.add(siblingConfig.server.port);
|
||||
}
|
||||
if (
|
||||
siblingConfig.database.mode === "embedded-postgres" &&
|
||||
Number.isInteger(siblingConfig.database.embeddedPostgresPort) &&
|
||||
siblingConfig.database.embeddedPostgresPort > 0
|
||||
) {
|
||||
@@ -542,7 +463,7 @@ export function maybeRepairLegacyWorktreeConfigAndEnvFiles(): {
|
||||
let repairedConfig = false;
|
||||
if (fs.existsSync(context.configPath)) {
|
||||
try {
|
||||
const runtimeConfig = withWorktreePortRegistryLock(context.homeDir, () => {
|
||||
const runtimeConfig = withWorktreePortRegistryLockSync(context.homeDir, () => {
|
||||
const parsed = JSON.parse(fs.readFileSync(context.configPath, "utf8")) as PaperclipConfig;
|
||||
let selectedConfig = parsed;
|
||||
const registeredConfigPaths = readWorktreePortRegistry(context.homeDir);
|
||||
@@ -671,4 +592,3 @@ export function maybePersistWorktreeRuntimePorts(input: {
|
||||
writeConfigFile(context.configPath, config);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user