mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 00:53:38 +02:00
fix(workspaces): make managed runtimes reliable across restarts (#11740)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Execution workspaces need isolated databases, ports, and runtime services > - Concurrent workspaces could reuse ports or lose service ownership after a restart > - A markerless worktree also needed seed recovery, but normal markerless instances still needed to boot > - This pull request makes seed, port, and service ownership state explicit and recoverable > - It also checks live process and listener identity before it reclaims shared resources > - The benefit is reliable workspace startup, restart, adoption, and concurrent provisioning ## Linked Issues or Issue Description **What happened?** Managed workspaces could lose runtime service ownership after a control-plane restart. Concurrent worktrees could also reuse a port when their parent paths differed. A seed recovery change made every markerless instance resolve a worktree seed source, so normal instances without a source could not start. **Expected behavior** Paperclip must preserve healthy managed services across restarts. It must reserve unique ports across worktree parents. It must provision a registered markerless worktree, but it must skip seed work for a normal markerless instance. **Steps to reproduce** 1. Start two managed worktrees under different parent paths at the same time. 2. Restart the control plane while a managed service stays alive. 3. Start Paperclip with a config that has no seed markers and no registered worktree source. 4. Observe duplicate port selection, lost service adoption, or a seed-source startup error. **Paperclip version or commit** Current `master` plus the workspace runtime reliability changes in this pull request. **Deployment mode** Local development with managed execution workspaces and embedded Postgres. ## What Changed - Added a shared port registry with lease heartbeats, process identity checks, and live listener probes. - Reserved worktree ports across custom parent paths and repaired duplicate legacy assignments. - Preserved and adopted healthy managed services across control-plane restarts. - Reconciled guest bind modes and verified listener ownership before termination or reuse. - Provisioned registered markerless worktree databases and kept normal markerless instance startup as a no-op. - Added CLI, shared, server, and shell regression tests for seed, port, listener, restart, and adoption behavior. - Updated the worktree development documentation. ## Verification - `pnpm exec vitest run cli/src/__tests__/worktree.test.ts --reporter=verbose` — 63 tests passed. - `pnpm exec vitest run packages/shared/src/worktree-port-registry.test.ts --reporter=verbose` — 5 tests passed. - Focused runtime Vitest set — 199 tests passed across 37 suites. - `node --test scripts/__tests__/provision-worktree-self-heal.test.mjs` — 10 tests passed. - `git diff --check` passed. ## Risks - Port reservation now depends on lease and process identity data. The fallback listener probe prevents early reclamation when process metadata is incomplete. - Runtime adoption is stricter about bind and owner identity. The tests cover healthy adoption, stale records, PID reuse, and unrelated listeners. - Markerless seed detection now separates registered worktrees from normal instances. The tests cover both paths. - There are no database schema migrations. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex with the `gpt-5` model family. The serving snapshot and context-window size are not exposed. The agent used reasoning, repository tools, code execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Dev Agent <dev@paperclip.ing>
This commit is contained in:
21 files changed
+2406
-345
No files matched your search
@@ -13,16 +13,16 @@ describe("forceLoopbackBindInCommand", () => {
|
||||
// PAP-17256 reproduction: this is verbatim what `workspace_runtime_services`
|
||||
// recorded for every lane the broker denied.
|
||||
expect(forceLoopbackBindInCommand("pnpm dev --bind lan")).toBe(
|
||||
"pnpm dev --bind custom --bind-host 127.0.0.1",
|
||||
"pnpm dev --bind loopback",
|
||||
);
|
||||
});
|
||||
|
||||
it("replaces an existing loopback-adjacent bind rather than duplicating it", () => {
|
||||
expect(forceLoopbackBindInCommand("pnpm dev --bind tailnet")).toBe(
|
||||
"pnpm dev --bind custom --bind-host 127.0.0.1",
|
||||
"pnpm dev --bind loopback",
|
||||
);
|
||||
expect(forceLoopbackBindInCommand("pnpm dev --bind custom --bind-host 10.0.0.4")).toBe(
|
||||
"pnpm dev --bind custom --bind-host 127.0.0.1",
|
||||
"pnpm dev --bind loopback",
|
||||
);
|
||||
});
|
||||
|
||||
@@ -33,7 +33,7 @@ describe("forceLoopbackBindInCommand", () => {
|
||||
|
||||
it("handles the =-separated spelling", () => {
|
||||
expect(forceLoopbackBindInCommand("pnpm dev --bind=lan --bind-host=0.0.0.0")).toBe(
|
||||
"pnpm dev --bind custom --bind-host 127.0.0.1",
|
||||
"pnpm dev --bind loopback",
|
||||
);
|
||||
});
|
||||
|
||||
@@ -41,7 +41,7 @@ describe("forceLoopbackBindInCommand", () => {
|
||||
// A bare `pnpm dev` lets an old guest runner infer its bind from HOST, so the
|
||||
// flags must be added rather than assumed.
|
||||
expect(forceLoopbackBindInCommand("pnpm dev")).toBe(
|
||||
"pnpm dev --bind custom --bind-host 127.0.0.1",
|
||||
"pnpm dev --bind loopback",
|
||||
);
|
||||
});
|
||||
|
||||
@@ -49,19 +49,19 @@ describe("forceLoopbackBindInCommand", () => {
|
||||
// `isPaperclipDevRuntimeService` matches `--tailscale-auth` as a substring;
|
||||
// an explicit `--bind` already beats the alias in every dev-runner version.
|
||||
expect(forceLoopbackBindInCommand("pnpm dev:once --tailscale-auth")).toBe(
|
||||
"pnpm dev:once --tailscale-auth --bind custom --bind-host 127.0.0.1",
|
||||
"pnpm dev:once --tailscale-auth --bind loopback",
|
||||
);
|
||||
});
|
||||
|
||||
it("never strips a value that is actually the next flag", () => {
|
||||
expect(forceLoopbackBindInCommand("pnpm dev --bind --verbose")).toBe(
|
||||
"pnpm dev --bind --verbose --bind custom --bind-host 127.0.0.1",
|
||||
"pnpm dev --bind --verbose --bind loopback",
|
||||
);
|
||||
});
|
||||
|
||||
it("does not touch a --bind occurrence that is not a flag boundary", () => {
|
||||
expect(forceLoopbackBindInCommand("pnpm dev --no--bind lan")).toBe(
|
||||
"pnpm dev --no--bind lan --bind custom --bind-host 127.0.0.1",
|
||||
"pnpm dev --no--bind lan --bind loopback",
|
||||
);
|
||||
});
|
||||
|
||||
@@ -74,8 +74,9 @@ describe("forceLoopbackBindInCommand", () => {
|
||||
expect(forceLoopbackBindInCommand(fixture)).toBe(fixture);
|
||||
});
|
||||
|
||||
it("uses the loopback host constant", () => {
|
||||
expect(forceLoopbackBindInCommand("pnpm dev")).toContain(RUNTIME_EXPOSURE_BIND_HOST);
|
||||
it("uses the loopback preset without turning it into a custom bind", () => {
|
||||
expect(forceLoopbackBindInCommand("pnpm dev")).toBe("pnpm dev --bind loopback");
|
||||
expect(RUNTIME_EXPOSURE_BIND_HOST).toBe("127.0.0.1");
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -18,8 +18,8 @@
|
||||
* Pure string functions only — no I/O, no process state.
|
||||
*/
|
||||
|
||||
/** The only address an exposed managed runtime may bind. */
|
||||
export const RUNTIME_EXPOSURE_BIND_MODE = "custom";
|
||||
/** The only bind preset an exposed managed runtime may use. */
|
||||
export const RUNTIME_EXPOSURE_BIND_MODE = "loopback";
|
||||
export const RUNTIME_EXPOSURE_BIND_HOST = "127.0.0.1";
|
||||
|
||||
/**
|
||||
@@ -72,7 +72,7 @@ export function forceLoopbackBindInCommand(command: string): string {
|
||||
if (!isPaperclipDevRunnerCommand(command)) return command;
|
||||
const stripped = command.replace(BIND_SELECTING_ARG, "").trim();
|
||||
if (stripped.length === 0) return command;
|
||||
return `${stripped} --bind ${RUNTIME_EXPOSURE_BIND_MODE} --bind-host ${RUNTIME_EXPOSURE_BIND_HOST}`;
|
||||
return `${stripped} --bind ${RUNTIME_EXPOSURE_BIND_MODE}`;
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -0,0 +1,147 @@
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import {
|
||||
withWorktreePortRegistryLock,
|
||||
withWorktreePortRegistryLockSync,
|
||||
} from "./worktree-port-registry.js";
|
||||
|
||||
const temporaryRoots: string[] = [];
|
||||
|
||||
function makeTemporaryRoot(): string {
|
||||
const root = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-port-registry-lock-"));
|
||||
temporaryRoots.push(root);
|
||||
return root;
|
||||
}
|
||||
|
||||
function deferred(): { promise: Promise<void>; resolve: () => void } {
|
||||
let resolve!: () => void;
|
||||
const promise = new Promise<void>((done) => {
|
||||
resolve = done;
|
||||
});
|
||||
return { promise, resolve };
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
for (const root of temporaryRoots.splice(0)) {
|
||||
fs.rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
describe("worktree port registry lock", () => {
|
||||
it("does not reclaim a stale lock while its fallback ownership probe responds", async () => {
|
||||
const homeDir = makeTemporaryRoot();
|
||||
const lockPath = path.join(homeDir, ".worktree-port-reservations.lock");
|
||||
const firstEntered = deferred();
|
||||
const releaseFirst = deferred();
|
||||
let secondEntered = false;
|
||||
|
||||
const first = withWorktreePortRegistryLock(homeDir, async () => {
|
||||
fs.renameSync(path.join(lockPath, "owner.json"), path.join(lockPath, "owner.unavailable.json"));
|
||||
const backupOwnerPath = path.join(lockPath, "owner.backup.json");
|
||||
const owner = JSON.parse(fs.readFileSync(backupOwnerPath, "utf8"));
|
||||
fs.writeFileSync(backupOwnerPath, `${JSON.stringify({
|
||||
...owner,
|
||||
processIdentity: "unavailable-process-identity",
|
||||
})}\n`);
|
||||
const oldTimestamp = new Date(Date.now() - 10_000);
|
||||
fs.utimesSync(lockPath, oldTimestamp, oldTimestamp);
|
||||
firstEntered.resolve();
|
||||
await releaseFirst.promise;
|
||||
});
|
||||
await firstEntered.promise;
|
||||
|
||||
expect(Date.now() - fs.statSync(lockPath).mtimeMs).toBeGreaterThan(5_000);
|
||||
|
||||
const second = withWorktreePortRegistryLock(homeDir, async () => {
|
||||
secondEntered = true;
|
||||
});
|
||||
await delay(100);
|
||||
|
||||
expect(secondEntered).toBe(false);
|
||||
releaseFirst.resolve();
|
||||
await Promise.all([first, second]);
|
||||
expect(secondEntered).toBe(true);
|
||||
}, 10_000);
|
||||
|
||||
it("refreshes the lease throughout an async critical section", async () => {
|
||||
const homeDir = makeTemporaryRoot();
|
||||
const lockPath = path.join(homeDir, ".worktree-port-reservations.lock");
|
||||
|
||||
await withWorktreePortRegistryLock(homeDir, async () => {
|
||||
await delay(5_250);
|
||||
expect(Date.now() - fs.statSync(lockPath).mtimeMs).toBeLessThan(2_000);
|
||||
});
|
||||
|
||||
expect(fs.existsSync(lockPath)).toBe(false);
|
||||
}, 10_000);
|
||||
|
||||
it("reclaims an old lock after its owner process exits", async () => {
|
||||
const homeDir = makeTemporaryRoot();
|
||||
const lockPath = path.join(homeDir, ".worktree-port-reservations.lock");
|
||||
fs.mkdirSync(lockPath);
|
||||
fs.writeFileSync(
|
||||
path.join(lockPath, "owner.json"),
|
||||
`${JSON.stringify({
|
||||
version: 1,
|
||||
pid: 2_147_483_647,
|
||||
processIdentity: "dead-process",
|
||||
probePort: 1,
|
||||
token: "dead-owner",
|
||||
})}\n`,
|
||||
);
|
||||
const oldTimestamp = new Date(Date.now() - 10_000);
|
||||
fs.utimesSync(lockPath, oldTimestamp, oldTimestamp);
|
||||
|
||||
let entered = false;
|
||||
await withWorktreePortRegistryLock(homeDir, async () => {
|
||||
entered = true;
|
||||
});
|
||||
|
||||
expect(entered).toBe(true);
|
||||
expect(fs.existsSync(lockPath)).toBe(false);
|
||||
});
|
||||
|
||||
it("reclaims an old lock when its pid belongs to a different process", async () => {
|
||||
const homeDir = makeTemporaryRoot();
|
||||
const lockPath = path.join(homeDir, ".worktree-port-reservations.lock");
|
||||
fs.mkdirSync(lockPath);
|
||||
fs.writeFileSync(
|
||||
path.join(lockPath, "owner.json"),
|
||||
`${JSON.stringify({
|
||||
version: 1,
|
||||
pid: process.pid,
|
||||
processIdentity: "reused-pid-owner",
|
||||
probePort: 1,
|
||||
token: "abandoned-owner",
|
||||
})}\n`,
|
||||
);
|
||||
const oldTimestamp = new Date(Date.now() - 10_000);
|
||||
fs.utimesSync(lockPath, oldTimestamp, oldTimestamp);
|
||||
|
||||
let entered = false;
|
||||
await withWorktreePortRegistryLock(homeDir, async () => {
|
||||
entered = true;
|
||||
});
|
||||
|
||||
expect(entered).toBe(true);
|
||||
expect(fs.existsSync(lockPath)).toBe(false);
|
||||
});
|
||||
|
||||
it("refreshes the lease while a synchronous critical section blocks the main thread", () => {
|
||||
const homeDir = makeTemporaryRoot();
|
||||
const lockPath = path.join(homeDir, ".worktree-port-reservations.lock");
|
||||
const blocker = new Int32Array(new SharedArrayBuffer(4));
|
||||
|
||||
withWorktreePortRegistryLockSync(homeDir, () => {
|
||||
const oldTimestamp = new Date(Date.now() - 10_000);
|
||||
fs.utimesSync(lockPath, oldTimestamp, oldTimestamp);
|
||||
Atomics.wait(blocker, 0, 0, 1_500);
|
||||
expect(Date.now() - fs.statSync(lockPath).mtimeMs).toBeLessThan(1_250);
|
||||
});
|
||||
|
||||
expect(fs.existsSync(lockPath)).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,411 @@
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { setTimeout as delay } from "node:timers/promises";
|
||||
import { Worker } from "node:worker_threads";
|
||||
|
||||
type WorktreePortRegistry = {
|
||||
version: 1;
|
||||
configPaths: string[];
|
||||
};
|
||||
|
||||
const WORKTREE_PORT_REGISTRY_FILE = "worktree-port-reservations.json";
|
||||
const WORKTREE_PORT_REGISTRY_LOCK_DIR = ".worktree-port-reservations.lock";
|
||||
const WORKTREE_PORT_REGISTRY_LOCK_OWNER_FILE = "owner.json";
|
||||
const WORKTREE_PORT_REGISTRY_LOCK_OWNER_BACKUP_FILE = "owner.backup.json";
|
||||
const WORKTREE_PORT_REGISTRY_LOCK_STALE_MS = 5_000;
|
||||
const WORKTREE_PORT_REGISTRY_LOCK_TIMEOUT_MS = 10_000;
|
||||
const WORKTREE_PORT_REGISTRY_LOCK_HEARTBEAT_MS = 1_000;
|
||||
const WORKTREE_PORT_REGISTRY_LOCK_PROBE_TIMEOUT_MS = 500;
|
||||
const sleepSyncBuffer = new Int32Array(new SharedArrayBuffer(4));
|
||||
|
||||
type RegistryLockOwner = {
|
||||
version: 1;
|
||||
pid: number;
|
||||
processIdentity: string;
|
||||
probePort: number;
|
||||
token: string;
|
||||
};
|
||||
|
||||
type RegistryLockLease = {
|
||||
token: string;
|
||||
worker: Worker;
|
||||
control: Int32Array;
|
||||
probePort: number;
|
||||
};
|
||||
|
||||
const REGISTRY_LOCK_HEARTBEAT_SOURCE = `
|
||||
const fs = require("node:fs");
|
||||
const net = require("node:net");
|
||||
const path = require("node:path");
|
||||
const { parentPort, workerData } = require("node:worker_threads");
|
||||
const control = new Int32Array(workerData.control);
|
||||
function touchOwnedLock() {
|
||||
let sawDifferentOwner = false;
|
||||
for (const ownerFile of workerData.ownerFiles) {
|
||||
try {
|
||||
const owner = JSON.parse(fs.readFileSync(path.join(workerData.lockPath, ownerFile), "utf8"));
|
||||
if (owner.token === workerData.token) {
|
||||
try {
|
||||
const now = new Date();
|
||||
fs.utimesSync(workerData.lockPath, now, now);
|
||||
return "refreshed";
|
||||
} catch {
|
||||
return "retry";
|
||||
}
|
||||
}
|
||||
sawDifferentOwner = true;
|
||||
} catch {
|
||||
// The redundant owner file may still be readable.
|
||||
}
|
||||
}
|
||||
return sawDifferentOwner ? "lost" : "retry";
|
||||
}
|
||||
let finished = false;
|
||||
let timer;
|
||||
function complete() {
|
||||
Atomics.store(control, 0, 2);
|
||||
Atomics.notify(control, 0);
|
||||
}
|
||||
function finish() {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
if (timer) clearInterval(timer);
|
||||
server.close(complete);
|
||||
setTimeout(complete, 250).unref();
|
||||
}
|
||||
const server = net.createServer((socket) => {
|
||||
socket.setEncoding("utf8");
|
||||
socket.setTimeout(workerData.probeTimeoutMs, () => socket.destroy());
|
||||
socket.once("data", (candidate) => {
|
||||
socket.end(candidate === workerData.token ? "owned" : "denied");
|
||||
});
|
||||
});
|
||||
parentPort.once("message", finish);
|
||||
server.once("error", () => {
|
||||
Atomics.store(control, 0, -1);
|
||||
Atomics.notify(control, 0);
|
||||
});
|
||||
server.listen(0, "127.0.0.1", () => {
|
||||
const address = server.address();
|
||||
if (!address || typeof address === "string") {
|
||||
Atomics.store(control, 0, -1);
|
||||
Atomics.notify(control, 0);
|
||||
return;
|
||||
}
|
||||
Atomics.store(control, 2, address.port);
|
||||
Atomics.store(control, 0, 1);
|
||||
Atomics.notify(control, 0);
|
||||
touchOwnedLock();
|
||||
timer = setInterval(() => {
|
||||
if (Atomics.load(control, 1) !== 0 || touchOwnedLock() === "lost") finish();
|
||||
}, workerData.heartbeatMs);
|
||||
});
|
||||
`;
|
||||
|
||||
const REGISTRY_LOCK_PROBE_SOURCE = `
|
||||
const net = require("node:net");
|
||||
const { workerData } = require("node:worker_threads");
|
||||
const control = new Int32Array(workerData.control);
|
||||
let finished = false;
|
||||
function finish(result) {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
Atomics.store(control, 0, result);
|
||||
Atomics.notify(control, 0);
|
||||
process.exit(0);
|
||||
}
|
||||
const socket = net.createConnection({ host: "127.0.0.1", port: workerData.port });
|
||||
socket.setEncoding("utf8");
|
||||
socket.setTimeout(workerData.timeoutMs, () => finish(2));
|
||||
socket.once("connect", () => socket.write(workerData.token));
|
||||
socket.once("data", (response) => finish(response === "owned" ? 1 : 2));
|
||||
socket.once("error", () => finish(2));
|
||||
socket.once("close", () => finish(2));
|
||||
`;
|
||||
|
||||
function resolveRegistryLockPath(homeDir: string): string {
|
||||
fs.mkdirSync(homeDir, { recursive: true });
|
||||
return path.resolve(homeDir, WORKTREE_PORT_REGISTRY_LOCK_DIR);
|
||||
}
|
||||
|
||||
function readRegistryLockOwner(lockPath: string): RegistryLockOwner | null {
|
||||
for (const ownerFile of [
|
||||
WORKTREE_PORT_REGISTRY_LOCK_OWNER_FILE,
|
||||
WORKTREE_PORT_REGISTRY_LOCK_OWNER_BACKUP_FILE,
|
||||
]) {
|
||||
try {
|
||||
const parsed = JSON.parse(
|
||||
fs.readFileSync(path.join(lockPath, ownerFile), "utf8"),
|
||||
) as Partial<RegistryLockOwner>;
|
||||
if (
|
||||
parsed.version === 1
|
||||
&& Number.isInteger(parsed.pid)
|
||||
&& (parsed.pid ?? 0) > 0
|
||||
&& typeof parsed.processIdentity === "string"
|
||||
&& parsed.processIdentity.length > 0
|
||||
&& Number.isInteger(parsed.probePort)
|
||||
&& (parsed.probePort ?? 0) > 0
|
||||
&& (parsed.probePort ?? 0) <= 65_535
|
||||
&& typeof parsed.token === "string"
|
||||
&& parsed.token.length > 0
|
||||
) {
|
||||
return parsed as RegistryLockOwner;
|
||||
}
|
||||
} catch {
|
||||
// Try the redundant owner record.
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function writeRegistryLockOwner(lockPath: string, owner: RegistryLockOwner): void {
|
||||
const contents = `${JSON.stringify(owner)}\n`;
|
||||
for (const ownerFile of [
|
||||
WORKTREE_PORT_REGISTRY_LOCK_OWNER_FILE,
|
||||
WORKTREE_PORT_REGISTRY_LOCK_OWNER_BACKUP_FILE,
|
||||
]) {
|
||||
const ownerPath = path.join(lockPath, ownerFile);
|
||||
const temporaryPath = `${ownerPath}.${owner.token}.tmp`;
|
||||
fs.writeFileSync(temporaryPath, contents, { mode: 0o600 });
|
||||
fs.renameSync(temporaryPath, ownerPath);
|
||||
}
|
||||
}
|
||||
|
||||
function isProcessAlive(pid: number): boolean {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (error) {
|
||||
return !(error instanceof Error && "code" in error && error.code === "ESRCH");
|
||||
}
|
||||
}
|
||||
|
||||
function readProcessIdentity(pid: number): string | null {
|
||||
if (!Number.isInteger(pid) || pid <= 0) return null;
|
||||
if (process.platform === "linux") {
|
||||
try {
|
||||
const stat = fs.readFileSync(`/proc/${pid}/stat`, "utf8");
|
||||
const commandEnd = stat.lastIndexOf(")");
|
||||
if (commandEnd < 0) return null;
|
||||
const fields = stat.slice(commandEnd + 1).trim().split(/\s+/);
|
||||
const startTicks = fields[19];
|
||||
if (!startTicks) return null;
|
||||
const bootId = fs.readFileSync("/proc/sys/kernel/random/boot_id", "utf8").trim();
|
||||
return bootId ? `linux:${bootId}:${startTicks}` : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
if (process.platform === "win32") {
|
||||
const ticks = execFileSync("powershell.exe", [
|
||||
"-NoProfile",
|
||||
"-NonInteractive",
|
||||
"-Command",
|
||||
`(Get-Process -Id ${pid}).StartTime.ToUniversalTime().Ticks`,
|
||||
], { encoding: "utf8", windowsHide: true }).trim();
|
||||
return ticks ? `win32:${ticks}` : null;
|
||||
}
|
||||
const startedAt = execFileSync("ps", ["-o", "lstart=", "-p", String(pid)], {
|
||||
encoding: "utf8",
|
||||
}).trim();
|
||||
return startedAt ? `${process.platform}:${startedAt}` : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function probeRegistryLockOwner(owner: RegistryLockOwner): boolean {
|
||||
const control = new Int32Array(new SharedArrayBuffer(4));
|
||||
const worker = new Worker(REGISTRY_LOCK_PROBE_SOURCE, {
|
||||
eval: true,
|
||||
execArgv: [],
|
||||
workerData: {
|
||||
control: control.buffer,
|
||||
port: owner.probePort,
|
||||
timeoutMs: WORKTREE_PORT_REGISTRY_LOCK_PROBE_TIMEOUT_MS,
|
||||
token: owner.token,
|
||||
},
|
||||
});
|
||||
Atomics.wait(control, 0, 0, WORKTREE_PORT_REGISTRY_LOCK_PROBE_TIMEOUT_MS + 250);
|
||||
const isOwner = Atomics.load(control, 0) === 1;
|
||||
void worker.terminate();
|
||||
return isOwner;
|
||||
}
|
||||
|
||||
function removeStaleRegistryLock(lockPath: string): boolean {
|
||||
try {
|
||||
const ageMs = Date.now() - fs.statSync(lockPath).mtimeMs;
|
||||
if (ageMs <= WORKTREE_PORT_REGISTRY_LOCK_STALE_MS) return false;
|
||||
const owner = readRegistryLockOwner(lockPath);
|
||||
if (owner && isProcessAlive(owner.pid)) {
|
||||
if (probeRegistryLockOwner(owner)) return false;
|
||||
const currentIdentity = readProcessIdentity(owner.pid);
|
||||
if (owner.processIdentity === currentIdentity) return false;
|
||||
}
|
||||
const currentOwner = readRegistryLockOwner(lockPath);
|
||||
if (owner ? currentOwner?.token !== owner.token : currentOwner !== null) return false;
|
||||
const currentAgeMs = Date.now() - fs.statSync(lockPath).mtimeMs;
|
||||
if (currentAgeMs <= WORKTREE_PORT_REGISTRY_LOCK_STALE_MS) return false;
|
||||
fs.rmSync(lockPath, { recursive: true, force: true });
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function startRegistryLockHeartbeat(lockPath: string, token: string): RegistryLockLease {
|
||||
const control = new Int32Array(new SharedArrayBuffer(12));
|
||||
const worker = new Worker(REGISTRY_LOCK_HEARTBEAT_SOURCE, {
|
||||
eval: true,
|
||||
execArgv: [],
|
||||
workerData: {
|
||||
control: control.buffer,
|
||||
heartbeatMs: WORKTREE_PORT_REGISTRY_LOCK_HEARTBEAT_MS,
|
||||
lockPath,
|
||||
ownerFiles: [
|
||||
WORKTREE_PORT_REGISTRY_LOCK_OWNER_FILE,
|
||||
WORKTREE_PORT_REGISTRY_LOCK_OWNER_BACKUP_FILE,
|
||||
],
|
||||
probeTimeoutMs: WORKTREE_PORT_REGISTRY_LOCK_PROBE_TIMEOUT_MS,
|
||||
token,
|
||||
},
|
||||
});
|
||||
Atomics.wait(control, 0, 0, 2_000);
|
||||
if (Atomics.load(control, 0) !== 1) {
|
||||
void worker.terminate();
|
||||
throw new Error(`Failed to start worktree port reservation lock heartbeat at ${lockPath}`);
|
||||
}
|
||||
const probePort = Atomics.load(control, 2);
|
||||
if (probePort <= 0) {
|
||||
void worker.terminate();
|
||||
throw new Error(`Failed to start worktree port reservation lock probe at ${lockPath}`);
|
||||
}
|
||||
return { token, worker, control, probePort };
|
||||
}
|
||||
|
||||
function stopRegistryLockHeartbeat(lease: RegistryLockLease): void {
|
||||
Atomics.store(lease.control, 1, 1);
|
||||
Atomics.notify(lease.control, 1);
|
||||
lease.worker.postMessage("stop");
|
||||
if (Atomics.load(lease.control, 0) === 1) {
|
||||
Atomics.wait(lease.control, 0, 1, 2_000);
|
||||
}
|
||||
void lease.worker.terminate();
|
||||
}
|
||||
|
||||
function acquireRegistryLock(lockPath: string, deadline: number): RegistryLockLease | null {
|
||||
try {
|
||||
fs.mkdirSync(lockPath);
|
||||
const token = `${process.pid}-${randomUUID()}`;
|
||||
let lease: RegistryLockLease | null = null;
|
||||
try {
|
||||
const processIdentity = readProcessIdentity(process.pid);
|
||||
if (!processIdentity) {
|
||||
throw new Error("Cannot determine worktree port reservation lock owner identity");
|
||||
}
|
||||
const owner: RegistryLockOwner = {
|
||||
version: 1,
|
||||
pid: process.pid,
|
||||
processIdentity,
|
||||
probePort: 1,
|
||||
token,
|
||||
};
|
||||
writeRegistryLockOwner(lockPath, owner);
|
||||
lease = startRegistryLockHeartbeat(lockPath, token);
|
||||
writeRegistryLockOwner(lockPath, { ...owner, probePort: lease.probePort });
|
||||
return lease;
|
||||
} catch (error) {
|
||||
if (lease) stopRegistryLockHeartbeat(lease);
|
||||
fs.rmSync(lockPath, { recursive: true, force: true });
|
||||
throw error;
|
||||
}
|
||||
} catch (error) {
|
||||
const code = error instanceof Error && "code" in error ? error.code : null;
|
||||
if (code !== "EEXIST") throw error;
|
||||
if (removeStaleRegistryLock(lockPath)) return null;
|
||||
if (Date.now() >= deadline) {
|
||||
throw new Error(`Timed out waiting for worktree port reservation lock at ${lockPath}`);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function releaseRegistryLock(lockPath: string, lease: RegistryLockLease): void {
|
||||
stopRegistryLockHeartbeat(lease);
|
||||
const owner = readRegistryLockOwner(lockPath);
|
||||
if (owner?.token !== lease.token) {
|
||||
return;
|
||||
}
|
||||
fs.rmSync(lockPath, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
export function withWorktreePortRegistryLockSync<T>(homeDir: string, run: () => T): T {
|
||||
const lockPath = resolveRegistryLockPath(homeDir);
|
||||
const deadline = Date.now() + WORKTREE_PORT_REGISTRY_LOCK_TIMEOUT_MS;
|
||||
let lease: RegistryLockLease | null = null;
|
||||
|
||||
while (!(lease = acquireRegistryLock(lockPath, deadline))) {
|
||||
Atomics.wait(sleepSyncBuffer, 0, 0, 25);
|
||||
}
|
||||
|
||||
try {
|
||||
return run();
|
||||
} finally {
|
||||
releaseRegistryLock(lockPath, lease);
|
||||
}
|
||||
}
|
||||
|
||||
export async function withWorktreePortRegistryLock<T>(
|
||||
homeDir: string,
|
||||
run: () => Promise<T>,
|
||||
): Promise<T> {
|
||||
const lockPath = resolveRegistryLockPath(homeDir);
|
||||
const deadline = Date.now() + WORKTREE_PORT_REGISTRY_LOCK_TIMEOUT_MS;
|
||||
let lease: RegistryLockLease | null = null;
|
||||
|
||||
while (!(lease = acquireRegistryLock(lockPath, deadline))) {
|
||||
await delay(25);
|
||||
}
|
||||
|
||||
try {
|
||||
return await run();
|
||||
} finally {
|
||||
releaseRegistryLock(lockPath, lease);
|
||||
}
|
||||
}
|
||||
|
||||
export function readWorktreePortRegistry(homeDir: string): Set<string> {
|
||||
const registryPath = path.resolve(homeDir, WORKTREE_PORT_REGISTRY_FILE);
|
||||
if (!fs.existsSync(registryPath)) return new Set();
|
||||
|
||||
try {
|
||||
const parsed = JSON.parse(fs.readFileSync(registryPath, "utf8")) as Partial<WorktreePortRegistry>;
|
||||
if (parsed.version !== 1 || !Array.isArray(parsed.configPaths)) return new Set();
|
||||
return new Set(
|
||||
parsed.configPaths
|
||||
.filter((configPath): configPath is string => typeof configPath === "string" && configPath.length > 0)
|
||||
.map((configPath) => path.resolve(configPath)),
|
||||
);
|
||||
} catch {
|
||||
return new Set();
|
||||
}
|
||||
}
|
||||
|
||||
export function writeWorktreePortRegistry(homeDir: string, configPaths: Iterable<string>): void {
|
||||
fs.mkdirSync(homeDir, { recursive: true });
|
||||
const registryPath = path.resolve(homeDir, WORKTREE_PORT_REGISTRY_FILE);
|
||||
const persistedPaths = Array.from(new Set(Array.from(configPaths, (configPath) => path.resolve(configPath))))
|
||||
.filter((configPath) => fs.existsSync(configPath))
|
||||
.sort();
|
||||
const registry: WorktreePortRegistry = {
|
||||
version: 1,
|
||||
configPaths: persistedPaths,
|
||||
};
|
||||
const temporaryPath = `${registryPath}.${process.pid}.tmp`;
|
||||
fs.writeFileSync(temporaryPath, `${JSON.stringify(registry, null, 2)}\n`, { mode: 0o600 });
|
||||
fs.renameSync(temporaryPath, registryPath);
|
||||
}
|
||||
@@ -14,6 +14,13 @@ export type CanonicalWorktreeSeedSource = {
|
||||
targetInstanceId: string;
|
||||
};
|
||||
|
||||
export type RegisteredWorktreeSeedSourceInput = {
|
||||
registeredBaseWorkspaceCwd?: string | null;
|
||||
explicitSourceConfigPath?: string | null;
|
||||
targetConfigPath: string;
|
||||
expectedTargetInstanceId: string;
|
||||
};
|
||||
|
||||
function readInstanceId(configPath: string, label: "source" | "target"): string {
|
||||
const envPath = path.join(path.dirname(configPath), ".env");
|
||||
if (!existsSync(envPath)) {
|
||||
@@ -47,23 +54,10 @@ function canonicalRegularFile(filePath: string, label: string): string {
|
||||
return canonical;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a worktree seed source without granting authority to the seed manifest.
|
||||
*
|
||||
* A managed caller supplies the project-workspace cwd from its server-owned row.
|
||||
* An operator may instead supply an explicit source config. When both are present,
|
||||
* the explicit path must still equal the registered project-workspace config.
|
||||
* Manifest source fields are diagnostic assertions only and can only make this
|
||||
* validation fail; they never select the returned source.
|
||||
*/
|
||||
export function resolveCanonicalWorktreeSeedSource(input: {
|
||||
registeredBaseWorkspaceCwd?: string | null;
|
||||
explicitSourceConfigPath?: string | null;
|
||||
targetConfigPath: string;
|
||||
expectedTargetInstanceId: string;
|
||||
manifestSource: WorktreeSeedSourceDiagnostic | null | undefined;
|
||||
manifestTargetInstanceId?: unknown;
|
||||
}): CanonicalWorktreeSeedSource {
|
||||
/** Resolve the authoritative source and target identities without consulting diagnostics. */
|
||||
export function resolveRegisteredWorktreeSeedSource(
|
||||
input: RegisteredWorktreeSeedSourceInput,
|
||||
): CanonicalWorktreeSeedSource {
|
||||
const registeredCwd = input.registeredBaseWorkspaceCwd?.trim();
|
||||
const explicitSource = input.explicitSourceConfigPath?.trim();
|
||||
if (!registeredCwd && !explicitSource) {
|
||||
@@ -120,6 +114,29 @@ export function resolveCanonicalWorktreeSeedSource(input: {
|
||||
throw new Error("Source and target Paperclip configs name the same instance.");
|
||||
}
|
||||
|
||||
return {
|
||||
baseWorkspaceCwd: canonicalBaseCwd,
|
||||
configPath: canonicalSourceConfigPath,
|
||||
instanceId: sourceInstanceId,
|
||||
targetConfigPath: canonicalTargetConfigPath,
|
||||
targetInstanceId,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a worktree seed source without granting authority to the seed manifest.
|
||||
*
|
||||
* A managed caller supplies the project-workspace cwd from its server-owned row.
|
||||
* An operator may instead supply an explicit source config. When both are present,
|
||||
* the explicit path must still equal the registered project-workspace config.
|
||||
* Manifest source fields are diagnostic assertions only and never select the
|
||||
* returned source.
|
||||
*/
|
||||
export function resolveCanonicalWorktreeSeedSource(input: RegisteredWorktreeSeedSourceInput & {
|
||||
manifestSource: WorktreeSeedSourceDiagnostic | null | undefined;
|
||||
manifestTargetInstanceId?: unknown;
|
||||
}): CanonicalWorktreeSeedSource {
|
||||
const registered = resolveRegisteredWorktreeSeedSource(input);
|
||||
const diagnosticPath = typeof input.manifestSource?.configPath === "string"
|
||||
? input.manifestSource.configPath.trim()
|
||||
: "";
|
||||
@@ -130,21 +147,14 @@ export function resolveCanonicalWorktreeSeedSource(input: {
|
||||
diagnosticPath,
|
||||
"Worktree seed manifest source diagnostic",
|
||||
);
|
||||
if (path.resolve(diagnosticPath) !== canonicalSourceConfigPath || canonicalDiagnosticPath !== canonicalSourceConfigPath) {
|
||||
if (path.resolve(diagnosticPath) !== registered.configPath || canonicalDiagnosticPath !== registered.configPath) {
|
||||
throw new Error("Worktree seed manifest source path does not match the registered canonical source.");
|
||||
}
|
||||
if (input.manifestSource?.instanceId !== sourceInstanceId) {
|
||||
if (input.manifestSource?.instanceId !== registered.instanceId) {
|
||||
throw new Error("Worktree seed manifest source instance does not match the registered source instance.");
|
||||
}
|
||||
if (input.manifestTargetInstanceId !== targetInstanceId) {
|
||||
if (input.manifestTargetInstanceId !== registered.targetInstanceId) {
|
||||
throw new Error("Worktree seed manifest target instance does not match the registered target instance.");
|
||||
}
|
||||
|
||||
return {
|
||||
baseWorkspaceCwd: canonicalBaseCwd,
|
||||
configPath: canonicalSourceConfigPath,
|
||||
instanceId: sourceInstanceId,
|
||||
targetConfigPath: canonicalTargetConfigPath,
|
||||
targetInstanceId,
|
||||
};
|
||||
return registered;
|
||||
}
|
||||
Reference in new issue
Block a user