mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-25 02:26:12 +02:00
fix(workspaces): make managed runtimes reliable across restarts (#11740)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Execution workspaces need isolated databases, ports, and runtime services > - Concurrent workspaces could reuse ports or lose service ownership after a restart > - A markerless worktree also needed seed recovery, but normal markerless instances still needed to boot > - This pull request makes seed, port, and service ownership state explicit and recoverable > - It also checks live process and listener identity before it reclaims shared resources > - The benefit is reliable workspace startup, restart, adoption, and concurrent provisioning ## Linked Issues or Issue Description **What happened?** Managed workspaces could lose runtime service ownership after a control-plane restart. Concurrent worktrees could also reuse a port when their parent paths differed. A seed recovery change made every markerless instance resolve a worktree seed source, so normal instances without a source could not start. **Expected behavior** Paperclip must preserve healthy managed services across restarts. It must reserve unique ports across worktree parents. It must provision a registered markerless worktree, but it must skip seed work for a normal markerless instance. **Steps to reproduce** 1. Start two managed worktrees under different parent paths at the same time. 2. Restart the control plane while a managed service stays alive. 3. Start Paperclip with a config that has no seed markers and no registered worktree source. 4. Observe duplicate port selection, lost service adoption, or a seed-source startup error. **Paperclip version or commit** Current `master` plus the workspace runtime reliability changes in this pull request. **Deployment mode** Local development with managed execution workspaces and embedded Postgres. ## What Changed - Added a shared port registry with lease heartbeats, process identity checks, and live listener probes. - Reserved worktree ports across custom parent paths and repaired duplicate legacy assignments. - Preserved and adopted healthy managed services across control-plane restarts. - Reconciled guest bind modes and verified listener ownership before termination or reuse. - Provisioned registered markerless worktree databases and kept normal markerless instance startup as a no-op. - Added CLI, shared, server, and shell regression tests for seed, port, listener, restart, and adoption behavior. - Updated the worktree development documentation. ## Verification - `pnpm exec vitest run cli/src/__tests__/worktree.test.ts --reporter=verbose` — 63 tests passed. - `pnpm exec vitest run packages/shared/src/worktree-port-registry.test.ts --reporter=verbose` — 5 tests passed. - Focused runtime Vitest set — 199 tests passed across 37 suites. - `node --test scripts/__tests__/provision-worktree-self-heal.test.mjs` — 10 tests passed. - `git diff --check` passed. ## Risks - Port reservation now depends on lease and process identity data. The fallback listener probe prevents early reclamation when process metadata is incomplete. - Runtime adoption is stricter about bind and owner identity. The tests cover healthy adoption, stale records, PID reuse, and unrelated listeners. - Markerless seed detection now separates registered worktrees from normal instances. The tests cover both paths. - There are no database schema migrations. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex with the `gpt-5` model family. The serving snapshot and context-window size are not exposed. The agent used reasoning, repository tools, code execution, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Dev Agent <dev@paperclip.ing>
This commit is contained in:
21 files changed
+2406
-345
No files matched your search
@@ -52,6 +52,7 @@ import {
|
||||
findAdoptableLocalService,
|
||||
isLocalServiceProcessOwnedBy,
|
||||
isLocalServiceProcessInWorkspace,
|
||||
openLocalServiceLogFile,
|
||||
readLocalServiceProcessCwd,
|
||||
readLocalServicePortOwner,
|
||||
removeLocalServiceRegistryRecord,
|
||||
@@ -475,7 +476,7 @@ type ProcessOutputAccumulator = {
|
||||
* broker fake handles the removal rather than the real host broker.
|
||||
*/
|
||||
export async function resetRuntimeServicesForTests(
|
||||
opts: { terminateProcesses?: boolean } = {},
|
||||
opts: { terminateProcesses?: boolean; simulateSupervisorExit?: boolean } = {},
|
||||
) {
|
||||
if (opts.terminateProcesses) {
|
||||
for (const serviceId of [...runtimeServicesById.keys()]) {
|
||||
@@ -484,6 +485,13 @@ export async function resetRuntimeServicesForTests(
|
||||
}
|
||||
for (const record of runtimeServicesById.values()) {
|
||||
clearIdleTimer(record);
|
||||
if (opts.simulateSupervisorExit) {
|
||||
// A real supervisor exit closes its side of every inherited pipe. Tests
|
||||
// use this to prove surviving request-logging services do not depend on
|
||||
// Paperclip keeping an anonymous stdio peer alive.
|
||||
record.child?.stdout?.destroy();
|
||||
record.child?.stderr?.destroy();
|
||||
}
|
||||
}
|
||||
runtimeServicesById.clear();
|
||||
runtimeServicesByReuseKey.clear();
|
||||
@@ -2757,6 +2765,20 @@ function quoteShellArg(value: string) {
|
||||
return `'${value.replace(/'/g, `'\\''`)}'`;
|
||||
}
|
||||
|
||||
const BUILTIN_WORKSPACE_PROVISION_COMMAND = "bash ./scripts/provision-worktree.sh";
|
||||
|
||||
function resolveWorkspaceProvisionCommand(
|
||||
strategy: Record<string, unknown>,
|
||||
repoRoot: string,
|
||||
) {
|
||||
const configuredCommand = asString(strategy.provisionCommand, "").trim();
|
||||
if (configuredCommand) return configuredCommand;
|
||||
|
||||
return existsSync(path.join(repoRoot, "scripts", "provision-worktree.sh"))
|
||||
? BUILTIN_WORKSPACE_PROVISION_COMMAND
|
||||
: "";
|
||||
}
|
||||
|
||||
function resolveRepoManagedWorkspaceCommand(command: string, repoRoot: string) {
|
||||
const patterns = [
|
||||
/^(?<prefix>(?:bash|sh|zsh)\s+)(?<quote>["']?)(?<relative>\.\/[^"'\s]+)\k<quote>(?<suffix>(?:\s.*)?)$/s,
|
||||
@@ -2953,7 +2975,7 @@ async function provisionExecutionWorktree(input: {
|
||||
created: boolean;
|
||||
recorder?: WorkspaceOperationRecorder | null;
|
||||
}) {
|
||||
const provisionCommand = asString(input.strategy.provisionCommand, "").trim();
|
||||
const provisionCommand = resolveWorkspaceProvisionCommand(input.strategy, input.repoRoot);
|
||||
if (!provisionCommand) return;
|
||||
const resolvedProvisionCommand = resolveRepoManagedWorkspaceCommand(provisionCommand, input.repoRoot);
|
||||
|
||||
@@ -3434,22 +3456,20 @@ export async function ensurePersistedExecutionWorkspaceAvailable(input: {
|
||||
});
|
||||
realized.warnings = [...repairWarnings, ...baseRefreshWarnings, ...baseDrift.warnings];
|
||||
realized.baseRefSha = refresh.baseRefSha ?? recordedBaseRefSha ?? baseDrift.branchBaseRefSha ?? baseDrift.currentBaseRefSha;
|
||||
if (provisionCommand) {
|
||||
await provisionExecutionWorktree({
|
||||
strategy: {
|
||||
type: "git_worktree",
|
||||
provisionCommand,
|
||||
},
|
||||
base: input.base,
|
||||
repoRoot,
|
||||
worktreePath: realized.worktreePath ?? cwd,
|
||||
branchName: realized.branchName ?? "",
|
||||
issue: input.issue,
|
||||
agent: input.agent,
|
||||
created: false,
|
||||
recorder: input.recorder ?? null,
|
||||
});
|
||||
}
|
||||
await provisionExecutionWorktree({
|
||||
strategy: {
|
||||
type: "git_worktree",
|
||||
...(provisionCommand ? { provisionCommand } : {}),
|
||||
},
|
||||
base: input.base,
|
||||
repoRoot,
|
||||
worktreePath: realized.worktreePath ?? cwd,
|
||||
branchName: realized.branchName ?? "",
|
||||
issue: input.issue,
|
||||
agent: input.agent,
|
||||
created: false,
|
||||
recorder: input.recorder ?? null,
|
||||
});
|
||||
return realized;
|
||||
}
|
||||
|
||||
@@ -5139,17 +5159,12 @@ export function resolveRuntimeProvisionCommand(input: {
|
||||
|
||||
const stateDir = path.join(input.workspace.cwd, ".paperclip");
|
||||
const manifestPath = path.join(stateDir, "seed-manifest.json");
|
||||
const pendingMarker = path.join(stateDir, "seed-pending");
|
||||
const completeMarker = path.join(stateDir, "seed-complete");
|
||||
const provisionScript = path.join(
|
||||
input.workspace.baseCwd,
|
||||
"scripts",
|
||||
"provision-worktree-runtime.sh",
|
||||
);
|
||||
let needsSeed = existsSync(pendingMarker) && !existsSync(completeMarker);
|
||||
if (existsSync(manifestPath)) {
|
||||
needsSeed = !hasVerifiedWorktreeSeedManifest(manifestPath);
|
||||
}
|
||||
const needsSeed = !existsSync(manifestPath) || !hasVerifiedWorktreeSeedManifest(manifestPath);
|
||||
if (!needsSeed || !existsSync(provisionScript)) {
|
||||
return "";
|
||||
}
|
||||
@@ -5535,7 +5550,7 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
// still rejected by the broker unless /proc proves loopback-only listeners.
|
||||
//
|
||||
// Three independent layers force the loopback bind, because a guest checkout
|
||||
// can be arbitrarily old (PAP-17256): the `--bind custom --bind-host` argv
|
||||
// can be arbitrarily old (PAP-17256): the `--bind loopback` argv
|
||||
// added above, these env vars for a runner that reads them, and HOST for one
|
||||
// old enough to ignore both and infer its bind mode from HOST alone.
|
||||
env.PAPERCLIP_BIND = RUNTIME_EXPOSURE_BIND_MODE;
|
||||
@@ -5748,12 +5763,21 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
}
|
||||
|
||||
const shell = resolveShell();
|
||||
const child = spawn(shell, ["-lc", command], {
|
||||
cwd: serviceCwd,
|
||||
env,
|
||||
detached: process.platform !== "win32",
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
const serviceLog = await openLocalServiceLogFile(serviceKey);
|
||||
let child: ChildProcess;
|
||||
try {
|
||||
child = spawn(shell, ["-lc", command], {
|
||||
cwd: serviceCwd,
|
||||
env,
|
||||
detached: process.platform !== "win32",
|
||||
// The service receives duplicate append-only file descriptors. Closing
|
||||
// Paperclip (or this parent handle below) cannot strand a request logger
|
||||
// on an orphaned socketpair during startup reconciliation.
|
||||
stdio: ["ignore", serviceLog.handle.fd, serviceLog.handle.fd],
|
||||
});
|
||||
} finally {
|
||||
await serviceLog.handle.close();
|
||||
}
|
||||
record.child = child;
|
||||
record.providerRef = child.pid ? String(child.pid) : null;
|
||||
record.processGroupId = child.pid ?? null;
|
||||
@@ -5763,24 +5787,23 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
});
|
||||
});
|
||||
const earlyExitPromise = new Promise<never>((_, reject) => {
|
||||
child.once("exit", (code, signal) => {
|
||||
// `close` follows `exit` after the child's inherited stdout/stderr file
|
||||
// descriptors are closed. Waiting for it makes the startup log excerpt
|
||||
// deterministic instead of racing the final validation line.
|
||||
child.once("close", (code, signal) => {
|
||||
reject(new Error(
|
||||
`service process exited before readiness (code ${code ?? "unknown"}, signal ${signal ?? "none"})`,
|
||||
));
|
||||
});
|
||||
});
|
||||
let stderrExcerpt = "";
|
||||
let stdoutExcerpt = "";
|
||||
child.stdout?.on("data", async (chunk) => {
|
||||
const text = String(chunk);
|
||||
stdoutExcerpt = (stdoutExcerpt + text).slice(-4096);
|
||||
if (input.onLog) await input.onLog("stdout", `[service:${serviceName}] ${text}`);
|
||||
});
|
||||
child.stderr?.on("data", async (chunk) => {
|
||||
const text = String(chunk);
|
||||
stderrExcerpt = (stderrExcerpt + text).slice(-4096);
|
||||
if (input.onLog) await input.onLog("stderr", `[service:${serviceName}] ${text}`);
|
||||
});
|
||||
const readServiceOutputExcerpt = async () => {
|
||||
try {
|
||||
const contents = await fs.readFile(serviceLog.logPath);
|
||||
return contents.subarray(Math.max(serviceLog.startOffset, contents.length - 4096)).toString("utf8");
|
||||
} catch {
|
||||
return "";
|
||||
}
|
||||
};
|
||||
|
||||
if (child.pid) {
|
||||
await writeLocalServiceRegistryRecord({
|
||||
@@ -5907,6 +5930,10 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
record.healthStatus = "healthy";
|
||||
record.lastUsedAt = new Date().toISOString();
|
||||
record.stoppedAt = null;
|
||||
const serviceOutputExcerpt = await readServiceOutputExcerpt();
|
||||
if (serviceOutputExcerpt && input.onLog) {
|
||||
await input.onLog("stdout", `[service:${serviceName}] ${serviceOutputExcerpt}`);
|
||||
}
|
||||
await touchLocalServiceRegistryRecord(record.serviceKey, {
|
||||
runtimeServiceId: record.id,
|
||||
lastSeenAt: record.lastUsedAt,
|
||||
@@ -5914,18 +5941,20 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
}).catch(async (err) => {
|
||||
releasePortReservation(reservedPort);
|
||||
releasePortReservation(claimedIdentityPort);
|
||||
const failureMessage = err instanceof Error ? err.message : String(err);
|
||||
const failureMessage = err instanceof Error ? err.message : String(err);
|
||||
const serviceOutputExcerpt = await readServiceOutputExcerpt();
|
||||
const bindCollision = !exposureConfig && (
|
||||
err instanceof RuntimeServicePortBindCollision || Boolean(
|
||||
port
|
||||
&& (input.allowFixedPortFallback || portType === "auto")
|
||||
&& /(?:EADDRINUSE|address already in use)/i.test(`${failureMessage}\n${stderrExcerpt}`),
|
||||
&& /(?:EADDRINUSE|address already in use)/i.test(`${failureMessage}\n${serviceOutputExcerpt}`),
|
||||
)
|
||||
);
|
||||
if (child.pid) {
|
||||
await terminateLocalService({
|
||||
pid: child.pid,
|
||||
processGroupId: child.pid,
|
||||
port,
|
||||
});
|
||||
}
|
||||
await cleanupRecordExposure(record, { preserveFailure: true });
|
||||
@@ -5938,8 +5967,14 @@ async function spawnLocalRuntimeService(input: StartLocalRuntimeServiceInput): P
|
||||
await persistRuntimeServiceRecord(record.db, record).catch(() => undefined);
|
||||
}
|
||||
if (bindCollision && port) throw new RuntimeServicePortBindCollision(port);
|
||||
const deploymentBindConflict = /local_trusted requires server\.bind=loopback/i.test(
|
||||
`${failureMessage}\n${serviceOutputExcerpt}`,
|
||||
);
|
||||
const actionableFailure = deploymentBindConflict
|
||||
? `${failureMessage} | deployment/bind conflict: local_trusted requires server.bind=loopback; the managed runtime requested an incompatible bind mode`
|
||||
: failureMessage;
|
||||
throw new Error(
|
||||
`Failed to start runtime service "${serviceName}": ${failureMessage}${stderrExcerpt ? ` | stderr: ${stderrExcerpt.trim()}` : ""}`,
|
||||
`Failed to start runtime service "${serviceName}": ${actionableFailure}${serviceOutputExcerpt ? ` | output: ${serviceOutputExcerpt.trim()}` : ""}`,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -6125,19 +6160,14 @@ async function stopRuntimeService(serviceId: string) {
|
||||
const record = runtimeServicesById.get(serviceId);
|
||||
if (!record) return;
|
||||
clearIdleTimer(record);
|
||||
record.status = "stopped";
|
||||
record.healthStatus = "unknown";
|
||||
record.lastUsedAt = new Date().toISOString();
|
||||
record.stoppedAt = new Date().toISOString();
|
||||
runtimeServicesById.delete(serviceId);
|
||||
if (record.reuseKey && runtimeServicesByReuseKey.get(record.reuseKey) === record.id) {
|
||||
runtimeServicesByReuseKey.delete(record.reuseKey);
|
||||
}
|
||||
// Remove any public exposure first, but keep the process registered and the
|
||||
// row non-stopped until verified termination succeeds.
|
||||
await cleanupRecordExposure(record);
|
||||
if (record.child && record.child.pid) {
|
||||
await terminateLocalService({
|
||||
pid: record.child.pid,
|
||||
processGroupId: record.processGroupId ?? record.child.pid,
|
||||
port: record.port,
|
||||
});
|
||||
} else if (record.providerRef) {
|
||||
const pid = Number.parseInt(record.providerRef, 10);
|
||||
@@ -6145,9 +6175,18 @@ async function stopRuntimeService(serviceId: string) {
|
||||
await terminateLocalService({
|
||||
pid,
|
||||
processGroupId: record.processGroupId,
|
||||
port: record.port,
|
||||
});
|
||||
}
|
||||
}
|
||||
record.status = "stopped";
|
||||
record.healthStatus = "unknown";
|
||||
record.lastUsedAt = new Date().toISOString();
|
||||
record.stoppedAt = new Date().toISOString();
|
||||
runtimeServicesById.delete(serviceId);
|
||||
if (record.reuseKey && runtimeServicesByReuseKey.get(record.reuseKey) === record.id) {
|
||||
runtimeServicesByReuseKey.delete(record.reuseKey);
|
||||
}
|
||||
await removeLocalServiceRegistryRecord(record.serviceKey);
|
||||
await persistRuntimeServiceRecord(record.db, record);
|
||||
}
|
||||
|
||||
Reference in new issue
Block a user