Files
PaperClipAI/server/src/services/workspace-instance-cleanup.ts
T
dcac49a4fd feat(workspaces): defer isolated setup until runtime start (#10653)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Isolated workspaces give each task a safe and reproducible checkout.
> - The existing setup cloned the development database before an agent
needed to run the app.
> - This made worktree creation slower and heavier for tasks that never
start a service.
> - Runtime services already use one server start path for heartbeat,
operator, and startup recovery flows.
> - This pull request moves heavy setup to that start path and keeps
worktree creation lean.
> - The benefit is faster isolated workspace creation with the same
reliable runtime setup when a service starts.

## Linked Issues or Issue Description

Related pull request: #10652 covers the initial deferred
database-seeding slice. This pull request supersedes it with end-to-end
runtime provisioning and safe cleanup.

**What existing behavior does this improve?**

This improves isolated worktree creation, runtime service startup, and
isolated instance cleanup.

**Subsystem affected**

Cross-cutting: CLI worktree setup, server runtime orchestration, shared
workspace contracts, and development scripts.

**Current behavior**

Paperclip seeds an isolated development database during worktree
creation. It can also leave an isolated instance directory after
workspace teardown. This work happens even when no runtime service
starts.

**Proposed behavior**

Paperclip creates the worktree with a lean eager setup. It runs an
idempotent runtime provision command before the first managed service
spawn. Concurrent starts share one provision attempt. Teardown removes
the isolated instance safely.

**Reason and benefit**

Many agent tasks only edit and test code. They do not need a running
Paperclip instance. Deferring the database seed reduces workspace
startup cost while preserving automatic setup for tasks that start the
app.

**Breaking changes**

None. The new runtime provision command is optional. Existing workspace
behavior is unchanged when it is absent.

## What Changed

- Split Paperclip worktree setup into a lean eager script and an
idempotent runtime provision script.
- Added `runtimeProvisionCommand` to project, issue, realized workspace,
and persisted workspace contracts.
- Added a per-workspace provision mutex before local service spawn for
heartbeat, operator, and startup recovery flows.
- Added a persisted `provisioning` service state and the
`workspace_runtime_provision` operation phase.
- Kept provision time outside the service readiness timeout and made
failed attempts visible and retryable.
- Reclaimed isolated instance data during safe workspace teardown.
- Serialized deferred database seeding across processes and bound
teardown to the instance root captured in persisted workspace metadata.
- Added tests for config flow, concurrency, retry, no-op behavior,
readiness timing, scripts, CLI commands, and cleanup.
- Documented the eager and runtime provisioning contracts.

## Verification

- `pnpm -r typecheck`
- `pnpm build`
- `pnpm test:run` (server: 3,201 passed; UI: 3,345 passed; the CLI phase
exposed one environment-sensitive AWS doctor assertion because the agent
runtime injects static AWS credentials)
- `env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY pnpm exec vitest
run cli/src/__tests__/secrets.test.ts -t 'passes AWS doctor checks when
non-secret provider config is present'`
- Focused runtime tests cover serialized provisioning, retry after
stderr failure, absent-command no-op behavior, operation logging,
persisted state order, and readiness timeout exclusion.
- Focused CLI and cleanup tests cover concurrent seed serialization,
stale-lock fail-closed behavior, persisted instance ownership, and
rewritten sibling pointers.

## Risks

- A faulty runtime provision script blocks service startup. Paperclip
records stderr, marks the service failed, and retries on the next start.
- Concurrent service requests share an in-process provision attempt,
while the seed command uses an atomic filesystem lock across processes.
A stale lock fails closed and requires an operator to verify no seed is
running before removing it.
- Isolated instance cleanup is destructive. The cleanup service
validates ownership and path containment before removal.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, `gpt-5.6-sol`, with agentic reasoning, tool use, and
code execution. The service does not expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-02 10:37:10 -05:00

387 lines
15 KiB
TypeScript

import { execFile } from "node:child_process";
import { createHash } from "node:crypto";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { setTimeout as delay } from "node:timers/promises";
import { promisify } from "node:util";
import { parse as parseEnvContents } from "dotenv";
import { expandHomePrefix } from "../home-paths.js";
import type { WorkspaceOperationRecorder } from "./workspace-operations.js";
const execFileAsync = promisify(execFile);
const INSTANCE_ID_RE = /^[A-Za-z0-9_-]+$/;
const POSTGRES_STOP_TIMEOUT_MS = 10_000;
export const WORKTREE_INSTANCE_ROOT_METADATA_KEY = "worktreeInstanceRoot";
export function deriveWorktreeInstanceId(workspacePath: string): string {
const resolvedWorkspacePath = path.resolve(workspacePath);
const normalized = path.basename(resolvedWorkspacePath)
.trim()
.toLowerCase()
.replace(/[^a-z0-9_-]+/g, "-")
.replace(/-+/g, "-")
.replace(/^[-_]+|[-_]+$/g, "");
const prefix = (normalized || "worktree").slice(0, 48);
const pathHash = createHash("sha256").update(resolvedWorkspacePath).digest("hex").slice(0, 12);
return `${prefix}-${pathHash}`;
}
export type WorktreeInstancePointer = {
envPath: string;
envContents: string;
};
export type WorktreeInstanceCleanupResult =
| { status: "not_configured" }
| { status: "already_absent"; instanceRoot: string }
| { status: "refused"; instanceRoot: string | null; warning: string }
| { status: "removed"; instanceRoot: string; postgresStopped: boolean };
export type WorktreeInstanceCleanupDependencies = {
stopEmbeddedPostgres: (dataDir: string) => Promise<boolean>;
removeInstanceRoot: (instanceRoot: string) => Promise<void>;
};
export type EmbeddedPostgresStopDependencies = {
processIsAlive: (pid: number) => boolean;
readVerifiedPostgresCommand: (pid: number, dataDir: string) => Promise<string | null>;
signalProcess: (pid: number, signal: NodeJS.Signals) => void;
wait: (milliseconds: number) => Promise<unknown>;
};
const defaultCleanupDependencies: WorktreeInstanceCleanupDependencies = {
stopEmbeddedPostgres: stopEmbeddedPostgresIfRunning,
removeInstanceRoot: async (instanceRoot) => {
await fs.rm(instanceRoot, { recursive: true, force: true });
},
};
const defaultPostgresStopDependencies: EmbeddedPostgresStopDependencies = {
processIsAlive,
readVerifiedPostgresCommand,
signalProcess: (pid, signal) => process.kill(pid, signal),
wait: delay,
};
function isStrictChildPath(candidatePath: string, rootPath: string): boolean {
const relative = path.relative(rootPath, candidatePath);
return relative.length > 0 && relative !== ".." && !relative.startsWith(`..${path.sep}`) && !path.isAbsolute(relative);
}
async function pathExists(value: string): Promise<boolean> {
return fs.lstat(value).then(() => true).catch(() => false);
}
function processIsAlive(pid: number): boolean {
try {
process.kill(pid, 0);
return true;
} catch (error) {
return (error as NodeJS.ErrnoException).code === "EPERM";
}
}
async function readVerifiedPostgresCommand(pid: number, dataDir: string): Promise<string | null> {
if (process.platform === "linux") {
const commandLinePath = `/proc/${pid}/cmdline`;
let commandLine: string;
try {
commandLine = await fs.readFile(commandLinePath, "utf8");
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
throw error;
}
const args = commandLine.split("\0").filter(Boolean);
const executable = path.basename(args[0] ?? "");
const dataDirFlagIndex = args.indexOf("-D");
const configuredDataDir = dataDirFlagIndex >= 0 ? args[dataDirFlagIndex + 1] : null;
if (!executable.includes("postgres") || !configuredDataDir) {
throw new Error(`Refusing to signal process ${pid}: it is not the expected embedded PostgreSQL process.`);
}
const canonicalConfiguredDataDir = await fs.realpath(configuredDataDir).catch(() => path.resolve(configuredDataDir));
if (canonicalConfiguredDataDir !== dataDir) {
throw new Error(`Refusing to signal process ${pid}: its PostgreSQL data directory does not match ${dataDir}.`);
}
return args.join(" ");
}
if (process.platform === "win32") {
throw new Error(`Refusing to signal process ${pid}: safe PostgreSQL process verification is unavailable on Windows.`);
}
try {
const { stdout } = await execFileAsync("ps", ["-p", String(pid), "-o", "command="], { encoding: "utf8" });
const command = stdout.trim();
if (!command) return null;
if (!/(?:^|\/)postgres(?:\s|$)/.test(command) || !command.includes(dataDir)) {
throw new Error(`Refusing to signal process ${pid}: it is not the expected embedded PostgreSQL process.`);
}
return command;
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ESRCH") return null;
throw error;
}
}
export async function stopEmbeddedPostgresIfRunning(
dataDir: string,
dependencies: EmbeddedPostgresStopDependencies = defaultPostgresStopDependencies,
): Promise<boolean> {
const postmasterPidPath = path.join(dataDir, "postmaster.pid");
if (!await pathExists(postmasterPidPath)) return false;
const canonicalDataDir = await fs.realpath(dataDir);
const pidContents = await fs.readFile(postmasterPidPath, "utf8");
const pidLines = pidContents.split(/\r?\n/);
const pid = Number(pidLines[0]?.trim());
const recordedDataDir = pidLines[1]?.trim();
if (!Number.isInteger(pid) || pid <= 0 || !recordedDataDir) {
throw new Error(`Refusing to remove ${dataDir}: its postmaster.pid is malformed.`);
}
const canonicalRecordedDataDir = await fs.realpath(recordedDataDir).catch(() => path.resolve(recordedDataDir));
if (canonicalRecordedDataDir !== canonicalDataDir) {
throw new Error(`Refusing to remove ${dataDir}: postmaster.pid points at a different PostgreSQL data directory.`);
}
if (!dependencies.processIsAlive(pid)) return false;
if (await dependencies.readVerifiedPostgresCommand(pid, canonicalDataDir) === null) return false;
try {
dependencies.signalProcess(pid, "SIGINT");
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ESRCH") return false;
throw error;
}
const deadline = Date.now() + POSTGRES_STOP_TIMEOUT_MS;
while (Date.now() < deadline) {
if (!dependencies.processIsAlive(pid)) return true;
await dependencies.wait(100);
}
throw new Error(`Embedded PostgreSQL process ${pid} did not stop within ${POSTGRES_STOP_TIMEOUT_MS}ms.`);
}
export async function readWorktreeInstancePointer(workspacePath: string): Promise<WorktreeInstancePointer | null> {
const envPath = path.join(workspacePath, ".paperclip", ".env");
try {
return {
envPath,
envContents: await fs.readFile(envPath, "utf8"),
};
} catch (error) {
if ((error as NodeJS.ErrnoException).code === "ENOENT") return null;
throw error;
}
}
function resolveConfiguredInstanceRoot(pointer: WorktreeInstancePointer, expectedInstanceId?: string):
| { instanceRoot: string; instanceId: string }
| { warning: string; instanceRoot: string | null; refusalReason: string | null } {
const env = parseEnvContents(pointer.envContents);
const configuredHome = env.PAPERCLIP_HOME?.trim();
const instanceId = env.PAPERCLIP_INSTANCE_ID?.trim();
if (!configuredHome || !instanceId) {
return { warning: "", instanceRoot: null, refusalReason: null };
}
if (!INSTANCE_ID_RE.test(instanceId)) {
return {
instanceRoot: null,
warning: `Refusing worktree instance cleanup from ${pointer.envPath}: PAPERCLIP_INSTANCE_ID is not a safe path segment.`,
refusalReason: "unsafe_instance_id",
};
}
const expandedHome = expandHomePrefix(configuredHome);
if (!path.isAbsolute(expandedHome)) {
return {
instanceRoot: null,
warning: `Refusing worktree instance cleanup from ${pointer.envPath}: PAPERCLIP_HOME is not absolute.`,
refusalReason: "non_absolute_home",
};
}
const instanceRoot = path.resolve(expandedHome, "instances", instanceId);
if (expectedInstanceId && instanceId !== expectedInstanceId) {
return {
instanceRoot,
warning: `Refusing worktree instance cleanup from ${pointer.envPath}: PAPERCLIP_INSTANCE_ID "${instanceId}" does not match the expected workspace instance "${expectedInstanceId}".`,
refusalReason: "instance_id_mismatch",
};
}
return { instanceRoot, instanceId };
}
function resolveManagedInstancesDir(worktreesDir?: string): string {
const managedWorktreesDir = path.resolve(
expandHomePrefix(
worktreesDir?.trim()
|| process.env.PAPERCLIP_WORKTREES_DIR?.trim()
|| path.join(os.homedir(), ".paperclip-worktrees"),
),
);
return path.join(managedWorktreesDir, "instances");
}
export async function readManagedWorktreeInstanceOwnership(
workspacePath: string,
worktreesDir?: string,
): Promise<{ instanceRoot: string; instanceId: string } | null> {
const pointer = await readWorktreeInstancePointer(workspacePath);
if (!pointer) return null;
const configured = resolveConfiguredInstanceRoot(pointer);
if ("warning" in configured) {
if (!configured.warning) return null;
throw new Error(configured.warning);
}
const managedInstancesDir = resolveManagedInstancesDir(worktreesDir);
if (!isStrictChildPath(configured.instanceRoot, managedInstancesDir)) {
throw new Error(
`Refusing to record worktree instance ownership for "${configured.instanceRoot}" because it is outside "${managedInstancesDir}".`,
);
}
return configured;
}
export async function cleanupWorktreeInstanceArtifacts(input: {
pointer: WorktreeInstancePointer;
workspaceId: string;
workspacePath: string;
expectedInstanceId: string;
expectedInstanceRoot: string | null;
recorder?: WorkspaceOperationRecorder | null;
worktreesDir?: string;
dependencies?: WorktreeInstanceCleanupDependencies;
}): Promise<WorktreeInstanceCleanupResult> {
const configured = resolveConfiguredInstanceRoot(input.pointer, input.expectedInstanceId);
if ("warning" in configured && !configured.warning) return { status: "not_configured" };
const managedInstancesDir = resolveManagedInstancesDir(input.worktreesDir);
const managedWorktreesDir = path.dirname(managedInstancesDir);
const configuredInstanceRoot = configured.instanceRoot;
let warning = "warning" in configured ? configured.warning : "";
const recordRefusal = async (
metadata: Record<string, unknown>,
refusalWarning = warning,
) => {
if (!input.recorder) return;
await input.recorder.recordOperation({
phase: "workspace_teardown",
cwd: input.workspacePath,
metadata: {
workspaceId: input.workspaceId,
workspacePath: input.workspacePath,
instanceRoot: configuredInstanceRoot,
managedInstancesDir,
cleanupAction: "remove_worktree_instance",
...metadata,
},
run: async () => ({ status: "skipped", system: `${refusalWarning}\n` }),
});
};
if ("warning" in configured) {
await recordRefusal({ refusalReason: configured.refusalReason }, configured.warning);
return { status: "refused", instanceRoot: configured.instanceRoot, warning: configured.warning };
}
if (!configuredInstanceRoot || !isStrictChildPath(configuredInstanceRoot, managedInstancesDir)) {
warning ||= `Refusing to remove instance directory "${configuredInstanceRoot ?? "unknown"}" because it is outside "${managedInstancesDir}".`;
await recordRefusal({ refusalReason: "outside_managed_instances_dir" });
return { status: "refused", instanceRoot: configuredInstanceRoot, warning };
}
const expectedInstanceRoot = input.expectedInstanceRoot
? path.resolve(input.expectedInstanceRoot)
: null;
if (expectedInstanceRoot && configuredInstanceRoot !== expectedInstanceRoot) {
warning = `Refusing to remove instance directory "${configuredInstanceRoot}" because it does not match execution workspace ${input.workspaceId}'s persisted instance root "${expectedInstanceRoot}".`;
await recordRefusal({
expectedInstanceRoot,
refusalReason: "instance_root_workspace_mismatch",
});
return { status: "refused", instanceRoot: configuredInstanceRoot, warning };
}
if (!await pathExists(configuredInstanceRoot)) {
return { status: "already_absent", instanceRoot: configuredInstanceRoot };
}
let canonicalManagedWorktreesDir: string;
let canonicalManagedInstancesDir: string;
let canonicalInstanceRoot: string;
try {
[canonicalManagedWorktreesDir, canonicalManagedInstancesDir, canonicalInstanceRoot] = await Promise.all([
fs.realpath(managedWorktreesDir),
fs.realpath(managedInstancesDir),
fs.realpath(configuredInstanceRoot),
]);
} catch (error) {
warning = `Refusing to remove instance directory "${configuredInstanceRoot}" because its canonical path could not be verified: ${error instanceof Error ? error.message : String(error)}`;
await recordRefusal({ refusalReason: "canonical_path_unavailable" });
return { status: "refused", instanceRoot: configuredInstanceRoot, warning };
}
if (canonicalManagedInstancesDir !== path.join(canonicalManagedWorktreesDir, "instances")) {
warning = `Refusing to remove instance directory "${configuredInstanceRoot}" because the managed instances directory resolves outside "${canonicalManagedWorktreesDir}".`;
await recordRefusal({
canonicalInstanceRoot,
canonicalManagedInstancesDir,
refusalReason: "managed_instances_dir_symlink",
});
return { status: "refused", instanceRoot: configuredInstanceRoot, warning };
}
if (!isStrictChildPath(canonicalInstanceRoot, canonicalManagedInstancesDir)) {
warning = `Refusing to remove instance directory "${configuredInstanceRoot}" because its canonical path "${canonicalInstanceRoot}" is outside "${canonicalManagedInstancesDir}".`;
await recordRefusal({
canonicalInstanceRoot,
canonicalManagedInstancesDir,
refusalReason: "canonical_path_outside_managed_instances_dir",
});
return { status: "refused", instanceRoot: configuredInstanceRoot, warning };
}
const dependencies = input.dependencies ?? defaultCleanupDependencies;
let postgresStopped = false;
const cleanup = async () => {
postgresStopped = await dependencies.stopEmbeddedPostgres(path.join(canonicalInstanceRoot, "db"));
const [currentManagedInstancesDir, currentInstanceRoot] = await Promise.all([
fs.realpath(managedInstancesDir),
fs.realpath(configuredInstanceRoot),
]);
if (
currentManagedInstancesDir !== canonicalManagedInstancesDir
|| currentInstanceRoot !== canonicalInstanceRoot
|| !isStrictChildPath(currentInstanceRoot, currentManagedInstancesDir)
) {
throw new Error(`Refusing to remove instance directory "${configuredInstanceRoot}" because its canonical path changed during cleanup.`);
}
await dependencies.removeInstanceRoot(currentInstanceRoot);
};
if (input.recorder) {
await input.recorder.recordOperation({
phase: "workspace_teardown",
cwd: input.workspacePath,
metadata: {
workspaceId: input.workspaceId,
workspacePath: input.workspacePath,
instanceRoot: canonicalInstanceRoot,
managedInstancesDir: canonicalManagedInstancesDir,
cleanupAction: "remove_worktree_instance",
},
run: async () => {
await cleanup();
return {
status: "succeeded",
system: `Removed worktree instance directory ${canonicalInstanceRoot}\n`,
metadata: { postgresStopped },
};
},
});
} else {
await cleanup();
}
return { status: "removed", instanceRoot: canonicalInstanceRoot, postgresStopped };
}