mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-06 10:48:12 +02:00
feat(workspaces): defer isolated setup until runtime start (#10653)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Isolated workspaces give each task a safe and reproducible checkout. > - The existing setup cloned the development database before an agent needed to run the app. > - This made worktree creation slower and heavier for tasks that never start a service. > - Runtime services already use one server start path for heartbeat, operator, and startup recovery flows. > - This pull request moves heavy setup to that start path and keeps worktree creation lean. > - The benefit is faster isolated workspace creation with the same reliable runtime setup when a service starts. ## Linked Issues or Issue Description Related pull request: #10652 covers the initial deferred database-seeding slice. This pull request supersedes it with end-to-end runtime provisioning and safe cleanup. **What existing behavior does this improve?** This improves isolated worktree creation, runtime service startup, and isolated instance cleanup. **Subsystem affected** Cross-cutting: CLI worktree setup, server runtime orchestration, shared workspace contracts, and development scripts. **Current behavior** Paperclip seeds an isolated development database during worktree creation. It can also leave an isolated instance directory after workspace teardown. This work happens even when no runtime service starts. **Proposed behavior** Paperclip creates the worktree with a lean eager setup. It runs an idempotent runtime provision command before the first managed service spawn. Concurrent starts share one provision attempt. Teardown removes the isolated instance safely. **Reason and benefit** Many agent tasks only edit and test code. They do not need a running Paperclip instance. Deferring the database seed reduces workspace startup cost while preserving automatic setup for tasks that start the app. **Breaking changes** None. The new runtime provision command is optional. Existing workspace behavior is unchanged when it is absent. ## What Changed - Split Paperclip worktree setup into a lean eager script and an idempotent runtime provision script. - Added `runtimeProvisionCommand` to project, issue, realized workspace, and persisted workspace contracts. - Added a per-workspace provision mutex before local service spawn for heartbeat, operator, and startup recovery flows. - Added a persisted `provisioning` service state and the `workspace_runtime_provision` operation phase. - Kept provision time outside the service readiness timeout and made failed attempts visible and retryable. - Reclaimed isolated instance data during safe workspace teardown. - Serialized deferred database seeding across processes and bound teardown to the instance root captured in persisted workspace metadata. - Added tests for config flow, concurrency, retry, no-op behavior, readiness timing, scripts, CLI commands, and cleanup. - Documented the eager and runtime provisioning contracts. ## Verification - `pnpm -r typecheck` - `pnpm build` - `pnpm test:run` (server: 3,201 passed; UI: 3,345 passed; the CLI phase exposed one environment-sensitive AWS doctor assertion because the agent runtime injects static AWS credentials) - `env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY pnpm exec vitest run cli/src/__tests__/secrets.test.ts -t 'passes AWS doctor checks when non-secret provider config is present'` - Focused runtime tests cover serialized provisioning, retry after stderr failure, absent-command no-op behavior, operation logging, persisted state order, and readiness timeout exclusion. - Focused CLI and cleanup tests cover concurrent seed serialization, stale-lock fail-closed behavior, persisted instance ownership, and rewritten sibling pointers. ## Risks - A faulty runtime provision script blocks service startup. Paperclip records stderr, marks the service failed, and retries on the next start. - Concurrent service requests share an in-process provision attempt, while the seed command uses an atomic filesystem lock across processes. A stale lock fails closed and requires an operator to verify no seed is running before removing it. - Isolated instance cleanup is destructive. The cleanup service validates ownership and path containment before removal. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex, `gpt-5.6-sol`, with agentic reasoning, tool use, and code execution. The service does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
47 files changed
+2167
-95
No files matched your search
@@ -20,6 +20,8 @@ import {
|
||||
import {
|
||||
copyGitHooksToWorktreeGitDir,
|
||||
copySeededSecretsKey,
|
||||
ensureWorktreeSeeded,
|
||||
markWorktreeSeedPending,
|
||||
pauseSeededScheduledRoutines,
|
||||
quarantineSeededWorktreeExecutionState,
|
||||
readSourceAttachmentBody,
|
||||
@@ -351,6 +353,198 @@ describe("worktree helpers", () => {
|
||||
expect(full.nullifyColumns).toEqual({});
|
||||
});
|
||||
|
||||
it("ensure-seeded seeds once and fast-exits on the seed-complete marker", async () => {
|
||||
const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-"));
|
||||
try {
|
||||
const sourceConfigPath = path.join(tempRoot, "source", "config.json");
|
||||
const targetRoot = path.join(tempRoot, "worktree");
|
||||
const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json");
|
||||
const targetPaths = resolveWorktreeLocalPaths({
|
||||
cwd: targetRoot,
|
||||
homeDir: path.join(tempRoot, "worktree-home"),
|
||||
instanceId: "ensure-seeded-test",
|
||||
});
|
||||
const sourceConfig = buildSourceConfig();
|
||||
const targetConfig = buildWorktreeConfig({
|
||||
sourceConfig,
|
||||
paths: targetPaths,
|
||||
serverPort: 3199,
|
||||
databasePort: 54999,
|
||||
});
|
||||
fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true });
|
||||
fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true });
|
||||
fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`);
|
||||
fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`);
|
||||
fs.writeFileSync(
|
||||
path.join(targetRoot, ".paperclip", ".env"),
|
||||
`PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`,
|
||||
);
|
||||
markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath });
|
||||
|
||||
const seedDatabase = vi.fn().mockResolvedValue({
|
||||
backupSummary: "snapshot.sql",
|
||||
pausedScheduledRoutines: 2,
|
||||
executionQuarantine: {
|
||||
disabledTimerHeartbeats: 1,
|
||||
resetRunningAgents: 1,
|
||||
quarantinedInProgressIssues: 1,
|
||||
unassignedTodoIssues: 1,
|
||||
unassignedReviewIssues: 1,
|
||||
},
|
||||
reboundWorkspaces: [],
|
||||
});
|
||||
|
||||
await expect(
|
||||
ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }),
|
||||
).resolves.toMatchObject({ seeded: true, reason: "seeded" });
|
||||
await expect(
|
||||
ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }),
|
||||
).resolves.toEqual({ seeded: false, reason: "complete_marker" });
|
||||
|
||||
expect(seedDatabase).toHaveBeenCalledTimes(1);
|
||||
expect(seedDatabase).toHaveBeenCalledWith(expect.objectContaining({
|
||||
sourceConfigPath,
|
||||
seedMode: "minimal",
|
||||
instanceId: "ensure-seeded-test",
|
||||
}));
|
||||
expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(false);
|
||||
expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(true);
|
||||
} finally {
|
||||
fs.rmSync(tempRoot, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("ensure-seeded keeps the pending marker when seeding fails", async () => {
|
||||
const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-failure-"));
|
||||
try {
|
||||
const sourceConfigPath = path.join(tempRoot, "source", "config.json");
|
||||
const targetRoot = path.join(tempRoot, "worktree");
|
||||
const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json");
|
||||
const targetPaths = resolveWorktreeLocalPaths({
|
||||
cwd: targetRoot,
|
||||
homeDir: path.join(tempRoot, "worktree-home"),
|
||||
instanceId: "ensure-seeded-failure",
|
||||
});
|
||||
const sourceConfig = buildSourceConfig();
|
||||
const targetConfig = buildWorktreeConfig({
|
||||
sourceConfig,
|
||||
paths: targetPaths,
|
||||
serverPort: 3198,
|
||||
databasePort: 54998,
|
||||
});
|
||||
fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true });
|
||||
fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true });
|
||||
fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`);
|
||||
fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`);
|
||||
fs.writeFileSync(
|
||||
path.join(targetRoot, ".paperclip", ".env"),
|
||||
`PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`,
|
||||
);
|
||||
markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath });
|
||||
|
||||
await expect(
|
||||
ensureWorktreeSeeded(
|
||||
{ config: targetConfigPath },
|
||||
{ seedDatabase: vi.fn().mockRejectedValue(new Error("seed failed")) },
|
||||
),
|
||||
).rejects.toThrow("seed failed");
|
||||
|
||||
expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(true);
|
||||
expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(false);
|
||||
expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed.lock"))).toBe(false);
|
||||
} finally {
|
||||
fs.rmSync(tempRoot, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("serializes concurrent ensure-seeded calls across the seed marker lock", async () => {
|
||||
const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-lock-"));
|
||||
try {
|
||||
const sourceConfigPath = path.join(tempRoot, "source", "config.json");
|
||||
const targetRoot = path.join(tempRoot, "worktree");
|
||||
const targetConfigPath = path.join(targetRoot, ".paperclip", "config.json");
|
||||
const targetPaths = resolveWorktreeLocalPaths({
|
||||
cwd: targetRoot,
|
||||
homeDir: path.join(tempRoot, "worktree-home"),
|
||||
instanceId: "ensure-seeded-lock",
|
||||
});
|
||||
const sourceConfig = buildSourceConfig();
|
||||
const targetConfig = buildWorktreeConfig({
|
||||
sourceConfig,
|
||||
paths: targetPaths,
|
||||
serverPort: 3197,
|
||||
databasePort: 54997,
|
||||
});
|
||||
fs.mkdirSync(path.dirname(sourceConfigPath), { recursive: true });
|
||||
fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true });
|
||||
fs.writeFileSync(sourceConfigPath, `${JSON.stringify(sourceConfig)}\n`);
|
||||
fs.writeFileSync(targetConfigPath, `${JSON.stringify(targetConfig)}\n`);
|
||||
fs.writeFileSync(
|
||||
path.join(targetRoot, ".paperclip", ".env"),
|
||||
`PAPERCLIP_HOME=${targetPaths.homeDir}\nPAPERCLIP_INSTANCE_ID=${targetPaths.instanceId}\n`,
|
||||
);
|
||||
markWorktreeSeedPending({ configPath: targetConfigPath, sourceConfigPath });
|
||||
|
||||
const seedDatabase = vi.fn(async () => {
|
||||
await new Promise((resolve) => setTimeout(resolve, 100));
|
||||
return {
|
||||
backupSummary: "snapshot.sql",
|
||||
pausedScheduledRoutines: 0,
|
||||
executionQuarantine: {
|
||||
disabledTimerHeartbeats: 0,
|
||||
resetRunningAgents: 0,
|
||||
quarantinedInProgressIssues: 0,
|
||||
unassignedTodoIssues: 0,
|
||||
unassignedReviewIssues: 0,
|
||||
},
|
||||
reboundWorkspaces: [],
|
||||
};
|
||||
});
|
||||
|
||||
const results = await Promise.all([
|
||||
ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }),
|
||||
ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }),
|
||||
]);
|
||||
|
||||
expect(results).toEqual(expect.arrayContaining([
|
||||
expect.objectContaining({ seeded: true, reason: "seeded" }),
|
||||
{ seeded: false, reason: "complete_marker" },
|
||||
]));
|
||||
expect(seedDatabase).toHaveBeenCalledTimes(1);
|
||||
expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed.lock"))).toBe(false);
|
||||
} finally {
|
||||
fs.rmSync(tempRoot, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("fails closed instead of racing to reclaim a stale seed lock", async () => {
|
||||
const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-stale-lock-"));
|
||||
try {
|
||||
const targetConfigPath = path.join(tempRoot, ".paperclip", "config.json");
|
||||
const lockPath = path.join(tempRoot, ".paperclip", "seed.lock");
|
||||
fs.mkdirSync(path.dirname(targetConfigPath), { recursive: true });
|
||||
fs.writeFileSync(
|
||||
lockPath,
|
||||
`${JSON.stringify({
|
||||
version: 1,
|
||||
pid: 2_147_483_647,
|
||||
token: "stale-owner",
|
||||
createdAt: new Date(0).toISOString(),
|
||||
})}\n`,
|
||||
);
|
||||
const seedDatabase = vi.fn();
|
||||
|
||||
await expect(
|
||||
ensureWorktreeSeeded({ config: targetConfigPath }, { seedDatabase }),
|
||||
).rejects.toThrow("belongs to exited process");
|
||||
|
||||
expect(seedDatabase).not.toHaveBeenCalled();
|
||||
expect(fs.existsSync(lockPath)).toBe(true);
|
||||
} finally {
|
||||
fs.rmSync(tempRoot, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
itEmbeddedPostgres("quarantines copied live execution state in seeded worktree databases", async () => {
|
||||
const tempDb = await startEmbeddedPostgresTestDatabase("paperclip-worktree-quarantine-");
|
||||
const db = createDb(tempDb.connectionString);
|
||||
|
||||
@@ -18,6 +18,7 @@ import {
|
||||
} from "../config/home.js";
|
||||
import { assertForegroundRunAllowed } from "../services/service-manager.js";
|
||||
import { printUpdateNotice } from "../update-notice.js";
|
||||
import { ensureWorktreeSeeded } from "./worktree.js";
|
||||
|
||||
interface RunOptions {
|
||||
config?: string;
|
||||
@@ -67,6 +68,11 @@ export async function runCommand(opts: RunOptions): Promise<void> {
|
||||
await onboard({ config: configPath, invokedByRun: true, bind: opts.bind });
|
||||
}
|
||||
|
||||
const seedResult = await ensureWorktreeSeeded({ config: configPath });
|
||||
if (seedResult.seeded) {
|
||||
p.log.success("Completed deferred worktree database seed.");
|
||||
}
|
||||
|
||||
p.log.step("Running doctor checks...");
|
||||
const summary = await doctor({
|
||||
config: configPath,
|
||||
|
||||
@@ -5,6 +5,9 @@ import { expandHomePrefix } from "../config/home.js";
|
||||
|
||||
export const DEFAULT_WORKTREE_HOME = "~/.paperclip-worktrees";
|
||||
export const WORKTREE_SEED_MODES = ["minimal", "full"] as const;
|
||||
export const WORKTREE_SEED_PENDING_MARKER = "seed-pending";
|
||||
export const WORKTREE_SEED_COMPLETE_MARKER = "seed-complete";
|
||||
export const WORKTREE_SEED_LOCK_MARKER = "seed.lock";
|
||||
|
||||
export type WorktreeSeedMode = (typeof WORKTREE_SEED_MODES)[number];
|
||||
|
||||
@@ -50,6 +53,21 @@ export type WorktreeUiBranding = {
|
||||
color: string;
|
||||
};
|
||||
|
||||
export type WorktreeSeedMarkerPaths = {
|
||||
pending: string;
|
||||
complete: string;
|
||||
lock: string;
|
||||
};
|
||||
|
||||
export function resolveWorktreeSeedMarkerPaths(configPath: string): WorktreeSeedMarkerPaths {
|
||||
const configDir = path.dirname(path.resolve(configPath));
|
||||
return {
|
||||
pending: path.resolve(configDir, WORKTREE_SEED_PENDING_MARKER),
|
||||
complete: path.resolve(configDir, WORKTREE_SEED_COMPLETE_MARKER),
|
||||
lock: path.resolve(configDir, WORKTREE_SEED_LOCK_MARKER),
|
||||
};
|
||||
}
|
||||
|
||||
export function isWorktreeSeedMode(value: string): value is WorktreeSeedMode {
|
||||
return (WORKTREE_SEED_MODES as readonly string[]).includes(value);
|
||||
}
|
||||
|
||||
@@ -15,6 +15,7 @@ import {
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { createServer } from "node:net";
|
||||
import { Readable } from "node:stream";
|
||||
import * as p from "@clack/prompts";
|
||||
@@ -64,6 +65,7 @@ import {
|
||||
isWorktreeSeedMode,
|
||||
resolveSuggestedWorktreeName,
|
||||
resolveWorktreeSeedPlan,
|
||||
resolveWorktreeSeedMarkerPaths,
|
||||
resolveWorktreeLocalPaths,
|
||||
sanitizeWorktreeInstanceId,
|
||||
type WorktreeSeedPlan,
|
||||
@@ -147,6 +149,14 @@ type WorktreeRepairOptions = {
|
||||
allowLiveTarget?: boolean;
|
||||
};
|
||||
|
||||
type WorktreeEnsureSeededOptions = {
|
||||
config?: string;
|
||||
fromConfig?: string;
|
||||
fromDataDir?: string;
|
||||
fromInstance?: string;
|
||||
preserveLiveWork?: boolean;
|
||||
};
|
||||
|
||||
type EmbeddedPostgresInstance = {
|
||||
initialise(): Promise<void>;
|
||||
start(): Promise<void>;
|
||||
@@ -194,6 +204,29 @@ type SeedWorktreeDatabaseResult = {
|
||||
}>;
|
||||
};
|
||||
|
||||
type WorktreeSeedPendingMarker = {
|
||||
version: 1;
|
||||
state: "pending";
|
||||
sourceConfigPath: string;
|
||||
seedMode: "minimal";
|
||||
createdAt: string;
|
||||
};
|
||||
|
||||
type WorktreeSeedCompleteMarker = {
|
||||
version: 1;
|
||||
state: "complete";
|
||||
seedMode: WorktreeSeedMode;
|
||||
completedAt: string;
|
||||
};
|
||||
|
||||
type SeedWorktreeDatabase = typeof seedWorktreeDatabase;
|
||||
|
||||
export type EnsureWorktreeSeededResult = {
|
||||
seeded: boolean;
|
||||
reason: "seeded" | "complete_marker" | "legacy_unmarked";
|
||||
details?: SeedWorktreeDatabaseResult;
|
||||
};
|
||||
|
||||
export type SeededWorktreeExecutionQuarantineSummary = {
|
||||
disabledTimerHeartbeats: number;
|
||||
resetRunningAgents: number;
|
||||
@@ -1359,6 +1392,224 @@ async function seedWorktreeDatabase(input: {
|
||||
}
|
||||
}
|
||||
|
||||
function writeWorktreeSeedMarker(
|
||||
filePath: string,
|
||||
marker: WorktreeSeedPendingMarker | WorktreeSeedCompleteMarker,
|
||||
): void {
|
||||
mkdirSync(path.dirname(filePath), { recursive: true });
|
||||
writeFileSync(filePath, `${JSON.stringify(marker, null, 2)}\n`, { mode: 0o600 });
|
||||
}
|
||||
|
||||
export function markWorktreeSeedPending(input: {
|
||||
configPath: string;
|
||||
sourceConfigPath: string;
|
||||
now?: Date;
|
||||
}): void {
|
||||
const markers = resolveWorktreeSeedMarkerPaths(input.configPath);
|
||||
rmSync(markers.complete, { force: true });
|
||||
writeWorktreeSeedMarker(markers.pending, {
|
||||
version: 1,
|
||||
state: "pending",
|
||||
sourceConfigPath: path.resolve(input.sourceConfigPath),
|
||||
seedMode: "minimal",
|
||||
createdAt: (input.now ?? new Date()).toISOString(),
|
||||
});
|
||||
}
|
||||
|
||||
export function markWorktreeSeedComplete(input: {
|
||||
configPath: string;
|
||||
seedMode?: WorktreeSeedMode;
|
||||
now?: Date;
|
||||
}): void {
|
||||
const markers = resolveWorktreeSeedMarkerPaths(input.configPath);
|
||||
writeWorktreeSeedMarker(markers.complete, {
|
||||
version: 1,
|
||||
state: "complete",
|
||||
seedMode: input.seedMode ?? "minimal",
|
||||
completedAt: (input.now ?? new Date()).toISOString(),
|
||||
});
|
||||
rmSync(markers.pending, { force: true });
|
||||
}
|
||||
|
||||
function readWorktreeSeedPendingMarker(filePath: string): WorktreeSeedPendingMarker {
|
||||
let parsed: unknown;
|
||||
try {
|
||||
parsed = JSON.parse(readFileSync(filePath, "utf8"));
|
||||
} catch (error) {
|
||||
throw new Error(
|
||||
`Invalid worktree seed-pending marker at ${filePath}: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
|
||||
if (
|
||||
!parsed
|
||||
|| typeof parsed !== "object"
|
||||
|| (parsed as { version?: unknown }).version !== 1
|
||||
|| (parsed as { state?: unknown }).state !== "pending"
|
||||
|| typeof (parsed as { sourceConfigPath?: unknown }).sourceConfigPath !== "string"
|
||||
|| !(parsed as { sourceConfigPath: string }).sourceConfigPath.trim()
|
||||
) {
|
||||
throw new Error(`Invalid worktree seed-pending marker at ${filePath}.`);
|
||||
}
|
||||
|
||||
return parsed as WorktreeSeedPendingMarker;
|
||||
}
|
||||
|
||||
const WORKTREE_SEED_LOCK_POLL_MS = 50;
|
||||
const WORKTREE_SEED_LOCK_MALFORMED_STALE_MS = 60_000;
|
||||
|
||||
type WorktreeSeedLockOwner = {
|
||||
version: 1;
|
||||
pid: number;
|
||||
token: string;
|
||||
createdAt: string;
|
||||
};
|
||||
|
||||
function processIsAlive(pid: number): boolean {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (error) {
|
||||
return (error as NodeJS.ErrnoException).code === "EPERM";
|
||||
}
|
||||
}
|
||||
|
||||
function parseWorktreeSeedLockOwner(raw: string): WorktreeSeedLockOwner | null {
|
||||
try {
|
||||
const value = JSON.parse(raw) as Partial<WorktreeSeedLockOwner>;
|
||||
if (
|
||||
value.version !== 1
|
||||
|| !Number.isInteger(value.pid)
|
||||
|| (value.pid ?? 0) <= 0
|
||||
|| typeof value.token !== "string"
|
||||
|| !value.token
|
||||
|| typeof value.createdAt !== "string"
|
||||
|| !value.createdAt
|
||||
) {
|
||||
return null;
|
||||
}
|
||||
return value as WorktreeSeedLockOwner;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function acquireWorktreeSeedLock(lockPath: string): Promise<() => Promise<void>> {
|
||||
while (true) {
|
||||
const owner: WorktreeSeedLockOwner = {
|
||||
version: 1,
|
||||
pid: process.pid,
|
||||
token: randomUUID(),
|
||||
createdAt: new Date().toISOString(),
|
||||
};
|
||||
try {
|
||||
const handle = await fsPromises.open(lockPath, "wx", 0o600);
|
||||
try {
|
||||
await handle.writeFile(`${JSON.stringify(owner)}\n`, "utf8");
|
||||
} catch (error) {
|
||||
await handle.close();
|
||||
await fsPromises.rm(lockPath, { force: true });
|
||||
throw error;
|
||||
}
|
||||
await handle.close();
|
||||
return async () => {
|
||||
const current = await fsPromises.readFile(lockPath, "utf8").catch(() => null);
|
||||
if (current && parseWorktreeSeedLockOwner(current)?.token === owner.token) {
|
||||
await fsPromises.rm(lockPath, { force: true });
|
||||
}
|
||||
};
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== "EEXIST") throw error;
|
||||
}
|
||||
|
||||
const [rawOwner, lockStat] = await Promise.all([
|
||||
fsPromises.readFile(lockPath, "utf8").catch(() => null),
|
||||
fsPromises.stat(lockPath).catch(() => null),
|
||||
]);
|
||||
const currentOwner = rawOwner ? parseWorktreeSeedLockOwner(rawOwner) : null;
|
||||
const malformedLockIsStale = Boolean(
|
||||
lockStat && Date.now() - lockStat.mtimeMs >= WORKTREE_SEED_LOCK_MALFORMED_STALE_MS,
|
||||
);
|
||||
if (currentOwner && !processIsAlive(currentOwner.pid)) {
|
||||
throw new Error(
|
||||
`Worktree seed lock ${lockPath} belongs to exited process ${currentOwner.pid}. `
|
||||
+ "Verify that no seed is running, then remove the stale lock and retry.",
|
||||
);
|
||||
}
|
||||
if (!currentOwner && malformedLockIsStale) {
|
||||
throw new Error(
|
||||
`Worktree seed lock ${lockPath} is stale or malformed. `
|
||||
+ "Verify that no seed is running, then remove the stale lock and retry.",
|
||||
);
|
||||
}
|
||||
await new Promise((resolve) => setTimeout(resolve, WORKTREE_SEED_LOCK_POLL_MS));
|
||||
}
|
||||
}
|
||||
|
||||
export async function ensureWorktreeSeeded(
|
||||
opts: WorktreeEnsureSeededOptions = {},
|
||||
dependencies: { seedDatabase?: SeedWorktreeDatabase } = {},
|
||||
): Promise<EnsureWorktreeSeededResult> {
|
||||
const configPath = resolveConfigPath(opts.config);
|
||||
const markers = resolveWorktreeSeedMarkerPaths(configPath);
|
||||
mkdirSync(path.dirname(markers.lock), { recursive: true });
|
||||
const releaseLock = await acquireWorktreeSeedLock(markers.lock);
|
||||
try {
|
||||
// These checks deliberately happen under the cross-process lock. A second
|
||||
// service process waits for the first seed transaction, then observes the
|
||||
// complete marker instead of cloning the same database concurrently.
|
||||
if (existsSync(markers.complete)) {
|
||||
return { seeded: false, reason: "complete_marker" };
|
||||
}
|
||||
if (!existsSync(markers.pending)) {
|
||||
// Worktrees created before lazy seeding shipped were seeded eagerly and
|
||||
// have neither marker. Preserve that compatibility without re-cloning.
|
||||
return { seeded: false, reason: "legacy_unmarked" };
|
||||
}
|
||||
|
||||
const pending = readWorktreeSeedPendingMarker(markers.pending);
|
||||
const sourceConfigPath = opts.fromConfig || opts.fromDataDir || opts.fromInstance
|
||||
? resolveSourceConfigPath({
|
||||
fromConfig: opts.fromConfig,
|
||||
fromDataDir: opts.fromDataDir,
|
||||
fromInstance: opts.fromInstance,
|
||||
})
|
||||
: path.resolve(pending.sourceConfigPath);
|
||||
|
||||
if (path.resolve(sourceConfigPath) === path.resolve(configPath)) {
|
||||
throw new Error(
|
||||
"Source and target Paperclip configs are the same. Pass --from-config for the source instance.",
|
||||
);
|
||||
}
|
||||
|
||||
const sourceConfig = readConfig(sourceConfigPath);
|
||||
if (!sourceConfig) {
|
||||
throw new Error(`Source config not found at ${sourceConfigPath}.`);
|
||||
}
|
||||
const targetConfig = readConfig(configPath);
|
||||
if (!targetConfig) {
|
||||
throw new Error(`Target config not found at ${configPath}.`);
|
||||
}
|
||||
|
||||
const targetRoot = path.dirname(path.dirname(configPath));
|
||||
const targetPaths = resolveWorktreeReseedTargetPaths({ configPath, rootPath: targetRoot });
|
||||
const seedDatabase = dependencies.seedDatabase ?? seedWorktreeDatabase;
|
||||
const details = await seedDatabase({
|
||||
sourceConfigPath,
|
||||
sourceConfig,
|
||||
targetConfig,
|
||||
targetPaths,
|
||||
instanceId: targetPaths.instanceId,
|
||||
seedMode: "minimal",
|
||||
preserveLiveWork: opts.preserveLiveWork,
|
||||
});
|
||||
markWorktreeSeedComplete({ configPath });
|
||||
return { seeded: true, reason: "seeded", details };
|
||||
} finally {
|
||||
await releaseLock();
|
||||
}
|
||||
}
|
||||
|
||||
export function resolveWorktreeSeedBackupEngine(seedPlan: WorktreeSeedPlan): "auto" | "javascript" {
|
||||
return seedPlan.excludedTables.length === 0 && Object.keys(seedPlan.nullifyColumns).length === 0
|
||||
? "auto"
|
||||
@@ -1401,6 +1652,9 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise<void> {
|
||||
// checkout, and a recursive rmSync here would nuke them all.
|
||||
rmSync(paths.configPath, { force: true });
|
||||
rmSync(paths.envPath, { force: true });
|
||||
const seedMarkers = resolveWorktreeSeedMarkerPaths(paths.configPath);
|
||||
rmSync(seedMarkers.pending, { force: true });
|
||||
rmSync(seedMarkers.complete, { force: true });
|
||||
rmSync(paths.instanceRoot, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
@@ -1420,6 +1674,10 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise<void> {
|
||||
});
|
||||
|
||||
writeConfig(targetConfig, paths.configPath);
|
||||
markWorktreeSeedPending({
|
||||
configPath: paths.configPath,
|
||||
sourceConfigPath,
|
||||
});
|
||||
const sourceEnvEntries = readPaperclipEnvEntries(resolvePaperclipEnvFile(sourceConfigPath));
|
||||
const existingAgentJwtSecret =
|
||||
nonEmpty(sourceEnvEntries.PAPERCLIP_AGENT_JWT_SECRET) ??
|
||||
@@ -1461,6 +1719,7 @@ async function runWorktreeInit(opts: WorktreeInitOptions): Promise<void> {
|
||||
seedExecutionQuarantineSummary = seeded.executionQuarantine;
|
||||
pausedScheduledRoutineCount = seeded.pausedScheduledRoutines;
|
||||
reboundWorkspaceSummary = seeded.reboundWorkspaces;
|
||||
markWorktreeSeedComplete({ configPath: paths.configPath, seedMode });
|
||||
spinner.stop(`Seeded isolated worktree database (${seedMode}).`);
|
||||
} catch (error) {
|
||||
spinner.stop(pc.red("Failed to seed worktree database."));
|
||||
@@ -1511,6 +1770,46 @@ export async function worktreeInitCommand(opts: WorktreeInitOptions): Promise<vo
|
||||
await runWorktreeInit(opts);
|
||||
}
|
||||
|
||||
export async function worktreeEnsureSeededCommand(opts: WorktreeEnsureSeededOptions): Promise<void> {
|
||||
printPaperclipCliBanner();
|
||||
p.intro(pc.bgCyan(pc.black(" paperclipai worktree ensure-seeded ")));
|
||||
|
||||
const markers = resolveWorktreeSeedMarkerPaths(resolveConfigPath(opts.config));
|
||||
if (existsSync(markers.complete) || !existsSync(markers.pending)) {
|
||||
const result = await ensureWorktreeSeeded(opts);
|
||||
const reason = result.reason === "complete_marker"
|
||||
? "Seed-complete marker already present."
|
||||
: "No seed-pending marker found; treating this legacy worktree as already seeded.";
|
||||
p.outro(pc.green(reason));
|
||||
return;
|
||||
}
|
||||
|
||||
const spinner = p.spinner();
|
||||
spinner.start("Seeding isolated worktree database from source instance (minimal)...");
|
||||
try {
|
||||
const result = await ensureWorktreeSeeded(opts);
|
||||
spinner.stop("Seeded isolated worktree database (minimal).");
|
||||
if (result.details) {
|
||||
p.log.message(pc.dim(`Seed snapshot: ${result.details.backupSummary}`));
|
||||
p.log.message(
|
||||
pc.dim(
|
||||
`Seed execution quarantine: ${formatSeededWorktreeExecutionQuarantineSummary(result.details.executionQuarantine)}`,
|
||||
),
|
||||
);
|
||||
p.log.message(pc.dim(`Paused scheduled routines: ${result.details.pausedScheduledRoutines}`));
|
||||
for (const rebound of result.details.reboundWorkspaces) {
|
||||
p.log.message(
|
||||
pc.dim(`Rebound workspace ${rebound.name}: ${rebound.fromCwd} -> ${rebound.toCwd}`),
|
||||
);
|
||||
}
|
||||
}
|
||||
p.outro(pc.green("Worktree database seed complete."));
|
||||
} catch (error) {
|
||||
spinner.stop(pc.red("Failed to seed worktree database."));
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
export async function worktreeMakeCommand(nameArg: string, opts: WorktreeMakeOptions): Promise<void> {
|
||||
printPaperclipCliBanner();
|
||||
p.intro(pc.bgCyan(pc.black(" paperclipai worktree:make ")));
|
||||
@@ -3158,6 +3457,7 @@ async function runWorktreeReseed(opts: WorktreeReseedOptions): Promise<void> {
|
||||
seedMode,
|
||||
preserveLiveWork: opts.preserveLiveWork,
|
||||
});
|
||||
markWorktreeSeedComplete({ configPath: targetEndpoint.configPath, seedMode });
|
||||
spinner.stop(`Reseeded ${targetEndpoint.label} (${seedMode}).`);
|
||||
p.log.message(pc.dim(`Source: ${source.configPath}`));
|
||||
p.log.message(pc.dim(`Target: ${targetEndpoint.configPath}`));
|
||||
@@ -3319,6 +3619,16 @@ export function registerWorktreeCommands(program: Command): void {
|
||||
.option("--json", "Print JSON instead of shell exports")
|
||||
.action(worktreeEnvCommand);
|
||||
|
||||
worktree
|
||||
.command("ensure-seeded")
|
||||
.description("Seed a seed-pending worktree database exactly once from its source instance")
|
||||
.option("-c, --config <path>", "Path to the target worktree config file")
|
||||
.option("--from-config <path>", "Source config.json to seed from (defaults to the seed-pending marker)")
|
||||
.option("--from-data-dir <path>", "Source PAPERCLIP_HOME used when deriving the source config")
|
||||
.option("--from-instance <id>", "Source instance id when deriving the source config")
|
||||
.option("--preserve-live-work", "Do not quarantine copied agent timers or assigned open issues", false)
|
||||
.action(worktreeEnsureSeededCommand);
|
||||
|
||||
program
|
||||
.command("worktree:list")
|
||||
.description("List git worktrees visible from this repo and whether they look like Paperclip worktrees")
|
||||
|
||||
Reference in new issue
Block a user