fix(workspaces): keep deferred seed databases reliable (#11706)

## Thinking Path

> - Paperclip manages agent work in isolated execution workspaces.
> - A workspace depends on a valid database seed before it can run.
> - Deferred seed failures were hidden behind a successful provision
status.
> - The seed restore also had two possible owners for the embedded
PostgreSQL process.
> - That allowed the target database to stop while the restore was still
running.
> - This pull request makes seed failures visible and gives the seed
process sole lifecycle ownership.
> - The benefit is that workspace provisioning reports the real result
and does not stop its own target database.

## Linked Issues or Issue Description

Related: #11684

**What happened?**

Initial worktree provisioning could report success before its deferred
database seed completed. The seed restore could also reuse a target
embedded PostgreSQL process with another shutdown owner. This could stop
the target database during the restore.

**Expected behavior**

Workspace status must show a failed deferred seed as a failure. The seed
restore must own the target embedded PostgreSQL process until restore,
migration, and validation finish.

**Steps to reproduce**

1. Provision a worktree with deferred database seeding.
2. Make the seed manifest end in a failed state while the command exits
with code 0.
3. Observe that the provision status remains successful on `master`.
4. Start a seed restore against an already-running target embedded
PostgreSQL process.
5. Observe that another lifecycle owner can stop the target during
restore.

**Paperclip version or commit**

`51a843e135`

**Deployment mode**

Local dev with execution workspaces and embedded PostgreSQL.

## What Changed

- Add a first-class `workspace_seed` operation for deferred database
seeds.
- Require terminal, verified seed evidence before the seed operation
succeeds.
- Surface the seed phase and failure metadata in workspace status and UI
state.
- Give the seed process exclusive lifecycle ownership of the target
embedded PostgreSQL process.
- Suppress imported embedded-Postgres exit hooks without removing
existing host listeners.
- Record a credential-safe shutdown diagnostic in failed seed manifests.

## Verification

- The original deferred-seed commit passed 4 server tests, 24
workspace-status UI tests, shared/server/UI typechecks, and the UI token
gate.
- The original PostgreSQL-lifecycle commit passed 3 lifecycle tests, 3
ownership/diagnostic tests, 1 real embedded-Postgres seed integration,
and the affected package typechecks.
- No local tests were rerun after the clean cherry-pick because the
operator requested the shortest landing path.
- Review the automatic PR checks for the clean `origin/master` replay.

## Risks

- A live target database now causes an early error instead of being
reused. The error includes recovery guidance.
- Workspace consumers must handle the new `workspace_seed` operation
type. Shared types and UI state handling are updated in this pull
request.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5, high-reasoning mode, with repository, shell, and
GitHub tool use. The runtime does not expose a more specific deployment
suffix or context-window value.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

cli/src/__tests__/worktree.test.ts

@@ -27,7 +27,9 @@ import {
 import {
   copyGitHooksToWorktreeGitDir,
   copySeededSecretsKey,
+  ensureEmbeddedPostgres,
   ensureWorktreeSeeded,
+  formatWorktreeSeedFailureDiagnostic,
   markWorktreeSeedPending,
   pauseSeededScheduledRoutines,
   quarantineSeededWorktreeExecutionState,
@@ -488,6 +490,34 @@ describe("worktree helpers", () => {
     expect(full.nullifyColumns).toEqual({});
   });
 
+  it("requires the seed process to own the target embedded Postgres lifecycle", async () => {
+    const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-live-target-"));
+    try {
+      fs.writeFileSync(
+        path.join(tempRoot, "postmaster.pid"),
+        `${process.pid}\n${tempRoot}\n0\n55432\n`,
+      );
+
+      await expect(ensureEmbeddedPostgres(tempRoot, 55432, { allowExisting: false }))
+        .rejects.toThrow("while it is already running");
+    } finally {
+      fs.rmSync(tempRoot, { recursive: true, force: true });
+    }
+  });
+
+  it("surfaces a credential-safe diagnostic when the target shuts down during restore", () => {
+    expect(formatWorktreeSeedFailureDiagnostic(
+      "restore",
+      new Error(
+        "Failed to restore seed.sql.gz: FATAL: the database system is shutting down; psql error: write EPIPE",
+      ),
+    )).toBe(
+      "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed.",
+    );
+    expect(formatWorktreeSeedFailureDiagnostic("migrations", new Error("secret connection failure")))
+      .toBe("Seed failed during migrations.");
+  });
+
   it("rejects a source migration journal that diverges from the code journal", () => {
     expect(() => resolveWorktreeSeedMigrationRevision({
       status: "upToDate",
@@ -714,7 +744,7 @@ describe("worktree helpers", () => {
     },
   );
 
-  it("ensure-seeded keeps the pending marker when seeding fails", async () => {
+  it("ensure-seeded records a target shutdown diagnostic when restore fails", async () => {
     const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-failure-"));
     try {
       const sourceConfigPath = path.join(tempRoot, "source", "config.json");
@@ -746,13 +776,28 @@ describe("worktree helpers", () => {
       await expect(
         ensureWorktreeSeeded(
           { config: targetConfigPath, fromConfig: sourceConfigPath },
-          { seedDatabase: vi.fn().mockRejectedValue(new Error("seed failed")) },
+          {
+            seedDatabase: vi.fn(async (input) => {
+              input.onPhase?.("restore", "started");
+              throw new Error(
+                "Failed to restore seed.sql.gz: FATAL: the database system is shutting down; psql error: write EPIPE",
+              );
+            }),
+          },
         ),
-      ).rejects.toThrow("seed failed");
+      ).rejects.toThrow("database system is shutting down");
 
       expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({
         state: "failed",
-        phase: "pending",
+        phase: "restore",
+        diagnostics: expect.arrayContaining([
+          expect.objectContaining({
+            phase: "restore",
+            status: "failed",
+            message:
+              "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed.",
+          }),
+        ]),
       });
       expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(false);
       expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(false);

cli/src/commands/worktree.ts

@@ -55,6 +55,7 @@ import {
   workspaceRuntimeServices,
   createEmbeddedPostgresLogBuffer,
   formatEmbeddedPostgresError,
+  loadWithoutEmbeddedPostgresExitHooks,
   prepareEmbeddedPostgresNativeRuntime,
 } from "@paperclipai/db";
 import type { Command } from "commander";
@@ -1105,11 +1106,15 @@ export function copySeededSecretsKey(input: {
   }
 }
 
-async function ensureEmbeddedPostgres(dataDir: string, preferredPort: number): Promise<EmbeddedPostgresHandle> {
+export async function ensureEmbeddedPostgres(
+  dataDir: string,
+  preferredPort: number,
+  options: { allowExisting?: boolean } = {},
+): Promise<EmbeddedPostgresHandle> {
   const moduleName = "embedded-postgres";
   let EmbeddedPostgres: EmbeddedPostgresCtor;
   try {
-    const mod = await import(moduleName);
+    const mod = await loadWithoutEmbeddedPostgresExitHooks(() => import(moduleName));
     EmbeddedPostgres = mod.default as EmbeddedPostgresCtor;
   } catch {
     throw new Error(
@@ -1121,6 +1126,12 @@ async function ensureEmbeddedPostgres(dataDir: string, preferredPort: number): P
   const postmasterPidFile = path.resolve(dataDir, "postmaster.pid");
   const runningPid = readRunningPostmasterPid(postmasterPidFile);
   if (runningPid) {
+    if (options.allowExisting === false) {
+      throw new Error(
+        `Cannot seed target embedded PostgreSQL at ${dataDir} while it is already running (pid=${runningPid}). `
+        + "Stop the worktree service that owns this database, then retry the seed.",
+      );
+    }
     return {
       port: readPidFilePort(postmasterPidFile) ?? preferredPort,
       startedByThisProcess: false,
@@ -1632,13 +1643,14 @@ async function seedWorktreeDatabase(input: {
     });
     input.onPhase?.("snapshot", "succeeded", `Created ${path.basename(backup.backupFile)}.`);
 
+    input.onPhase?.("restore", "started");
     targetHandle = await ensureEmbeddedPostgres(
       input.targetConfig.database.embeddedPostgresDataDir,
       input.targetConfig.database.embeddedPostgresPort,
+      { allowExisting: false },
     );
 
     const adminConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${targetHandle.port}/postgres`;
-    input.onPhase?.("restore", "started");
     await resetPostgresDatabase(adminConnectionString, "paperclip");
     const targetConnectionString = `postgres://paperclip:paperclip@127.0.0.1:${targetHandle.port}/paperclip`;
     await runDatabaseRestore({
@@ -1703,6 +1715,23 @@ const WORKTREE_SEED_DIAGNOSTIC_LIMIT = 32;
 const WORKTREE_SEED_DIAGNOSTIC_MESSAGE_LIMIT = 512;
 const activeSeedInterruptHandlers = new Map<string, (signal: NodeJS.Signals) => void>();
 
+export function formatWorktreeSeedFailureDiagnostic(
+  phase: WorktreeSeedPhase,
+  error: unknown,
+): string {
+  const message = error instanceof Error ? error.message : String(error ?? "");
+  if (
+    phase === "restore"
+    && /database system is shutting down|terminating connection due to administrator command/i.test(message)
+  ) {
+    return "Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed.";
+  }
+  if (phase === "restore" && /Cannot seed target embedded PostgreSQL.+already running/i.test(message)) {
+    return "Target embedded PostgreSQL is owned by a running worktree service. Stop that service and retry the seed.";
+  }
+  return `Seed failed during ${phase}.`;
+}
+
 function dispatchSeedInterruption(signal: NodeJS.Signals): void {
   for (const handler of activeSeedInterruptHandlers.values()) {
     try {
@@ -2096,7 +2125,7 @@ async function runVerifiedWorktreeSeed(input: {
       state: "failed",
       // Do not persist the underlying error: database/driver errors may contain
       // connection credentials. The CLI still returns the exact error to its caller.
-      message: `Seed failed during ${activePhase}.`,
+      message: formatWorktreeSeedFailureDiagnostic(activePhase, error),
     });
     throw error;
   } finally {