fix(workspaces): keep deferred seed databases reliable (#11706)

## Thinking Path

> - Paperclip manages agent work in isolated execution workspaces.
> - A workspace depends on a valid database seed before it can run.
> - Deferred seed failures were hidden behind a successful provision
status.
> - The seed restore also had two possible owners for the embedded
PostgreSQL process.
> - That allowed the target database to stop while the restore was still
running.
> - This pull request makes seed failures visible and gives the seed
process sole lifecycle ownership.
> - The benefit is that workspace provisioning reports the real result
and does not stop its own target database.

## Linked Issues or Issue Description

Related: #11684

**What happened?**

Initial worktree provisioning could report success before its deferred
database seed completed. The seed restore could also reuse a target
embedded PostgreSQL process with another shutdown owner. This could stop
the target database during the restore.

**Expected behavior**

Workspace status must show a failed deferred seed as a failure. The seed
restore must own the target embedded PostgreSQL process until restore,
migration, and validation finish.

**Steps to reproduce**

1. Provision a worktree with deferred database seeding.
2. Make the seed manifest end in a failed state while the command exits
with code 0.
3. Observe that the provision status remains successful on `master`.
4. Start a seed restore against an already-running target embedded
PostgreSQL process.
5. Observe that another lifecycle owner can stop the target during
restore.

**Paperclip version or commit**

`51a843e135`

**Deployment mode**

Local dev with execution workspaces and embedded PostgreSQL.

## What Changed

- Add a first-class `workspace_seed` operation for deferred database
seeds.
- Require terminal, verified seed evidence before the seed operation
succeeds.
- Surface the seed phase and failure metadata in workspace status and UI
state.
- Give the seed process exclusive lifecycle ownership of the target
embedded PostgreSQL process.
- Suppress imported embedded-Postgres exit hooks without removing
existing host listeners.
- Record a credential-safe shutdown diagnostic in failed seed manifests.

## Verification

- The original deferred-seed commit passed 4 server tests, 24
workspace-status UI tests, shared/server/UI typechecks, and the UI token
gate.
- The original PostgreSQL-lifecycle commit passed 3 lifecycle tests, 3
ownership/diagnostic tests, 1 real embedded-Postgres seed integration,
and the affected package typechecks.
- No local tests were rerun after the clean cherry-pick because the
operator requested the shortest landing path.
- Review the automatic PR checks for the clean `origin/master` replay.

## Risks

- A live target database now causes an early error instead of being
reused. The error includes recovery guidance.
- Workspace consumers must handle the new `workspace_seed` operation
type. Shared types and UI state handling are updated in this pull
request.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, GPT-5, high-reasoning mode, with repository, shell, and
GitHub tool use. The runtime does not expose a more specific deployment
suffix or context-window value.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-08-19 10:41:29 -05:00
1 parent 51a843e135
commit e1df4c6068
13 files changed
+443 -38

No files matched your search

+49 -4
View File
@@ -27,7 +27,9 @@ import {
import {
copyGitHooksToWorktreeGitDir,
copySeededSecretsKey,
ensureEmbeddedPostgres,
ensureWorktreeSeeded,
formatWorktreeSeedFailureDiagnostic,
markWorktreeSeedPending,
pauseSeededScheduledRoutines,
quarantineSeededWorktreeExecutionState,
@@ -488,6 +490,34 @@ describe("worktree helpers", () => {
expect(full.nullifyColumns).toEqual({});
});
it("requires the seed process to own the target embedded Postgres lifecycle", async () => {
const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-live-target-"));
try {
fs.writeFileSync(
path.join(tempRoot, "postmaster.pid"),
`${process.pid}\n${tempRoot}\n0\n55432\n`,
);
await expect(ensureEmbeddedPostgres(tempRoot, 55432, { allowExisting: false }))
.rejects.toThrow("while it is already running");
} finally {
fs.rmSync(tempRoot, { recursive: true, force: true });
}
});
it("surfaces a credential-safe diagnostic when the target shuts down during restore", () => {
expect(formatWorktreeSeedFailureDiagnostic(
"restore",
new Error(
"Failed to restore seed.sql.gz: FATAL: the database system is shutting down; psql error: write EPIPE",
),
)).toBe(
"Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed.",
);
expect(formatWorktreeSeedFailureDiagnostic("migrations", new Error("secret connection failure")))
.toBe("Seed failed during migrations.");
});
it("rejects a source migration journal that diverges from the code journal", () => {
expect(() => resolveWorktreeSeedMigrationRevision({
status: "upToDate",
@@ -714,7 +744,7 @@ describe("worktree helpers", () => {
},
);
it("ensure-seeded keeps the pending marker when seeding fails", async () => {
it("ensure-seeded records a target shutdown diagnostic when restore fails", async () => {
const tempRoot = fs.mkdtempSync(path.join(os.tmpdir(), "paperclip-worktree-ensure-seeded-failure-"));
try {
const sourceConfigPath = path.join(tempRoot, "source", "config.json");
@@ -746,13 +776,28 @@ describe("worktree helpers", () => {
await expect(
ensureWorktreeSeeded(
{ config: targetConfigPath, fromConfig: sourceConfigPath },
{ seedDatabase: vi.fn().mockRejectedValue(new Error("seed failed")) },
{
seedDatabase: vi.fn(async (input) => {
input.onPhase?.("restore", "started");
throw new Error(
"Failed to restore seed.sql.gz: FATAL: the database system is shutting down; psql error: write EPIPE",
);
}),
},
),
).rejects.toThrow("seed failed");
).rejects.toThrow("database system is shutting down");
expect(readWorktreeSeedManifest(targetConfigPath)).toMatchObject({
state: "failed",
phase: "pending",
phase: "restore",
diagnostics: expect.arrayContaining([
expect.objectContaining({
phase: "restore",
status: "failed",
message:
"Target embedded PostgreSQL shut down during restore. Stop any competing worktree service and retry the seed.",
}),
]),
});
expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-pending"))).toBe(false);
expect(fs.existsSync(path.join(targetRoot, ".paperclip", "seed-complete"))).toBe(false);