feat(workspaces): defer isolated setup until runtime start (#10653)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Isolated workspaces give each task a safe and reproducible checkout.
> - The existing setup cloned the development database before an agent
needed to run the app.
> - This made worktree creation slower and heavier for tasks that never
start a service.
> - Runtime services already use one server start path for heartbeat,
operator, and startup recovery flows.
> - This pull request moves heavy setup to that start path and keeps
worktree creation lean.
> - The benefit is faster isolated workspace creation with the same
reliable runtime setup when a service starts.

## Linked Issues or Issue Description

Related pull request: #10652 covers the initial deferred
database-seeding slice. This pull request supersedes it with end-to-end
runtime provisioning and safe cleanup.

**What existing behavior does this improve?**

This improves isolated worktree creation, runtime service startup, and
isolated instance cleanup.

**Subsystem affected**

Cross-cutting: CLI worktree setup, server runtime orchestration, shared
workspace contracts, and development scripts.

**Current behavior**

Paperclip seeds an isolated development database during worktree
creation. It can also leave an isolated instance directory after
workspace teardown. This work happens even when no runtime service
starts.

**Proposed behavior**

Paperclip creates the worktree with a lean eager setup. It runs an
idempotent runtime provision command before the first managed service
spawn. Concurrent starts share one provision attempt. Teardown removes
the isolated instance safely.

**Reason and benefit**

Many agent tasks only edit and test code. They do not need a running
Paperclip instance. Deferring the database seed reduces workspace
startup cost while preserving automatic setup for tasks that start the
app.

**Breaking changes**

None. The new runtime provision command is optional. Existing workspace
behavior is unchanged when it is absent.

## What Changed

- Split Paperclip worktree setup into a lean eager script and an
idempotent runtime provision script.
- Added `runtimeProvisionCommand` to project, issue, realized workspace,
and persisted workspace contracts.
- Added a per-workspace provision mutex before local service spawn for
heartbeat, operator, and startup recovery flows.
- Added a persisted `provisioning` service state and the
`workspace_runtime_provision` operation phase.
- Kept provision time outside the service readiness timeout and made
failed attempts visible and retryable.
- Reclaimed isolated instance data during safe workspace teardown.
- Serialized deferred database seeding across processes and bound
teardown to the instance root captured in persisted workspace metadata.
- Added tests for config flow, concurrency, retry, no-op behavior,
readiness timing, scripts, CLI commands, and cleanup.
- Documented the eager and runtime provisioning contracts.

## Verification

- `pnpm -r typecheck`
- `pnpm build`
- `pnpm test:run` (server: 3,201 passed; UI: 3,345 passed; the CLI phase
exposed one environment-sensitive AWS doctor assertion because the agent
runtime injects static AWS credentials)
- `env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY pnpm exec vitest
run cli/src/__tests__/secrets.test.ts -t 'passes AWS doctor checks when
non-secret provider config is present'`
- Focused runtime tests cover serialized provisioning, retry after
stderr failure, absent-command no-op behavior, operation logging,
persisted state order, and readiness timeout exclusion.
- Focused CLI and cleanup tests cover concurrent seed serialization,
stale-lock fail-closed behavior, persisted instance ownership, and
rewritten sibling pointers.

## Risks

- A faulty runtime provision script blocks service startup. Paperclip
records stderr, marks the service failed, and retries on the next start.
- Concurrent service requests share an in-process provision attempt,
while the seed command uses an atomic filesystem lock across processes.
A stale lock fails closed and requires an operator to verify no seed is
running before removing it.
- Isolated instance cleanup is destructive. The cleanup service
validates ownership and path containment before removal.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- OpenAI Codex, `gpt-5.6-sol`, with agentic reasoning, tool use, and
code execution. The service does not expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

scripts/provision-worktree.sh

@@ -8,7 +8,10 @@ paperclip_instance_id="${PAPERCLIP_INSTANCE_ID:-default}"
 paperclip_dir="$worktree_cwd/.paperclip"
 worktree_config_path="$paperclip_dir/config.json"
 worktree_env_path="$paperclip_dir/.env"
+seed_pending_marker_path="$paperclip_dir/seed-pending"
+seed_complete_marker_path="$paperclip_dir/seed-complete"
 worktree_name="${PAPERCLIP_WORKSPACE_BRANCH:-$(basename "$worktree_cwd")}"
+created_worktree_config=0
 worktree_instance_id="$(WORKTREE_CWD="$worktree_cwd" node <<'EOF'
 const crypto = require("node:crypto");
 const path = require("node:path");
@@ -111,7 +114,7 @@ run_isolated_worktree_init() {
   if ensure_base_cli_healthy; then
     (
       cd "$worktree_cwd" &&
-        node "$base_cli_runner_path" "$base_cli_entry_path" worktree init --force --seed-mode minimal --name "$worktree_name" --instance "$worktree_instance_id" --from-config "$source_config_path"
+        node "$base_cli_runner_path" "$base_cli_entry_path" worktree init --force --no-seed --seed-mode minimal --name "$worktree_name" --instance "$worktree_instance_id" --from-config "$source_config_path"
     )
     return
   fi
@@ -119,7 +122,7 @@ run_isolated_worktree_init() {
   if command -v pnpm >/dev/null 2>&1 && pnpm paperclipai --help >/dev/null 2>&1; then
     (
       cd "$worktree_cwd" &&
-        pnpm paperclipai worktree init --force --seed-mode minimal --name "$worktree_name" --instance "$worktree_instance_id" --from-config "$source_config_path"
+        pnpm paperclipai worktree init --force --no-seed --seed-mode minimal --name "$worktree_name" --instance "$worktree_instance_id" --from-config "$source_config_path"
     )
     return
   fi
@@ -127,7 +130,7 @@ run_isolated_worktree_init() {
   if command -v paperclipai >/dev/null 2>&1; then
     (
       cd "$worktree_cwd" &&
-        paperclipai worktree init --force --seed-mode minimal --name "$worktree_name" --instance "$worktree_instance_id" --from-config "$source_config_path"
+        paperclipai worktree init --force --no-seed --seed-mode minimal --name "$worktree_name" --instance "$worktree_instance_id" --from-config "$source_config_path"
     )
     return
   fi
@@ -234,6 +237,31 @@ for (const rawValue of runtimePaths) {
 EOF
 }
 
+write_seed_pending_marker() {
+  SEED_PENDING_MARKER_PATH="$seed_pending_marker_path" \
+  SEED_COMPLETE_MARKER_PATH="$seed_complete_marker_path" \
+  SOURCE_CONFIG_PATH="$source_config_path" \
+  node <<'EOF'
+const fs = require("node:fs");
+const path = require("node:path");
+
+const pendingPath = process.env.SEED_PENDING_MARKER_PATH;
+const completePath = process.env.SEED_COMPLETE_MARKER_PATH;
+fs.rmSync(completePath, { force: true });
+fs.writeFileSync(
+  pendingPath,
+  `${JSON.stringify({
+    version: 1,
+    state: "pending",
+    sourceConfigPath: path.resolve(process.env.SOURCE_CONFIG_PATH),
+    seedMode: "minimal",
+    createdAt: new Date().toISOString(),
+  }, null, 2)}\n`,
+  { mode: 0o600 },
+);
+EOF
+}
+
 write_fallback_worktree_config() {
   WORKTREE_NAME="$worktree_name" \
   BASE_CWD="$base_cwd" \
@@ -520,6 +548,11 @@ else
     echo "paperclipai worktree init unavailable; writing isolated fallback config without DB seeding." >&2
     write_fallback_worktree_config
   fi
+  created_worktree_config=1
+fi
+
+if [[ "$created_worktree_config" -eq 1 && ! -e "$seed_pending_marker_path" && ! -e "$seed_complete_marker_path" ]]; then
+  write_seed_pending_marker
 fi
 
 list_base_node_modules_paths() {