fix(recovery): keep restarted Retry attempts as task directories and settle failed re-preparation (#19759)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Task recovery lets a user click Retry after a run fails, and a
native run restarts its own preparation after a control-plane restart
> - A Retry for a task without a project runs in a fresh task directory
under `isolated-workspace-attempts`
> - When a restart hit that Retry during bootstrap, preparing it again
treated the task directory as the base of a new git worktree and failed
> - The failed run then kept its native recovery claim open, so the task
showed Retry but every Retry click created no run
> - This pull request keeps the restarted attempt in its task directory
and settles a failed re-preparation
> - The benefit is that a Retry finishes after a restart, and a failed
one can be retried again

## Linked Issues or Issue Description

Refs #19666

**What happened?**
A task without a project failed after a restart
(`native_session_interrupted`). The user clicked Retry. The Retry run
started in a fresh task directory. A restart hit it during bootstrap.
Restart recovery prepared the same run again and failed with
`workspace_validation_failed`: "requested isolated_workspace with
git_worktree, but base workspace
`.../isolated-workspace-attempts/<company>/<issue>/<run>` is not a git
checkout". The task rested blocked with Retry shown. A later Retry click
created no run: the receipt stayed deferred with "New work must wait
while a native run recovers its unfinished accounting." A second storm
wave showed the same failure on a Claude C1 task.

**Expected behavior**
Preparing a run again after a restart reuses the task directory that run
already bound. A git worktree base is only the project's checkout or the
task's own worktree. A run whose re-preparation fails leaves a working
Retry.

**Paperclip version or commit**
`master` at `51eb4ff705` and `131b566ca4` (both include #19666).
Restart-chaos waves on 2026-11-07.

**Steps to reproduce**
1. Use a task with no project and isolated workspaces enabled. Let a
native run fail after a restart.
2. Click Retry. The run starts in
`isolated-workspace-attempts/<company>/<issue>/<run>`.
3. Restart the control plane while the run bootstraps.
4. The re-preparation fails with `git_worktree_base_not_git_checkout`.
Click Retry again: no run starts.

Root cause:
- `server/src/services/fresh-workspace-continuation.ts`
`readFreshWorkspaceReadiness` returns `null` for a native run that
already has admitted input and a bound workspace (from #19693). This is
correct: the restart must resume the admitted workspace.
- `server/src/services/heartbeat/environment-selection.ts`
(`boundTaskDirectory`) excluded every native recovery binding from the
task-directory path. With readiness no longer fresh,
`useIsolatedTaskDirectory` was false. The default `isolated_workspace`
strategy (`git_worktree`) applied, with the task directory as its base.
- The restart recovery claim set the native coordinator to `observed`
with its lease owner. `failRunSetup` failed the run, but nothing
released that claim. `budgets.ts` reads an open coordinator as
unfinished accounting, and Retry admission reads it as
`controller_settling`.

## What Changed

- Environment selection treats a native recovery binding as the task
directory when it is the task's current binding and a bound isolated
task directory. The directory strategy is kept; it never becomes a git
worktree base. A recovery binding the issue no longer points to keeps
its existing path.
- New `settleFailedNativeRecoveryPreparation`. When a
`bootstrap_incomplete` or `resume_dead_runner` restart claim fails its
re-preparation before any provider dispatch, it settles the claimed
coordinator as `terminal_failure` and wakes deferred work. It only
changes a coordinator that is still `observed`, owned by that claim,
without a result, for a failed run. Reattach claims (live runner) are
not changed.
- `doc/task-recovery.md` describes both rules.

## Verification

- `environment-selection.test.ts`, RC-58 cases: a restarted fresh
attempt bound to the task re-prepares as its task directory
(`useIsolatedTaskDirectory` true). It failed before the fix and passes
now. A binding the issue moved away from keeps the old behavior.
- `native-runtime/native-recovery-preparation-failure.test.ts` (embedded
Postgres): before settlement, Retry admission reports
`controller_settling` (the dead Retry). After settlement the coordinator
is `terminal_failure` with the run's error code, and admission no longer
waits on the controller. Settlement is idempotent. Another owner, a
reattach claim, a run that is not failed, and a coordinator finalizing
elsewhere are left unchanged.
- Local regression: `environment-selection`, `workspace-preparation`,
`retry-workspace-continuity`, `native-recovery-preparation-failure`,
`explicit-native-continuation` (588 tests passed).
- Restart-chaos: local restart-chaos runs on `3a7abef4e5` passed, but
neither one reproduced the exact failure (a restart during the Retry's
bootstrap). The race depends on timing, so the unit tests above are the
red/green proof for both defects.
- `--replay w-20261107T011010-s7403/t02` (codex C1 K4 with the original
fault sequence): PASS, R1-R6 and T. The replay's Retry did not cross a
bootstrap restart.
- Wave `w-20261107T064432-s7210` (codex+claude, C1+B1, K1/K4/K6,
`--triggers retry`, concurrency 2): 8/8 PASS. It had 4 Retries,
including C1 tasks with a fault fired right after the Retry click. No
`git_worktree_base_not_git_checkout` and no dead Retry. The server log
has no "incomplete native runner bootstrap" restart in this wave either.
- `ce7e3f9b19` registers execution work before the settlement writes
(Greptile finding).

## Risks

- A native recovery now re-prepares a bound task directory through the
task-directory path. That path still validates the binding, owner,
status and directory identity before use.
- Settling a failed re-preparation ends that run's recovery claim. It
applies only when the previous runner is verified gone
(`bootstrap_incomplete`, `resume_dead_runner`) and no provider dispatch
started in this attempt. A Retry still passes its normal stop, ownership
and admission checks.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

doc/task-recovery.md

@@ -38,7 +38,14 @@ repeating the failed step. A Retry never silently drops the agent's work:
 
 Either choice consumes the generation's one alternative attempt. Preparing the
 same attempt again after a restart reuses the directory that attempt already
-created and bound; it does not ask for another fresh directory.
+created and bound; it does not ask for another fresh directory. A native
+restart re-prepares from its admitted workspace. When that is the task's
+current task directory, it keeps the directory strategy; a task directory is
+never the base of a new git worktree. A git worktree base is always the
+project's checkout or the task's own worktree. If a restart's re-preparation
+still fails before the provider starts, the run settles its native coordinator
+as a terminal failure, so a later Retry is admitted instead of waiting on an
+open recovery claim.
 
 The existing exact-run retry endpoint retains its source-bound contract.
 Connected conversations use their existing inbound authorization owner. A board

server/src/services/heartbeat.ts

@@ -23,6 +23,7 @@ import { readUpgradeReadiness } from "./heartbeat/upgrade-readiness.js";
 import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
 import { writeExecutionControl } from "./execution-control-notifications.js";
 import { readFreshWorkspaceReadiness } from "./fresh-workspace-continuation.js";
+import { settleFailedNativeRecoveryPreparation } from "./native-runtime/native-recovery-preparation-failure.js";
 import { promptInjectionPreventionService } from "./prompt-injection-prevention.js";
 import { createHeartbeatAgentStatus } from "./heartbeat/agent-status.js";
 import { heartbeatExternalProvider, nativeRuntimeHasInstructionFilesystem } from "./heartbeat/external-provider.js";
@@ -3810,6 +3811,11 @@ export function heartbeatService(
           });
           legacyPreparationFailedBeforeDispatch = !legacyAdapterEntered && !nativeDispatchStarted;
         }
+        // A failed re-preparation must not keep its recovery claim open.
+        if (runOptions.nativeRestartRecovery && !legacyAdapterEntered && !nativeDispatchStarted)
+          await settleFailedNativeRecoveryPreparation(db, { companyId: run.companyId, runId: run.id,
+            claim: runOptions.nativeRestartRecovery }).catch(error =>
+            logger.warn({ err: error, runId: run.id }, "Failed native recovery preparation could not be settled"));
       }
     } finally {
       let caseHandoffCleanupSettled = false;

server/src/services/heartbeat/environment-selection.test.ts

@@ -13,6 +13,7 @@ import { instanceSettingsService } from "../instance-settings.js";
 import { buildNativeExecutionInput } from "../native-runtime/native-execution-input.js";
 import { nativeRuntimeContextFixture } from "../native-runtime/runtime-context.test-fixture.js";
 import { selectHeartbeatEnvironment, type HeartbeatEnvironmentSelectionInput } from "./environment-selection.js";
+import { materializeIsolatedTaskDirectory } from "../isolated-task-directory.js";
 
 const restoration = vi.hoisted(() => ({ removed: false }));
 vi.mock("../git-workspace-restoration.js", async importOriginal => {
@@ -96,6 +97,35 @@ describe.skipIf(!support.supported)("heartbeat environment selection boundary",
     expect(result.persistedNativeExecutionInput?.binding.executionWorkspaceId).toBe(original.id);
   });
 
+  it.each(["bound", "issue_moved"] as const)(
+    "re-prepares a restarted fresh Retry attempt as its task directory, never as a git worktree base (RC-58, %s)", async binding => {
+      const { input, issue, workspace, recover } = await fixture();
+      // A project-less task's fresh Retry attempt directory, created and bound by the first preparation.
+      await db.update(issues).set({ projectId: null }).where(eq(issues.id, issue.id));
+      const attemptCwd = await materializeIsolatedTaskDirectory({ companyId: input.agent.companyId, issueId: issue.id, attemptId: input.run.id });
+      const created = await executionWorkspaceService(db).create({ companyId: input.agent.companyId, projectId: null,
+        sourceIssueId: issue.id, name: "Fresh attempt", mode: "isolated_workspace", strategyType: "project_primary",
+        providerType: "local_fs", cwd: attemptCwd });
+      if (!created) throw new Error("Attempt workspace fixture was not created");
+      await db.update(issues).set({ executionWorkspaceId: created.id, executionWorkspacePreference: "reuse_existing",
+        executionWorkspaceSettings: { mode: "isolated_workspace" } }).where(eq(issues.id, issue.id));
+      const issueRef = { ...input.task.issueRef!, projectId: null, executionWorkspaceId: created.id,
+        executionWorkspacePreference: "reuse_existing" };
+      input.task.issueRef = issueRef;
+      input.task.executionProjectId = null;
+      input.policy.requestedExecutionWorkspaceMode = "isolated_workspace";
+      input.policy.issueExecutionWorkspaceSettings = { mode: "isolated_workspace" };
+      if (binding === "issue_moved") issueRef.executionWorkspaceId = (await workspace()).id;
+      // The restart re-prepares from the admitted native input; readiness is no longer "fresh".
+      recover(created.id);
+      input.task.freshWorkspace = null;
+      const result = await selectHeartbeatEnvironment(db, input);
+      expect(result.nativeRecoveryExecutionWorkspaceId).toBe(created.id);
+      expect(result.reusableExistingExecutionWorkspace?.id).toBe(created.id);
+      // Only the task's current binding may resume as a task directory.
+      expect(result.useIsolatedTaskDirectory).toBe(binding === "bound");
+    });
+
   it("selects the saved execution workspace without inspecting live Git delivery state", async () => {
     const { input, workspace, recover } = await fixture();
     const saved = await workspace();

server/src/services/heartbeat/environment-selection.ts

@@ -494,7 +494,11 @@ export async function selectHeartbeatEnvironment(db: Db, input: HeartbeatEnviron
     ...(reusableExistingExecutionWorkspace?.providerType === "adapter_managed" ? [config.workspaceStrategy] : []),
     issueAssigneeOverrides?.adapterConfig?.workspaceStrategy,
     projectExecutionWorkspacePolicy?.workspaceStrategy, issueExecutionWorkspaceSettings?.workspaceStrategy];
-  const boundTaskDirectory = !controllerOnly && !fresh && !nativeRecoveryExecutionWorkspaceId && issueId !== null && issueRef !== null &&
+  // RC-58: a native restart re-prepares its admitted workspace. When that is
+  // the task's own bound directory (for example a fresh Retry attempt), it keeps
+  // its directory strategy; it never becomes the base of a new git worktree.
+  const boundTaskDirectory = !controllerOnly && !fresh && issueId !== null && issueRef !== null &&
+    (!nativeRecoveryExecutionWorkspaceId || nativeRecoveryExecutionWorkspaceId === issueRef.executionWorkspaceId) &&
     requestedShouldReuseExisting && requestedExecutionWorkspaceMode === "isolated_workspace" &&
     !workspaceProjectContext?.hasWorkspace && !issueRef.projectWorkspaceId &&
     taskWorkspaceIntent?.selection.kind !== "configured_source" &&

server/src/services/native-runtime/native-recovery-preparation-failure.test.ts

@@ -0,0 +1,95 @@
+import { randomUUID } from "node:crypto";
+import { eq } from "drizzle-orm";
+import { afterAll, beforeAll, describe, expect, it } from "vitest";
+import {
+  agents, applyPendingMigrations, companies, createDb, heartbeatRunEvents, heartbeatRuns, issueComments,
+  issueRecoveryActions, issues, nativeRunFinalizations, type Db,
+} from "@paperclipai/db";
+import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
+import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
+import { admitExplicitNativeContinuation } from "../explicit-native-continuation.js";
+import { settleFailedNativeRecoveryPreparation } from "./native-recovery-preparation-failure.js";
+
+const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_RECOVERY_PREPARATION_TEST_DATABASE_URL");
+const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
+
+/** RC-58: a failed re-preparation after a restart must leave a working Retry. */
+describe.skipIf(!support.supported)("failed native recovery preparation", () => {
+  let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
+  let db: Db;
+  beforeAll(async () => {
+    if (externalUrl) {
+      await applyPendingMigrations(externalUrl);
+      database = { connectionString: externalUrl, cleanup: async () => {} } as typeof database;
+    } else database = await startEmbeddedPostgresTestDatabase("native-recovery-preparation-");
+    db = createDb(database.connectionString);
+  }, 90_000);
+  afterAll(async () => { await db?.$client.end(); await database?.cleanup(); });
+
+  async function seed() {
+    const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), runId = randomUUID(), commentId = randomUUID();
+    const leaseOwner = `boot:2:${randomUUID()}`;
+    await db.insert(companies).values({ id: companyId, name: "Recovery preparation", defaultResponsibleUserId: "board",
+      issuePrefix: `P${companyId.replaceAll("-", "").slice(0, 8)}` });
+    await db.insert(agents).values({ id: agentId, companyId, name: "Native", role: "engineer", adapterType: "paperclip_runner",
+      status: "idle", runtimeConfig: { heartbeat: { maxConcurrentRuns: 1 } } });
+    await db.insert(issues).values({ id: issueId, companyId, title: "Write the report", status: "blocked", assigneeAgentId: agentId });
+    // The restart claimed the run (bootstrap_incomplete); its re-preparation then failed.
+    await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, nativeIssueId: issueId, runtimeMode: "native",
+      status: "failed", errorCode: "workspace_validation_failed", nativePhase: "observed", contextSnapshot: { issueId },
+      startedAt: new Date(Date.now() - 120_000), finishedAt: new Date(Date.now() - 60_000) });
+    await db.insert(nativeRunFinalizations).values({ runId, companyId, issueId, phase: "observed", attempt: 1,
+      leaseOwner, leaseExpiresAt: new Date(Date.now() + 60_000), controllerGeneration: 2, recoveryState: "bootstrap_incomplete" });
+    // Server-authored process trail: the first bootstrap's runner was verified stopped.
+    await db.insert(heartbeatRunEvents).values([
+      { companyId, runId, agentId, seq: 1, eventType: "native.process_start_requested", stream: "system" },
+      { companyId, runId, agentId, seq: 2, eventType: "native.process_identity_recorded", stream: "system" },
+      { companyId, runId, agentId, seq: 3, eventType: "native.local_process_stopped", stream: "system" },
+    ]);
+    await db.insert(issueRecoveryActions).values({ companyId, sourceIssueId: issueId, kind: "active_run_watchdog",
+      cause: "native_continuation_requires_reconciliation", fingerprint: runId, status: "active", ownerType: "board",
+      returnOwnerAgentId: agentId, nextAction: "Retry the task.", evidence: { runId } });
+    await db.insert(issueComments).values({ id: commentId, companyId, issueId, authorType: "user", authorUserId: "board",
+      body: "Please continue.", createdAt: new Date() });
+    return { companyId, agentId, issueId, runId, commentId, leaseOwner };
+  }
+
+  async function admissionBlocks(f: Awaited<ReturnType<typeof seed>>) {
+    const blocked: string[] = [];
+    await db.transaction(async tx => admitExplicitNativeContinuation({ db: tx as unknown as Db, companyId: f.companyId,
+      issueId: f.issueId, agentId: f.agentId, actorId: "board", actorType: "user", reason: "issue_commented",
+      commentId: f.commentId, successorRunId: randomUUID(), dryRun: true, onBlocked: reason => blocked.push(reason) }));
+    return blocked;
+  }
+
+  it("settles its own recovery claim so a later Retry is no longer held by the controller", async () => {
+    const f = await seed();
+    expect(await admissionBlocks(f)).toEqual(["controller_settling"]);
+    expect(await settleFailedNativeRecoveryPreparation(db, { companyId: f.companyId, runId: f.runId,
+      claim: { kind: "bootstrap_incomplete", leaseOwner: f.leaseOwner } })).toBe(true);
+    const [coordinator] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
+    expect(coordinator).toMatchObject({ phase: "terminal_failure", leaseOwner: null, resultId: null,
+      failureCode: "workspace_validation_failed" });
+    const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
+    expect(run).toMatchObject({ status: "failed", nativePhase: "terminal_failure" });
+    expect(await admissionBlocks(f)).not.toContain("controller_settling");
+    // Settling is idempotent.
+    expect(await settleFailedNativeRecoveryPreparation(db, { companyId: f.companyId, runId: f.runId,
+      claim: { kind: "bootstrap_incomplete", leaseOwner: f.leaseOwner } })).toBe(false);
+  }, 30_000);
+
+  it.each(["other_owner", "reattach_claim", "run_not_failed", "finalizing_elsewhere"] as const)(
+    "leaves a coordinator it does not own unchanged: %s", async kind => {
+      const f = await seed();
+      if (kind === "run_not_failed") await db.update(heartbeatRuns).set({ status: "running", finishedAt: null })
+        .where(eq(heartbeatRuns.id, f.runId));
+      if (kind === "finalizing_elsewhere") await db.update(nativeRunFinalizations).set({ phase: "committing" })
+        .where(eq(nativeRunFinalizations.runId, f.runId));
+      const claim = { kind: kind === "reattach_claim" ? "reattach_existing_runner" as const : "resume_dead_runner" as const,
+        leaseOwner: kind === "other_owner" ? `boot:3:${randomUUID()}` : f.leaseOwner };
+      const [before] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
+      expect(await settleFailedNativeRecoveryPreparation(db, { companyId: f.companyId, runId: f.runId, claim })).toBe(false);
+      const [after] = await db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
+      expect(after).toEqual(before);
+    }, 30_000);
+});

server/src/services/native-runtime/native-recovery-preparation-failure.ts

@@ -0,0 +1,48 @@
+import { and, eq, isNull } from "drizzle-orm";
+import { heartbeatRuns, nativeRunFinalizations, type Db } from "@paperclipai/db";
+import { notifyExecutionRunWork } from "../execution-control-notifications.js";
+import type { NativeRestartRecoveryClaim } from "./native-restart-recovery.js";
+
+/** Claims whose previous runner process is verified gone. A reattach claim
+ * still has a live runner, so its coordinator keeps its own recovery owner. */
+const SETTLEABLE_CLAIMS = new Set<NativeRestartRecoveryClaim["kind"]>(["bootstrap_incomplete", "resume_dead_runner"]);
+
+/**
+ * RC-58: a restart recovery claims the run's native coordinator, then prepares
+ * the run again. When that preparation fails before any provider dispatch, the
+ * run is already failed but the claim would keep the coordinator `observed`
+ * and owned. That reads as unfinished native accounting and an unsettled
+ * controller, so a later Retry can never be admitted. Settle exactly this
+ * claim as a terminal failure. A coordinator that another owner holds, or one
+ * with a result, is left unchanged. Returns whether this call settled it.
+ */
+export async function settleFailedNativeRecoveryPreparation(db: Db, input: {
+  companyId: string; runId: string; claim: Pick<NativeRestartRecoveryClaim, "kind" | "leaseOwner">;
+}): Promise<boolean> {
+  if (!SETTLEABLE_CLAIMS.has(input.claim.kind)) return false;
+  return db.transaction(async tx => {
+    const [run] = await tx.select({ status: heartbeatRuns.status, errorCode: heartbeatRuns.errorCode })
+      .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)))
+      .for("update");
+    if (run?.status !== "failed") return false;
+    // Register the work before the writes it announces. A spare wake when no
+    // coordinator matches below is harmless.
+    await notifyExecutionRunWork(tx as unknown as Db, { companyId: input.companyId });
+    const now = new Date();
+    const [settled] = await tx.update(nativeRunFinalizations).set({
+      phase: "terminal_failure", leaseOwner: null, leaseExpiresAt: null, nextAttemptAt: null, controlDeadlineAt: null,
+      failureCode: run.errorCode ?? "native_recovery_preparation_failed",
+      failureDetail: { recoveryPreparationFailed: { claim: input.claim.kind, errorCode: run.errorCode },
+        nextAction: "Preparation failed before the provider was started again. Retry the task." },
+      recoveryState: "blocked", updatedAt: now,
+    }).where(and(
+      eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId),
+      eq(nativeRunFinalizations.phase, "observed"), eq(nativeRunFinalizations.leaseOwner, input.claim.leaseOwner),
+      isNull(nativeRunFinalizations.resultId),
+    )).returning({ runId: nativeRunFinalizations.runId });
+    if (!settled) return false;
+    await tx.update(heartbeatRuns).set({ nativePhase: "terminal_failure", nativePhaseUpdatedAt: now, executionControlDeadlineAt: null })
+      .where(and(eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.id, input.runId)));
+    return true;
+  });
+}