fix(native): recover saved listener startup failures (#19479)

## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native recovery must preserve saved work and respect current task
authority.
> - A closed local API listener can stop startup before a provider
starts.
> - #19462 gives new failures a bounded bootstrap retry.
> - Older saved failures still have a generic terminal code and a
recovery hold.
> - This pull request verifies the old producer outcome and uses the
existing safe replacement path.
> - Eligible tasks continue once. Unknown provider outcomes and user
holds remain blocked.

## Linked Issues or Issue Description

Refs: #19462

**What happened?**

A restart can leave an old listener startup failure in
`terminal_failure` with `native_session_interrupted`. Startup recovery
does not claim it after #19462. A completed file recovery can also
retain its no-replay hold.

**Expected behavior**

Continue automatically when durable records prove that no provider
started, the source stopped, and the current files were saved. Keep the
existing attempt budget and normal admission checks.

**Steps to reproduce**

1. Close the actual API listener after a fresh native coordinator claim.
2. Persist the old executor's generic failure. Save the workspace and
confirm source termination.
3. Restart recovery. The baseline schedules no successor. The repair
schedules one successor after all receipts pass.

**Paperclip version or commit**

Baseline: `0252421d5c`. Old failure producer: `2f35eb1080`.

**Deployment mode**

Built from source. Local macOS arm64, Node 26.4.0, real embedded
PostgreSQL, independent filesystem execution. No external AI service is
used by the controlled acceptance.

## What Changed

- Recognize the exact saved local-listener guard through one proof
predicate shared by discovery and the locked transaction.
- Require a fresh first attempt, matching execution bindings, no prior
provider lineage, no provider events or tool calls, no checkpoint or
admission receipt, and no cancellation.
- Reuse current snapshot, exact stop, source block, attempt budget, and
successor idempotency checks.
- Wait for completed source file-recovery receipts. Retire only that
same source's verified preservation hold in the successor transaction.
- Bound discovery to 100 candidates per pass, keep completed history
quiet, and wake through existing transactional producer notifications.
- Retain retries after transient database and final ownership-proof
errors; advance the bounded cursor and force a rescan so a failing page
cannot hide later work.
- Let only the replacement duty discharge a proved-empty notification
during drain through its existing read-only readiness check, once per
notification generation. Keep real work and concurrent changes
outstanding without running replacement during drain.
- Wake replacement at canonical local-stop proof writes and the existing
dispatch eligibility boundary; these hints grant no new execution
authority.
- Add database regressions, scheduler quietness checks, real controller
handoff coverage, and an opt-in controlled Runner continuation.

## Verification

Final production head: `0df41b2fef2644195d1224d36984af3827da3e3b`.

Local environment: macOS arm64, Node 26.4.0, TypeScript 7.0.2, Vitest
5.0.3, frozen-lockfile dependencies. Tests execute freshly;
`--fsModuleCache` reuses only content-validated module transforms.

- Native safe replacement: 62/62 PASS on isolated PostgreSQL 17 (63.18
seconds), including clean per-suite database disposal. All 62 assertions
also passed on embedded PostgreSQL 18 beta, but its Darwin cleanup hook
timed out; that invocation is not counted as a pass. Docker's disk was
full, so the disposable PostgreSQL 17 data used a size-limited RAM
mount, with no host mounts. The owned container was removed afterward.
- Actual stopped-process producer suite: 31/31 PASS. A real exited child
produces the stop receipt, and its replacement notification settles only
after the outer transaction commits.
- Workspace and separate-process drain regressions: 33/33 PASS (75.90
seconds).
- Shared coordinator and execution-control lifecycle checks: 39/39 PASS
(16.43 seconds). Pending work, concurrent changes, cold startup,
shutdown, and passive drain boundaries remain covered.
- Actual suppressed Cloud routine/drain workflow: PASS (34.64 seconds),
using the unchanged test and timeouts. Its matching local RED reproduced
CI's 462-query, zero-pending-query, one-idle-hold timeout after an
admitted write committed during drain (69.46 seconds).
- Actual historical controller handoff: PASS (48.40 seconds). A separate
controller loads the original executor from `2f35eb1080`, writes its
generic listener failure, and exits. Recovery at the final head admits
exactly one successor. The real Rust Runner and controlled Codex process
complete governed `paperclip_finish`; attempt two commits and the saved
file remains intact. Only the child loads the old artifact, and both
isolated artifacts are restored afterward. Rust source subtree
`77961b70f177ec3c38d11abcaec92fd3d0e96ec0` matches the built binary. No
external AI service is used.
- Seven corrected existing fixtures pass: transactional stop-proof calls
(5 cases) and notification inventory (2 cases). Their earlier CI
failures exposed fixture calls outside the documented transaction and an
outdated expected queue set.
- Server typecheck: PASS.

Red/green proofs also cover the original saved failure on baseline
`0252421d5c` (zero successors instead of one), held-history quietness,
bounded forward scanning through 100 parked rows and concurrent wakes,
real snapshot/cleanup producer notifications, actual issue-row lock
timeout, and task/history eligibility notifications. An obsolete
prior-run question can be skipped to request another proof check; actual
skip while the task is paused or the source owns provider tool activity
schedules no successor.

The failed-page regression starts with 25 persistent verifier errors and
a healthy candidate 26. Baseline RED repeats the first page. Repair
GREEN advances the healthy candidate, revisits the original failures
after their fault clears, creates all 26 successors exactly once, and
becomes quiet.

Greptile is SUCCESS, 5/5 on the exact final head (2026-11-06 17:09:21
UTC). Final-head CI is GREEN: 56 successful checks, three intentional
skips, one neutral result, and no pending or failing checks. [Attempt
two and
results](https://github.com/paperclipai/paperclip/actions/runs/42258352876/attempts/2)
include passing build, typecheck, Rust, native server, package, browser
E2E, serialized server, general server/chat tests, and final
verification. Five jobs in the first attempt were interrupted together
at 17:13:12 UTC; the native-server log records an explicit runner
shutdown signal. The single infrastructure retry reran only those five
jobs and dependent verification, preserving the 49 successful checks on
the unchanged head. No timeout was weakened, and setup failures or
interrupted invocations are not counted as passes.

Focused commands from the repository root:

```sh
./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/services/remote-workspaces.integration.test.ts server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts -t 'native safe-replacement decision|reconciles the drained remote backend' --fsModuleCache --reporter=dot
./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/__tests__/execution-control-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts -t 'execution-control lifecycle|work coordinator' --fsModuleCache --reporter=dot
node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit --pretty false
```

The historical acceptance sets `PAPERCLIP_TEST_HISTORICAL_LISTENER=1`
and uses an original-executor child artifact with a `try/finally`
restore. Normal regressions require no Rust binary.

## Risks

- Compatibility is deliberately limited to positively proved fresh
remote startup failures. Other old ambiguous failures remain held.
- Existing provider work, a changed task block, incomplete preservation,
or revoked authority must prevent execution. Normal retry admission
remains in control.
- This PR does not deploy the change or mutate a hosted task. Hosted
recovery still requires the repaired revision and all current proof
gates.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The exact backend model identifier and context window are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/services/native-runtime/native-case-handoff.test.ts

@@ -84,7 +84,7 @@ describe("native persisted case handoff runtime", () => {
     expect(await isCaseHandoffSourceReleased(db, f.input)).toBe(false);
     const exited = once(child, "exit"); child.kill("SIGTERM"); await exited;
     const [run] = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
-    expect(await recordNativeLocalProcessStop(db, run!)).toBe(true);
+    expect(await db.transaction(tx => recordNativeLocalProcessStop(tx as unknown as typeof db, run!))).toBe(true);
     // A new worker has no process-local state and reconstructs the release.
     expect(await isCaseHandoffSourceReleased(db, f.input)).toBe(true);
     expect((await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0]?.resultJson).toHaveProperty("caseHandoffExecutionReleased");

server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts

@@ -1,11 +1,21 @@
 import { startNativeSessionExecutionLeaseRenewal, type NativeExecutionLeaseRenewal } from "./native-execution-lease.js";
 import { randomUUID } from "node:crypto";
-import { resolve } from "node:path";
+import { resolve, join } from "node:path";
+import { mkdir, readFile, writeFile } from "node:fs/promises";
 import { eq, sql } from "drizzle-orm";
-import { environmentLeases, environments, heartbeatRuns, nativeRunFinalizations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
+import { environmentLeases, environments, heartbeatRuns, nativeRunFinalizations, issueRecoveryActions, issues, completionContracts, nativeRunResults, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS } from "@paperclipai/db";
 import { beforeAll, expect, it, vi } from "vitest";
-import { DurablePrpControlPlane } from "../../vendor/paperclip-runner/index.js";
+import { spawn } from "node:child_process";
+import { createRequire } from "node:module";
+import { DurablePrpControlPlane, createNativeSessionBackend, createRunnerdCodexTransport, defaultCapabilityRunnerdBinary, canonicalNativeRuntimeContextDigest } from "../../vendor/paperclip-runner/index.js";
 import { buildNativeExecutionInput } from "./native-execution-input.js";
+import { buildNativeCompletionContract } from "./completion-contracts.js";
+import { reconcileNativeFinalizations, claimNativeSessionResumptions } from "./native-finalization-reconciler.js";
+import { prepareNativeWorkspaceSync } from "./native-workspace-sync.js";
+import { workspaceOperationService } from "../workspace-operations.js";
+import type { EnvironmentLease } from "@paperclipai/shared";
+import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
+import { materializeAsset } from "./runtime-context.js";
 import { nativeRuntimeContextFixture } from "./runtime-context.test-fixture.js";
 import { executePaperclipNativeSession } from "./native-session-executor.js";
 import { registerNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
@@ -21,6 +31,147 @@ describeEmbeddedPostgres("native runner restart recovery: preparation", () => {
   let successor: typeof environment.current.successor;
   beforeAll(() => { ({ runtimeRoot, companyId, agentId, successor } = environment.current); });
 
+  it.each(["recover", "provider event", "cancelled task"] as const)("reconciles the drained remote backend with %s", async mode => {
+    const fixture = await seedRun(`DRAINED-BACKEND-${mode.replaceAll(" ", "-")}`);
+    const environmentId = randomUUID(), leaseId = randomUUID();
+    await fixture.db.insert(environments).values({ id: environmentId, name: `Drained backend ${environmentId}`, driver: "sandbox" });
+    const [lease] = await fixture.db.insert(environmentLeases).values({ id: leaseId, companyId, environmentId,
+      heartbeatRunId: fixture.runId, issueId: fixture.issueId, status: "active",
+      provider: "fixture", providerLeaseId: `drained-backend-${leaseId}` }).returning();
+    const [issue] = await fixture.db.select().from(issues).where(eq(issues.id, fixture.issueId));
+    const [contract] = await fixture.db.select().from(completionContracts).where(eq(completionContracts.issueId, fixture.issueId));
+    const runtimeContext = nativeRuntimeContextFixture();
+    runtimeContext.instructions.bundle = await materializeAsset([{ path: "AGENTS.md", mode: 0o444,
+      content: Buffer.from("Complete the controlled restart fixture task.\n") }]);
+    runtimeContext.aggregateDigest = canonicalNativeRuntimeContextDigest(runtimeContext);
+    const sandbox = join(runtimeRoot, `workspace-${fixture.runId}`);
+    await mkdir(sandbox, { recursive: true });
+    const savedFile = join(sandbox, "saved-before-drain.txt");
+    await writeFile(savedFile, "Preserve admitted work.\n");
+    const execution = buildNativeExecutionInput({ companyId, agentId, runId: fixture.runId,
+      issue: { id: fixture.issueId, identifier: "NRR-DRAINED", title: "Continue after drain", description: null, workMode: "standard" },
+      taskPrompt: "Continue the admitted work once.", provider: "codex",
+      workspace: { id: fixture.runId, cwd: sandbox, repoUrl: null, repoRef: null, branchName: null },
+      normalizedSessionId: fixture.native.normalizedSessionId, runtimeContext,
+      completionContract: { id: contract.id, sha256: contract.canonicalSha256, schemaVersion: "paperclip.completion-contract.v1",
+        contract: buildNativeCompletionContract(issue, { revision: contract.revision }) },
+    });
+    await fixture.db.update(heartbeatRuns).set({ runnerProfileJson: {
+      ...fixture.preparedProfile, nativeExecutionInput: execution,
+    } }).where(eq(heartbeatRuns.id, fixture.runId));
+    const remoteSandbox = join(runtimeRoot, `remote-${fixture.runId}`);
+    const shell: CommandManagedRuntimeRunner = { execute: async input => {
+      const startedAt = new Date().toISOString();
+      const child = spawn(input.command, input.args ?? [], { cwd: input.cwd,
+        env: { ...process.env, ...input.env }, stdio: ["pipe", "pipe", "pipe"] });
+      let stdout = "", stderr = "", timedOut = false;
+      child.stdout.setEncoding("utf8").on("data", chunk => { stdout += chunk; });
+      child.stderr.setEncoding("utf8").on("data", chunk => { stderr += chunk; });
+      child.stdin.on("error", () => undefined);
+      child.stdin.end(input.stdin);
+      const timeout = setTimeout(() => { timedOut = true; child.kill("SIGKILL"); }, input.timeoutMs ?? 30_000);
+      try { return await new Promise((resolveResult, reject) => {
+        child.once("error", reject);
+        child.once("close", (exitCode, signal) => resolveResult({ stdout, stderr, exitCode, signal, timedOut,
+          pid: child.pid ?? null, startedAt }));
+      }); } finally { clearTimeout(timeout); }
+    } };
+    const workspaceSync = await prepareNativeWorkspaceSync({ db: fixture.db, runId: fixture.runId, companyId,
+      workspaceId: fixture.runId, workspaceLocalDir: sandbox, lease: lease as unknown as EnvironmentLease,
+      target: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox, runner: shell } });
+    expect(workspaceSync).not.toBeNull();
+    const child = spawn(process.execPath, ["--import", createRequire(import.meta.url).resolve("tsx"),
+      resolve(import.meta.dirname, "../../__tests__/helpers/native-drain-controller.test-worker.ts")],
+      { stdio: ["ignore", "pipe", "pipe", "ipc"], env: process.env });
+    const logs: string[] = [];
+    child.stdout?.on("data", data => logs.push(String(data)));
+    child.stderr?.on("data", data => logs.push(String(data)));
+    const exited = new Promise<number | null>(resolveExit => child.once("exit", resolveExit));
+    try {
+      const result = await new Promise<{ commands: number; errorName: string; errorMessage: string }>((resolveResult, reject) => {
+        const deadline = setTimeout(() => reject(new Error(`Controller handoff timed out: ${logs.join("").slice(-2000)}`)), 60_000);
+        child.once("error", error => { clearTimeout(deadline); reject(error); });
+        child.once("exit", code => { clearTimeout(deadline); reject(new Error(`Controller exited before handoff (${code}): ${logs.join("").slice(-2000)}`)); });
+        child.on("message", (message: { ready?: boolean; done?: boolean; commands: number; errorName: string; errorMessage: string; error?: string }) => {
+          if (message.ready) child.send({ connectionString: environment.current.temporary.connectionString, providerEvidence: mode === "provider event",
+            options: { execution, runnerInstanceId: fixture.native.runnerInstanceId, useRunnerd: true,
+              runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox } } });
+          if (message.done) { clearTimeout(deadline); resolveResult(message); }
+          if (message.error) { clearTimeout(deadline); reject(new Error(message.error)); }
+        });
+      });
+      expect(result).toMatchObject({ commands: 0, errorMessage: "local_runtime_api_listener_unavailable" });
+      expect(await exited).toBe(0);
+      const [handoff] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
+      if (mode === "provider event") {
+        expect(handoff).toMatchObject({ phase: "terminal_failure", nextAttemptAt: null,
+          failureDetail: { recoveryMode: "ambiguous_state", providerSessionEstablished: true } });
+        expect(await claimNativeSessionResumptions({ db: fixture.db, runnerInstanceId: "no-provider-replay",
+          runIds: [fixture.runId], deferIneligible: false })).toHaveLength(0);
+        return;
+      }
+      expect(handoff).toMatchObject({ phase: "retryable_failure", attempt: 1,
+        failureDetail: { originalFailureCode: "local_runtime_api_listener_unavailable", recoveryMode: "bootstrap_retry",
+          providerSessionEstablished: false, providerEventsExist: false, checkpointExists: false } });
+      if (mode === "cancelled task") {
+        await fixture.db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, fixture.issueId));
+      }
+      const claims = (await Promise.all(["successor-a", "successor-b"].map(runnerInstanceId =>
+        claimNativeSessionResumptions({ db: fixture.db, runnerInstanceId, runIds: [fixture.runId],
+          now: new Date(handoff.nextAttemptAt!.getTime() + 1), deferIneligible: false }),
+      ))).flat();
+      if (mode === "cancelled task") { expect(claims).toHaveLength(0); return; }
+      expect(claims).toHaveLength(1);
+      const [run] = await fixture.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, fixture.runId));
+      expect(run).toMatchObject({ status: "running", nativeSessionId: fixture.native.normalizedSessionId });
+      const [action] = await fixture.db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, fixture.issueId));
+      expect(action).toMatchObject({ ownerType: "agent", ownerAgentId: agentId });
+      if (process.env.PAPERCLIP_TEST_DRAIN_CONTINUATION === "1") {
+        const claim = claims[0];
+        const root = join(runtimeRoot, `continued-${fixture.runId}`);
+        const home = join(root, "codex-home");
+        await mkdir(home, { recursive: true });
+        const report = join(root, "result.json");
+        const calls = join(root, "calls.log");
+        await writeFile(report, JSON.stringify({ schema: "paperclip.run_result.v1", reportedWorkDisposition: "done",
+          summary: "Admitted work completed after controller restart.", completionClaim: {
+            contractRevision: execution.completionContract.contract.revision, objectiveSatisfied: true,
+            criteria: execution.completionContract.contract.criteria.map(criterion => ({ criterionId: criterion.id,
+              status: "satisfied", evidenceRefs: [] })), remainingWork: [] },
+          evidence: [], verification: [], artifacts: [], attentionRequests: [] }));
+        const bundle = createRunnerdCodexTransport({ ...transportOptions(fixture, root),
+          runnerBinary: defaultCapabilityRunnerdBinary(),
+          codexArgs: ["--state-file", join(root, "provider.json"), "--terminal-result-file", report,
+            "--call-log", calls, "--record-process-start"],
+          runtimeContext: execution.runtimeContext, runnerRuntimeContext: execution.runtimeContext,
+          sourceCodexHome: home, environment: { CODEX_HOME: home, PATH: process.env.PATH },
+          lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null },
+        });
+        const backend = createNativeSessionBackend(execution, { runnerInstanceId: fixture.native.runnerInstanceId,
+          environment: { CODEX_HOME: home, PATH: process.env.PATH }, codexTransportFactory: () => bundle.transport });
+        try {
+          expect(await executePaperclipNativeSession({ db: fixture.db, execution, backend, useRunnerd: true,
+            runnerInstanceId: fixture.native.runnerInstanceId, leaseOwner: claim.leaseOwner,
+            runnerExecutionTarget: { kind: "remote", transport: "sandbox", environmentId, leaseId, remoteCwd: remoteSandbox,
+              runner: { execute: async () => { throw new Error("Unexpected remote preparation after the controlled backend owns execution"); } } as never },
+            turnTimeoutMs: 15_000 })).toMatchObject({ exitCode: 0, timedOut: false });
+          const recorder = workspaceOperationService(fixture.db).createRecorder({ companyId,
+            issueId: fixture.issueId, heartbeatRunId: fixture.runId });
+          const saved = await recorder.recordOperation({ phase: "workspace_finalize", cwd: sandbox,
+            run: async () => { await workspaceSync!.restoreWorkspace(); return { status: "succeeded", exitCode: 0 }; } });
+          expect(saved.status).toBe("succeeded");
+          await reconcileNativeFinalizations(fixture.db, [fixture.runId]);
+          expect(await fixture.db.select().from(nativeRunResults).where(eq(nativeRunResults.runId, fixture.runId)))
+            .toMatchObject([{ schemaStatus: "accepted" }]);
+          const [finished] = await fixture.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, fixture.runId));
+          expect(finished, JSON.stringify(finished.failureDetail)).toMatchObject({ phase: "committed", attempt: 2 });
+          expect(await readFile(savedFile, "utf8")).toBe("Preserve admitted work.\n");
+          expect((await readFile(calls, "utf8")).split("\n").filter(line => line === "process-start")).toHaveLength(1);
+        } finally { await bundle.transport.close(); }
+      }
+    } finally { child.kill("SIGKILL"); await exited; await workspaceSync?.cleanup(); }
+  }, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS);
+
   it.each(["preparation", "claim lock"])("gives a fresh remote claim its full lease through blocked %s", async (blockedPhase) => {
     const fixture = await seedRun(`FRESH-SLOW-${blockedPhase === "claim lock" ? "CLAIM-LOCK" : "PREPARATION"}`);
     const environmentId = randomUUID(), leaseId = randomUUID();

server/src/services/native-runtime/native-safe-replacement.test.ts

@@ -16,6 +16,11 @@ import {
   deliverReconciledExecutions,
 } from "../execution-recovery-resolution.js";
 import { randomUUID } from "node:crypto";
+import { spawn } from "node:child_process";
+import { persistHeartbeatRunProcessMetadata } from "../heartbeat/run-lifecycle.js";
+import { hasNativeLocalProcessStop, releaseSettledNativeRunProcess } from "../native-local-process-stop.js";
+import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
+import { executionControlSchedule } from "../execution-control-work.js";
 import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
 import { tmpdir } from "node:os";
 import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
@@ -26,6 +31,7 @@ import {
   agents,
   companies,
   createDb,
+  subscribeDatabaseWork,
   heartbeatRunEvents,
   heartbeatRuns,
   issueRecoveryActions,
@@ -41,7 +47,7 @@ import {
   getEmbeddedPostgresTestSupport,
   startEmbeddedPostgresTestDatabase,
 } from "../../__tests__/helpers/embedded-postgres.js";
-import { reconcileSafeNativeReplacements } from "./native-safe-replacement.js";
+import { invalidateNativeReplacementDiscovery, reconcileSafeNativeReplacements } from "./native-safe-replacement.js";
 import { reconcileAbandonedExecutionControl } from "../execution-control-reconciliation.js";
 const externalDatabaseUrl = process.env.PAPERCLIP_TEST_DATABASE_URL;
 const support = externalDatabaseUrl
@@ -219,6 +225,103 @@ const support = externalDatabaseUrl
         }
       },
     );
+    it("wakes a completed replacement scan after actual local stop proof commits", async () => {
+      const source = await seed(2);
+      const workspace = await mkdtemp(join(tmpdir(), "paperclip-late-stop-"));
+      const child = spawn(process.execPath, ["-e", "process.exit(0)"], { detached: true, stdio: "ignore" });
+      const pid = child.pid!;
+      await new Promise<void>((resolve, reject) => { child.once("exit", () => resolve()); child.once("error", reject); });
+      await persistHeartbeatRunProcessMetadata(db, source.runId, { pid, processGroupId: pid,
+        startedAt: new Date().toISOString(), targetKind: "local" });
+      await db.update(heartbeatRuns).set({ finishedAt: new Date(), runnerProfileJson: {
+        nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } },
+      } }).where(eq(heartbeatRuns.id, source.runId));
+      await db.update(nativeRunFinalizations).set({ failureCode: "provider_transport_failed" })
+        .where(eq(nativeRunFinalizations.runId, source.runId));
+      await appendHeartbeatRunEvent(db, { companyId: source.companyId, runId: source.runId, agentId: source.agentId,
+        eventType: "harness.ready", stream: "system", message: "Provider started before its transport failed." });
+      const verifyStoppedSession = async () => await hasNativeLocalProcessStop(db, source.companyId, source.runId)
+        ? { evidence: {}, retire: () => true } : null;
+      invalidateNativeReplacementDiscovery(db);
+      expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 });
+      expect(await executionControlSchedule(db).replacement()).toBeNull();
+      let wakes = 0;
+      const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => {
+        if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); }
+      });
+      try {
+        await db.transaction(async tx => {
+          expect(await releaseSettledNativeRunProcess(tx as unknown as typeof db, source.runId)).toBe("released");
+          expect(wakes).toBe(0);
+        });
+        expect(wakes).toBeGreaterThan(0);
+        expect(await executionControlSchedule(db).replacement()).not.toBeNull();
+        expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 1 });
+        expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 });
+        expect(await executionControlSchedule(db).replacement()).toBeNull();
+        const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId));
+        expect(successors).toHaveLength(1);
+      } finally { unsubscribe(); await rm(workspace, { recursive: true, force: true }); }
+    });
+    it("wakes completed recovery when the real task becomes eligible", async () => {
+      const source = await seed();
+      const workspace = await mkdtemp(join(tmpdir(), "paperclip-late-eligibility-"));
+      await db.update(heartbeatRuns).set({ runnerProfileJson: { recoveryEventInventoryVersion: 1,
+        nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } },
+      } }).where(eq(heartbeatRuns.id, source.runId));
+      await issueService(db).update(source.issueId, { status: "backlog" });
+      invalidateNativeReplacementDiscovery(db);
+      expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
+      expect(await executionControlSchedule(db).replacement()).toBeNull();
+      let wakes = 0;
+      const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.executionReplacement, event => {
+        if (event === "settled") { wakes++; invalidateNativeReplacementDiscovery(db); }
+      });
+      try {
+        await issueService(db).update(source.issueId, { status: "in_progress" });
+        expect(wakes).toBeGreaterThan(0);
+        expect(await executionControlSchedule(db).replacement()).not.toBeNull();
+        expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 1 });
+        expect(await reconcileSafeNativeReplacements(db)).toMatchObject({ scheduled: 0 });
+        expect(await executionControlSchedule(db).replacement()).toBeNull();
+        expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId))).toHaveLength(1);
+      } finally { unsubscribe(); await rm(workspace, { recursive: true, force: true }); }
+    });
+    it("advances past a failed full page and retries it after later work progresses", async () => {
+      const workspace = await mkdtemp(join(tmpdir(), "paperclip-failed-page-"));
+      const sources: Awaited<ReturnType<typeof seed>>[] = [];
+      for (let index = 0; index < 26; index++) {
+        const source = await seed(); sources.push(source);
+        await db.update(heartbeatRuns).set({ runnerProfileJson: {
+          nativeExecutionInput: { provider: { kind: "codex" }, workspace: { cwd: workspace } },
+        } }).where(eq(heartbeatRuns.id, source.runId));
+        await db.update(nativeRunFinalizations).set({ failureCode: "provider_transport_failed" })
+          .where(eq(nativeRunFinalizations.runId, source.runId));
+      }
+      sources.sort((a, b) => a.runId.localeCompare(b.runId));
+      const later = sources.at(-1)!;
+      let failing = true;
+      const verifyStoppedSession = vi.fn(async (run: typeof heartbeatRuns.$inferSelect) => {
+        if (failing && run.id !== later.runId) throw new Error("Controlled stopped-session verifier failure");
+        return { evidence: {}, retire: () => true };
+      });
+      invalidateNativeReplacementDiscovery(db);
+      try {
+        await expect(reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession }))
+          .rejects.toThrow("native_replacement_decision_retry_required");
+        expect(verifyStoppedSession).toHaveBeenCalledTimes(25);
+        expect(await executionControlSchedule(db).replacement()).not.toBeNull();
+        expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 1 });
+        expect(verifyStoppedSession).toHaveBeenCalledTimes(26);
+        expect(await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, later.runId))).toHaveLength(1);
+        failing = false;
+        expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 25 });
+        expect(await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession })).toMatchObject({ scheduled: 0 });
+        expect(await executionControlSchedule(db).replacement()).toBeNull();
+        const successors = await db.select().from(heartbeatRuns).where(inArray(heartbeatRuns.retryOfRunId, sources.map(source => source.runId)));
+        expect(successors).toHaveLength(26);
+      } finally { await rm(workspace, { recursive: true, force: true }); }
+    }, 60_000);
     it.each(["unproven", "changed", "verified"] as const)("requires stopped-session proof through commit (%s)", async (mode) => {
       const source = await seed(2);
       await db.update(nativeRunFinalizations).set({ failureCode: "native_session_cleanup_quarantined" }).where(eq(nativeRunFinalizations.runId, source.runId));
@@ -235,8 +338,11 @@ const support = externalDatabaseUrl
         eventType: "tool.execution.started", stream: "system",
         payload: { name: "paperclip_finish", executionId: "completion", transport: "process" },
       });
-      await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession });
-      await reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession });
+      for (let sweep = 0; sweep < 2; sweep++) {
+        const decision = reconcileSafeNativeReplacements(db, new Date(), { verifyStoppedSession });
+        if (mode === "changed") await expect(decision).rejects.toThrow("native_replacement_decision_retry_required");
+        else await decision;
+      }
       const successors = await db.select().from(heartbeatRuns).where(eq(heartbeatRuns.retryOfRunId, source.runId));
       expect(successors).toHaveLength(mode === "verified" ? 1 : 0);
       const [hold] = await db.select().from(issueRecoveryActions).where(eq(issueRecoveryActions.sourceIssueId, source.issueId));

server/src/services/native-runtime/native-safe-replacement.ts

@@ -5,7 +5,7 @@ import { notifyExecutionRunWork } from "../execution-control-notifications.js";
 import { logger } from "../../middleware/logger.js";
 import { createHash, randomUUID } from "node:crypto";
 import { stat } from "node:fs/promises";
-import { and, eq, inArray, isNull, sql } from "drizzle-orm";
+import { and, asc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm";
 import {
   agentWakeupRequests,
   environmentLeases,
@@ -27,6 +27,8 @@ import { issueRecoveryActionService } from "../issue-recovery-actions.js";
 import { buildExecutionContinuation } from "../execution-continuation.js";
 import { appendHeartbeatRunEvent } from "../heartbeat-run-events.js";
 import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
+import { NATIVE_PROVIDER_DURABLE_EVENT_TYPES } from "./native-retired-backup-source.js";
+import { completedWorkspaceRecoveryCondition } from "../workspace-recovery-receipt.js";
 
 export const NATIVE_SAFE_REPLACEMENT_REASON = "native_safe_replacement";
 const record = (v: unknown): Record<string, unknown> =>
@@ -45,6 +47,66 @@ const processAlive = (pid: number | null) => {
 
 type ReplacementDb = Db | Parameters<Parameters<Db["transaction"]>[0]>[0];
 
+const listenerUnavailable = "local_runtime_api_listener_unavailable";
+function historicalListenerGuardPending() {
+  const profile = heartbeatRuns.runnerProfileJson, detail = nativeRunFinalizations.failureDetail;
+  const providerEvents = sql.join([...NATIVE_PROVIDER_DURABLE_EVENT_TYPES].map(value => sql`${value}`), sql`, `);
+  return and(
+    eq(nativeRunFinalizations.failureCode, "native_session_interrupted"),
+    eq(heartbeatRuns.errorCode, "native_session_interrupted"), eq(heartbeatRuns.error, listenerUnavailable),
+    sql`${detail}->>'message' = ${listenerUnavailable}`,
+    sql`${detail}->>'originalFailureCode' = 'native_session_interrupted'`,
+    sql`${detail}->>'recoveryMode' = 'ambiguous_state'`,
+    sql`${detail}->'providerSessionEstablished' = 'false'::jsonb`,
+    sql`${detail}->'providerEventsExist' = 'false'::jsonb`,
+    sql`${detail}->'checkpointExists' = 'false'::jsonb`,
+    eq(nativeRunFinalizations.attempt, 1),
+    isNull(nativeRunFinalizations.leaseOwner), isNull(nativeRunFinalizations.leaseExpiresAt),
+    isNull(nativeRunFinalizations.assessmentId), isNull(nativeRunFinalizations.decisionId),
+    isNull(heartbeatRuns.processPid), isNull(heartbeatRuns.processGroupId),
+    isNull(heartbeatRuns.sessionIdBefore), isNull(heartbeatRuns.sessionIdAfter), isNull(heartbeatRuns.retryOfRunId),
+    sql`not (coalesce(${heartbeatRuns.resultJson}, '{}'::jsonb) ?| array['cancellation', 'nativeCancellation'])`,
+    sql`coalesce(${profile}->'sessionCheckpoint', 'null'::jsonb) = 'null'::jsonb`,
+    sql`coalesce(${profile}->'nativeProviderPackAdmission', 'null'::jsonb) = 'null'::jsonb`,
+    sql`coalesce(${profile}#>>'{nativeProviderPackAdmissionState,state}', 'pending') = 'pending'`,
+    sql`${profile}#>>'{nativeExecutionInput,binding,runId}' = ${heartbeatRuns.id}::text`,
+    sql`${profile}#>>'{nativeExecutionInput,binding,companyId}' = ${heartbeatRuns.companyId}::text`,
+    sql`${profile}#>>'{nativeExecutionInput,binding,agentId}' = ${heartbeatRuns.agentId}::text`,
+    sql`${profile}#>>'{nativeExecutionInput,binding,issueId}' = ${nativeRunFinalizations.issueId}::text`,
+    sql`${profile}#>>'{nativeExecutionInput,session,normalizedSessionId}' = ${heartbeatRuns.nativeSessionId}::text`,
+    sql`not exists (select 1 from ${heartbeatRuns} prior where prior.company_id = ${heartbeatRuns.companyId}
+      and prior.native_session_id = ${heartbeatRuns.nativeSessionId} and prior.id <> ${heartbeatRuns.id})`,
+    sql`not exists (select 1 from ${heartbeatRunEvents} event where event.company_id = ${heartbeatRuns.companyId}
+      and event.run_id = ${heartbeatRuns.id} and (event.source_event_id is not null
+        or event.event_type in (${providerEvents}) or event.event_type ~ '^(turn|item|tool|semantic_tool|run[.]result)[.]'))`,
+    sql`not exists (select 1 from ${toolInvocations} invocation where invocation.company_id = ${heartbeatRuns.companyId}
+      and invocation.run_id = ${heartbeatRuns.id})`,
+    sql`not exists (select 1 from ${issueRecoveryActions}
+      where ${issueRecoveryActions.companyId} = ${heartbeatRuns.companyId}
+        and ${issueRecoveryActions.sourceIssueId} = ${nativeRunFinalizations.issueId}
+        and ${issueRecoveryActions.cause} = 'native_continuation_requires_reconciliation'
+        and ${issueRecoveryActions.evidence}->>'runId' = ${heartbeatRuns.id}::text
+        and ${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'
+        and not coalesce((${completedWorkspaceRecoveryCondition(heartbeatRuns.id)}), false))`,
+    sql`exists (select 1 from ${issues} task where task.company_id = ${heartbeatRuns.companyId}
+      and task.id = ${nativeRunFinalizations.issueId} and task.assignee_agent_id = ${heartbeatRuns.agentId}
+      and task.status in ('todo', 'in_progress', 'in_review', 'blocked')
+      and (task.status <> 'blocked' or exists (select 1 from ${issueRecoveryActions} hold
+        where hold.company_id = task.company_id and hold.source_issue_id = task.id and hold.kind = 'active_run_watchdog'
+          and hold.cause = ${nativeRunFinalizations.failureCode} and hold.evidence->>'runId' = ${heartbeatRuns.id}::text
+          and hold.evidence#>>'{nativeFailureBlock,runId}' = ${heartbeatRuns.id}::text
+          and hold.evidence#>'{nativeFailureBlock,statusVersion}' = to_jsonb(task.status_version))))`,
+  );
+}
+
+async function historicalListenerGuardEvidence(db: ReplacementDb, run: typeof heartbeatRuns.$inferSelect) {
+  const [evidence] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns)
+    .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id)).where(and(
+      eq(heartbeatRuns.companyId, run.companyId), eq(heartbeatRuns.id, run.id), historicalListenerGuardPending(),
+    )).limit(1);
+  return Boolean(evidence);
+}
+
 /** A saved remote source is proved by its current ledger, never a controller path. */
 async function remoteWorkspaceReplacementEvidence(
   db: ReplacementDb,
@@ -113,16 +175,95 @@ export function nativeReplacementPending() {
   return and(
     eq(heartbeatRuns.status, "failed"),
     eq(nativeRunFinalizations.phase, "terminal_failure"),
-    inArray(
+    or(inArray(
       nativeRunFinalizations.failureCode,
       ["native_provider_terminal_failed", "native_session_cleanup_quarantined", "provider_transport_failed"],
-    ),
+    ), historicalListenerGuardPending()),
     isNull(nativeRunFinalizations.resultId),
     sql`coalesce(${nativeRunFinalizations.failureDetail}->>'successorRunId', '') = ''`,
     sql`coalesce(${nativeRunFinalizations.failureDetail}->>'replacementDenied', '') = ''`,
   );
 }
 
+const REPLACEMENT_SCAN_LIMIT = 100;
+const replacementDiscovery = new WeakMap<object, { after: string | undefined; complete: boolean; generation: number; rescan: boolean }>();
+function replacementDiscoveryState(db: Db) {
+  let state = replacementDiscovery.get(db);
+  if (!state) { state = { after: undefined, complete: false, generation: 0, rescan: false }; replacementDiscovery.set(db, state); }
+  return state;
+}
+
+export function invalidateNativeReplacementDiscovery(db: Db) {
+  const state = replacementDiscoveryState(db);
+  state.complete = false; state.rescan = true; state.generation++;
+}
+
+async function replacementHistoryReady(db: Db, run: typeof heartbeatRuns.$inferSelect, issueId: string, retryDatabaseErrors = false) {
+  try {
+    const continuation = await buildExecutionContinuation({ db, companyId: run.companyId, issueId,
+      agentId: run.agentId, context: record(run.contextSnapshot), summary: null, exposeLowTrustRaw: false });
+    return continuation.unresolvedInteractionIds.length === 0;
+  } catch (error) {
+    let cause: unknown = error;
+    for (let depth = 0; retryDatabaseErrors && depth < 4 && cause && typeof cause === "object"; depth++) {
+      const failure = cause as { code?: unknown; cause?: unknown };
+      if (typeof failure.code === "string" && /^[A-Z0-9_]+$/.test(failure.code)) throw error;
+      cause = failure.cause;
+    }
+    return null;
+  }
+}
+
+async function discoverNativeReplacements(db: Db, after: string | undefined, limit = 25) {
+  const candidates: Array<{ run: typeof heartbeatRuns.$inferSelect; coordinator: typeof nativeRunFinalizations.$inferSelect }> = [];
+  let scanned = 0;
+  for (;;) {
+    const rows = await db.select({ run: heartbeatRuns, coordinator: nativeRunFinalizations }).from(heartbeatRuns)
+      .innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
+      .where(and(nativeReplacementPending(), after ? gt(heartbeatRuns.id, after) : undefined))
+      .orderBy(asc(heartbeatRuns.id)).limit(Math.min(25, REPLACEMENT_SCAN_LIMIT - scanned));
+    if (!rows.length) return { candidates, scanned, after: undefined };
+    for (const row of rows) {
+      after = row.run.id; scanned++;
+      let ready = true;
+      if (row.coordinator.failureCode === "native_session_interrupted") {
+        ready = false;
+        if (row.run.runnerProfileJson?.remoteWorkspace != null) {
+          const leases = await db.select().from(environmentLeases).where(and(
+            eq(environmentLeases.companyId, row.run.companyId), eq(environmentLeases.heartbeatRunId, row.run.id),
+          ));
+          if (leases.every(lease => lease.releasedAt !== null)) {
+            const evidence = await remoteWorkspaceReplacementEvidence(db, row.run, row.coordinator.issueId, leases);
+            ready = Boolean(evidence?.workspacePreserved && evidence.providerStopped)
+              && await replacementHistoryReady(db, row.run, row.coordinator.issueId, true) === true;
+          }
+        }
+      }
+      if (ready) candidates.push(row);
+      if (candidates.length >= limit || scanned >= REPLACEMENT_SCAN_LIMIT) return { candidates, scanned, after };
+    }
+    if (rows.length < 25) return { candidates, scanned, after: undefined };
+  }
+}
+
+function settleReplacementDiscovery(state: ReturnType<typeof replacementDiscoveryState>, after: string | undefined, generation: number) {
+  const rescan = state.rescan || generation !== state.generation;
+  state.after = after; state.complete = !after && !rescan;
+  state.rescan = Boolean(after && rescan);
+}
+
+export async function hasPendingNativeReplacements(db: Db) {
+  const state = replacementDiscoveryState(db);
+  if (state.complete) return false;
+  if (state.after) return true;
+  state.rescan = false;
+  const generation = state.generation;
+  const result = await discoverNativeReplacements(db, undefined, 1);
+  if (result.candidates.length) return true;
+  settleReplacementDiscovery(state, result.after, generation);
+  return !state.complete;
+}
+
 /** Retryable outbox: terminal_failure plus a new-format failure code is the durable candidate. */
 export async function reconcileSafeNativeReplacements(
   db: Db,
@@ -136,18 +277,14 @@ export async function reconcileSafeNativeReplacements(
     failpoint?: (phase: "successor_inserted" | "lineage_committed") => void;
   } = {},
 ) {
-  const candidates = await db
-    .select({ run: heartbeatRuns, coordinator: nativeRunFinalizations })
-    .from(heartbeatRuns)
-    .innerJoin(
-      nativeRunFinalizations,
-      eq(nativeRunFinalizations.runId, heartbeatRuns.id),
-    )
-    .where(
-      nativeReplacementPending(),
-    )
-    .limit(25);
-  let scheduled = 0;
+  const discovery = replacementDiscoveryState(db);
+  if (discovery.complete) discovery.after = undefined;
+  discovery.complete = false;
+  if (!discovery.after) discovery.rescan = false;
+  const generation = discovery.generation;
+  const selection = await discoverNativeReplacements(db, discovery.after);
+  const candidates = selection.candidates;
+  let scheduled = 0, pendingFailures = 0;
   for (const { run, coordinator } of candidates) {
     try {
       if (
@@ -170,11 +307,13 @@ export async function reconcileSafeNativeReplacements(
         );
       // Teardown is still in progress. The next sweep rechecks its durable outcome.
       if (leases.some((lease) => lease.releasedAt === null)) continue;
-      const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed"
+      const historicalListener = coordinator.failureCode === "native_session_interrupted";
+      if (historicalListener && !await historicalListenerGuardEvidence(db, run)) continue;
+      const stoppedSession = coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener
         ? await options.verifyStoppedSession?.(run) ?? null : null;
       // A transport label alone is not evidence. Keep inspecting these candidates
       // as process cleanup and the final transcript become durable.
-      if (coordinator.failureCode !== "native_provider_terminal_failed" && !stoppedSession) continue;
+      if (coordinator.failureCode !== "native_provider_terminal_failed" && !historicalListener && !stoppedSession) continue;
       const invocations = await db
         .select()
         .from(toolInvocations)
@@ -263,23 +402,11 @@ export async function reconcileSafeNativeReplacements(
           ? [String(p.kind ?? p.type)]
           : [];
       });
-      let historyComplete = false;
-      try {
-        const continuation = await buildExecutionContinuation({
-          db,
-          companyId: run.companyId,
-          issueId: coordinator.issueId,
-          agentId: run.agentId,
-          context: record(run.contextSnapshot),
-          summary: null,
-          exposeLowTrustRaw: false,
-        });
-        if (continuation.unresolvedInteractionIds.length > 0) continue;
-        historyComplete = true;
-      } catch {
-        /* The named operator outcome retains incomplete context. */
-      }
+      const history = await replacementHistoryReady(db, run, coordinator.issueId);
+      if (history === false) continue;
+      const historyComplete = history === true;
       const remoteWorkspace = await remoteWorkspaceReplacementEvidence(db, run, coordinator.issueId, leases);
+      if (historicalListener && (!remoteWorkspace?.workspacePreserved || !remoteWorkspace.providerStopped || invocations.length)) continue;
       const decision = decideNativeReplacement({
         failedSession: true,
         // A facade transport failure can conceal an authoritative protocol
@@ -413,6 +540,13 @@ export async function reconcileSafeNativeReplacements(
           current.attempt >= 3
         )
           return false;
+        let historicalRun: typeof heartbeatRuns.$inferSelect | undefined;
+        if (historicalListener) {
+          const [currentRun] = await tx.select().from(heartbeatRuns).where(and(
+            eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, run.companyId),
+          )).for("update");
+          historicalRun = currentRun;
+        }
         if (task.status === "blocked") {
           const failureHolds = await tx.select().from(issueRecoveryActions).where(and(
             eq(issueRecoveryActions.companyId, run.companyId),
@@ -427,6 +561,14 @@ export async function reconcileSafeNativeReplacements(
           });
           if (!ownsBlock) return false;
         }
+        const preservationHolds = historicalListener ? await tx.select().from(issueRecoveryActions).where(and(
+          eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, task.id),
+          eq(issueRecoveryActions.kind, "active_run_watchdog"),
+          eq(issueRecoveryActions.cause, "native_continuation_requires_reconciliation"),
+          sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
+          sql`${issueRecoveryActions.evidence}->'automaticRecovery'->>'replay' = 'blocked'`,
+        )).for("update") : [];
+        if (historicalListener && (!historicalRun || !await historicalListenerGuardEvidence(tx, historicalRun))) return false;
         if (remoteWorkspace) {
           const currentWorkspace = await remoteWorkspaceReplacementEvidence(tx, run, coordinator.issueId, leases, true);
           if (!currentWorkspace?.workspacePreserved || !currentWorkspace.providerStopped) return false;
@@ -548,6 +690,7 @@ export async function reconcileSafeNativeReplacements(
           eq(issueRecoveryActions.cause, current.failureCode!),
           sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
         )).for("update");
+        holds.push(...preservationHolds);
         if (holds.some(hold => hold.status === "active" || hold.status === "escalated")) {
           await notifyUnsafeWorkspaceExportRecovery(tx, { companyId: run.companyId, issueId: task.id });
         }
@@ -587,11 +730,17 @@ export async function reconcileSafeNativeReplacements(
       }
     } catch (error) {
       if (options.failpoint) throw error;
+      pendingFailures++;
       logger.warn(
         { runId: run.id },
         "Native replacement remains pending; another sweep will retry its durable decision",
       );
     }
   }
-  return { scanned: candidates.length, scheduled };
+  settleReplacementDiscovery(discovery, selection.after, generation);
+  if (pendingFailures) {
+    discovery.complete = false; discovery.rescan = true;
+    throw new Error("native_replacement_decision_retry_required");
+  }
+  return { scanned: selection.scanned, scheduled };
 }