perf(native): parallelize durability eligibility checks (#20646)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A web server must prove that retained run state is durable before
volume-free enrollment.
> - The proof reads selected native sessions and validates their
existing recovery receipts.
> - Serial database checks made a valid inventory take 59.665 seconds.
> - The signed preparation request has a 55-second limit.
> - This pull request overlaps independent reads within fixed limits.
> - The benefit is a faster eligibility check with the same preservation
rules.

## Linked Issues or Issue Description

Refs #20599 and #20601. A search found no duplicate of this remaining
serial-read bottleneck.

**What happened?**

An actual full startup eligibility check returned eligible with no
blockers in 59.665 seconds. Controller checks took 44.134 seconds and
harness checks took 8.592 seconds. Signed preparation timed out at
55.002 seconds.

**Expected behavior**

A durable instance can complete signed preparation within its existing
deadline.

**Steps to reproduce**

1. Retain many native task sessions with database-backed controller
checkpoints and valid filesystem-independence receipts.
2. Run the full startup eligibility check.
3. Observe serial checkpoint and receipt reads consume the signed
request budget.

## What Changed

- Check the existing terminal-controller metadata and
checkpoint-existence predicate within the existing eight-reader source
batch.
- Validate harness independence receipts with at most four concurrent
readers per existing page.
- Stop admitting more harness reads after an invalid result or error.
Join all started reads before returning or throwing.
- Keep large controller, full-checkpoint and legacy archive reads
serial.
- Add real PostgreSQL regressions for overlap, concurrency limits and
failure settlement.

## Verification

- Genuine controller regression RED: all 16 checkpoint reads ran, but
peak overlap was one. GREEN: 10 focused PostgreSQL controls passed,
including bounded overlap, joined failures, active and warm handoffs,
and failed-source preservation. The other 45 cases were filtered, not
claimed as passes.
- Genuine harness regression RED: overlap was one instead of four.
GREEN: all three new PostgreSQL controls passed. Six existing authority,
replacement and refusal controls also passed.
- Final head `c059e031`: the same 10 controller and 9 harness PostgreSQL
controls passed after rebasing onto current master and refreshing Runner
artifacts. `node node_modules/typescript/bin/tsc --noEmit -p
server/tsconfig.json` also passed. The first `pnpm exec` attempt could
not start the compiler because package-manager bootstrap fetches failed;
it is not counted as a typecheck pass.
- `git diff --check` passed.
- Actual private-process acceptance with deployed `8e4c5e81`
dependencies and only the two changed owners: the complete check passed
in 37.596 seconds with no blockers. Nine runs were running before and
after; 1,750 sessions were selected. This is 17.4 seconds inside the
unchanged request limit. The earlier complete baseline was 59.665
seconds on 1,745 selected sessions, so these are observed timings on a
changing dataset, not a controlled exact speedup. No serving process was
patched.
- The first private candidate check stopped on a native-controller
blocker in 18.216 seconds. A targeted follow-up found no blocker without
any preservation or business write. Its cause was not identified. It is
retained as a nonpass and is not a speedup result.
- Current-head Greptile review is 5/5 with no actionable findings.
Required CI is running:
https://github.com/paperclipai/paperclip/actions/runs/42898013184.

## Risks

- The scans now use more simultaneous database reads. Limits remain
eight for controller source readers and four for harness proofs.
- Invalid or failed proofs still refuse eligibility. All started readers
settle before failure leaves the scan.
- Previous-release state carries forward unchanged. This changes no
stored shape, checkpoint identity, lease rule, schema, deadline or
provider protocol.
- Large legacy payload reads remain serial, so their existing memory
bound is unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. This is a measured
performance defect in the existing enrollment path.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, code execution and
parallel agents. The runtime does not expose the exact model variant or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/services/native-runtime/native-controller-migration.integration.test.ts

@@ -11,6 +11,7 @@ import * as sessionResume from "./native-session-resume.js";
 import { nativeControllerMigrationDescriptor } from "./native-session-executor.js";
 import { hasLocalNativeControllerObligations, migrateNativeControllers, prestageNativeControllers } from "./native-controller-migration.js";
 import { insertMigratedNativeRunnerCheckpoint, loadNativeRunnerCheckpoint, nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js";
+import * as checkpointReader from "./native-runner-checkpoint.js";
 import { currentNativeControllerIdentity } from "./native-restart-recovery.js";
 import { createNativeControllerRecovery, resolveNativeControllerRecovery } from "./native-controller-recovery.js";
 import { resolveRemoteProviderPackAuthority } from "./remote-provider-pack-authority.js";
@@ -189,8 +190,11 @@ describeEmbeddedPostgres.each(["archive_online", "archive_online_owned", "archiv
     await f.db.insert(agentTaskSessions).values(Array.from({ length: 15 }, () => ({ companyId: f.companyId,
       agentId: f.agentId, adapterType: "paperclip_runner", taskKey: randomUUID(), lastRunId: f.runId })));
     const original = sessionResume.findNativeSessionResumeRun;
+    const originalCheckpoint = checkpointReader.hasNativeRunnerCheckpoint;
     let started = 0, finished = 0, inFlight = 0, peak = 0, fail = false;
+    let checkpointStarted = 0, checkpointFinished = 0, checkpointInFlight = 0, checkpointPeak = 0, checkpointFail = false;
     const reason = new Error("retained source reader failed");
+    const checkpointReason = new Error("durable checkpoint reader failed");
     const spy = vi.spyOn(sessionResume, "findNativeSessionResumeRun").mockImplementation(async (...args) => {
       const first = ++started === 1; peak = Math.max(peak, ++inFlight);
       try {
@@ -199,16 +203,33 @@ describeEmbeddedPostgres.each(["archive_online", "archive_online_owned", "archiv
         return await original(...args);
       } finally { inFlight--; finished++; }
     });
+    const checkpointSpy = vi.spyOn(checkpointReader, "hasNativeRunnerCheckpoint").mockImplementation(async (...args) => {
+      const first = ++checkpointStarted === 1;
+      checkpointPeak = Math.max(checkpointPeak, ++checkpointInFlight);
+      try {
+        await new Promise(resolve => setTimeout(resolve, first && checkpointFail ? 1 : 20));
+        if (first && checkpointFail) throw checkpointReason;
+        return await originalCheckpoint(...args);
+      } finally { checkpointInFlight--; checkpointFinished++; }
+    });
     try {
       const before = performance.now();
       expect(await hasLocalNativeControllerObligations(f.db)).toBe(false);
-      console.info(JSON.stringify({ state: "retained_source_readers", elapsedMs: performance.now() - before, peak, started, finished }));
+      console.info(JSON.stringify({ state: "retained_source_readers", elapsedMs: performance.now() - before, peak, started, finished,
+        checkpointStarted, checkpointFinished, checkpointPeak }));
       expect(started).toBe(16); expect(finished).toBe(16); expect(peak).toBeGreaterThan(1); expect(peak).toBeLessThanOrEqual(8);
+      expect(checkpointStarted).toBe(16); expect(checkpointFinished).toBe(16);
+      expect(checkpointPeak).toBeGreaterThan(1); expect(checkpointPeak).toBeLessThanOrEqual(8);
       started = finished = peak = 0; fail = true;
       await expect(hasLocalNativeControllerObligations(f.db)).rejects.toBe(reason);
       expect(started).toBe(8); expect(finished).toBe(8); expect(inFlight).toBe(0);
+      started = finished = peak = 0; fail = false;
+      checkpointStarted = checkpointFinished = checkpointPeak = 0; checkpointFail = true;
+      await expect(hasLocalNativeControllerObligations(f.db)).rejects.toBe(checkpointReason);
+      expect(checkpointStarted).toBe(8); expect(checkpointFinished).toBe(8); expect(checkpointInFlight).toBe(0);
+      expect(started).toBe(8); expect(finished).toBe(8); expect(inFlight).toBe(0);
       expect(await readFile(f.file, "utf8")).toBe(f.source);
-    } finally { spy.mockRestore(); }
+    } finally { checkpointSpy.mockRestore(); spy.mockRestore(); }
   });
   if (scenario === "active_durable") it("accepts a complete active database authority without terminal metadata and refuses stale or incomplete handoff proof", async () => {
     const f = await fixture(1, false, true), controller = await currentNativeControllerIdentity();

server/src/services/native-runtime/native-controller-migration.ts

@@ -77,7 +77,7 @@ async function hasConfirmedOlderController(db: Db, selected: Selected,
   } catch { return false; }
 }
 
-async function* retainedSources(db: Db, signal?: AbortSignal, concurrency = 1) {
+async function* retainedSources(db: Db, signal?: AbortSignal, concurrency = 1, inspectDatabaseRecovery = false) {
   let after: string | undefined;
   for (;;) {
     signal?.throwIfAborted();
@@ -89,15 +89,20 @@ async function* retainedSources(db: Db, signal?: AbortSignal, concurrency = 1) {
         ${after ? sql`and s.id > ${after}::uuid` : sql``} order by s.id limit 100`);
     for (let offset = 0; offset < rows.length; offset += concurrency) {
       signal?.throwIfAborted();
-      const batch = await Promise.allSettled(rows.slice(offset, offset + concurrency).map(async selected => ({
-        selected, source: await findNativeSessionResumeRun(db, { companyId: selected.companyId, agentId: selected.agentId,
+      const batch = await Promise.allSettled(rows.slice(offset, offset + concurrency).map(async selected => {
+        const source = await findNativeSessionResumeRun(db, { companyId: selected.companyId, agentId: selected.agentId,
           issueId: selected.issueId, normalizedSessionId: selected.sessionId,
-          beforeCreatedAt: new Date(selected.createdAt), includeProgressBarrier: true }),
-      })));
+          beforeCreatedAt: new Date(selected.createdAt), includeProgressBarrier: true });
+        const databaseRecoveryPreserved = Boolean(inspectDatabaseRecovery && source && source.status !== "running"
+          && readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint)
+          && await hasNativeRunnerCheckpoint(db, { companyId: selected.companyId, issueId: selected.issueId, runId: source.id }));
+        return { selected, source, databaseRecoveryPreserved };
+      }));
       for (const result of batch) {
         signal?.throwIfAborted();
         if (result.status === "rejected") throw result.reason;
-        if (result.value.source) yield { selected: result.value.selected, source: result.value.source };
+        if (result.value.source) yield { selected: result.value.selected, source: result.value.source,
+          databaseRecoveryPreserved: result.value.databaseRecoveryPreserved };
       }
     }
     if (rows.length < 100) return;
@@ -256,11 +261,9 @@ export async function hasLocalNativeControllerObligations(db: Db, storage?: Stor
     if (error.code === "ENOENT") return [] as string[]; throw error;
   });
   if (!roots.length) return false;
-  for await (const { selected, source } of retainedSources(db, undefined, 8)) {
+  for await (const { selected, source, databaseRecoveryPreserved } of retainedSources(db, undefined, 8, true)) {
     const binding = { companyId: selected.companyId, issueId: selected.issueId, runId: source.id };
-    if (source.status !== "running"
-      && readNativeControllerRecovery(source.runnerProfileJson.nativeControllerRecovery, source.runnerProfileJson.sessionCheckpoint)
-      && await hasNativeRunnerCheckpoint(db, binding)) continue;
+    if (databaseRecoveryPreserved) continue;
     if (source.status === "failed" && source.runnerProfileJson.sessionCheckpoint == null
       && preservedFailedCheckpoint(source, selected.sessionId, await loadNativeRunnerCheckpoint(db, binding))) continue;
     const paths = await localPaths(source);

server/src/services/native-runtime/native-harness-backup-migration.ts

@@ -12,6 +12,8 @@ import { NativeHarnessBackupSourceError, readNativeHarnessBackupSource } from ".
 import { settledNativeHarnessBackupOwner } from "./native-harness-preservation-source.js";
 import { hasValidNativeHarnessFilesystemIndependence } from "./native-harness-filesystem-independence.js";
 
+const MAX_PARALLEL_INDEPENDENCE_CHECKS = 4;
+
 export async function hasLocalNativeHarnessObligations(db: Db): Promise<boolean> {
   let afterSession = "", afterScope = "";
   for (;;) {
@@ -31,9 +33,19 @@ export async function hasLocalNativeHarnessObligations(db: Db): Promise<boolean>
         and b.scope_digest = e.metadata->'nativeHarnessBackup'->>'sessionScopeSha256' and b.status = 'current')
       and (s.id::text, coalesce(e.metadata->'nativeHarnessBackup'->>'sessionScopeSha256', '')) > (${afterSession}, ${afterScope})
     order by selected_session_id, scope_digest limit 100`);
-    for (const row of rows) if (!row.has_receipt || !await hasValidNativeHarnessFilesystemIndependence(db, {
-      companyId: row.company_id, selectedSessionId: row.selected_session_id, scopeDigest: row.scope_digest,
-    })) return true;
+    let next = 0, invalid = false;
+    const checked = await Promise.allSettled(Array.from({ length: Math.min(MAX_PARALLEL_INDEPENDENCE_CHECKS, rows.length) }, async () => {
+      while (!invalid && next < rows.length) {
+        const row = rows[next++]!;
+        try {
+          if (!row.has_receipt || !await hasValidNativeHarnessFilesystemIndependence(db, {
+            companyId: row.company_id, selectedSessionId: row.selected_session_id, scopeDigest: row.scope_digest,
+          })) invalid = true;
+        } catch (error) { invalid = true; throw error; }
+      }
+    }));
+    for (const result of checked) if (result.status === "rejected") throw result.reason;
+    if (invalid) return true;
     if (rows.length < 100) return false;
     afterSession = rows.at(-1)!.selected_session_id; afterScope = rows.at(-1)!.scope_digest;
   }

server/src/services/native-runtime/native-harness-filesystem-independence.integration.test.ts

@@ -18,6 +18,7 @@ import { preserveRuntimeSkillAssets } from "./runtime-skill-preservation.js";
 import { nativeControllerMigrationDescriptor, scopedRunnerdStateRoot } from "./native-session-executor.js";
 import { hasLocalNativeHarnessObligations } from "./native-harness-backup-migration.js";
 import { preserveNativeHarnessFilesystemIndependence, hasValidNativeHarnessFilesystemIndependence } from "./native-harness-filesystem-independence.js";
+import * as filesystemIndependence from "./native-harness-filesystem-independence.js";
 import { originalHarnessVolumeAnchorPath, verifyOriginalHarnessVolumeSource, type OriginalHarnessVolumeAnchor,
   type ReviewedOriginalHarnessVolumeWitness } from "./native-harness-original-volume.js";
 import { removeRuntimeAssetTree } from "./runtime-asset-retention.js";
@@ -197,6 +198,64 @@ describeEmbeddedPostgres("owner-verified native harness filesystem independence"
     return { ...f, execution, controllerPath, raw, supervisor, supervisorPath: descriptor.supervisorPath, identity };
   }
 
+  it("validates retained WEB receipts with bounded parallel database proofs", async () => {
+    const fixtures = [];
+    for (let index = 0; index < 5; index++) {
+      const f = await controllerFixture(true, "none");
+      await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" });
+      fixtures.push(f);
+    }
+    const validate = filesystemIndependence.hasValidNativeHarnessFilesystemIndependence;
+    let active = 0, maximum = 0, completed = 0, release!: () => void;
+    const blocked = new Promise<void>(resolve => { release = resolve; });
+    const reader = vi.spyOn(filesystemIndependence, "hasValidNativeHarnessFilesystemIndependence").mockImplementation(async (...args) => {
+      active++; maximum = Math.max(maximum, active);
+      try { await blocked; const valid = await validate(...args); completed++; return valid; }
+      finally { active--; }
+    });
+    const scan = hasLocalNativeHarnessObligations(db);
+    try {
+      await vi.waitFor(() => expect(active).toBe(4));
+      expect(reader).toHaveBeenCalledTimes(4);
+      release();
+      expect(await scan).toBe(false);
+      expect(maximum).toBe(4); expect(completed).toBe(fixtures.length); expect(active).toBe(0);
+      expect(reader).toHaveBeenCalledTimes(fixtures.length);
+    } finally { release(); await scan; }
+  });
+
+  it.each(["invalid", "error"] as const)("joins started WEB proofs after an %s result without admitting more", async outcome => {
+    for (let index = 0; index < 5; index++) {
+      const f = await controllerFixture(true, "none");
+      await preserveNativeHarnessFilesystemIndependence(db, { ...f.input, mode: "web_evidence" });
+    }
+    const validate = filesystemIndependence.hasValidNativeHarnessFilesystemIndependence;
+    let started = 0, active = 0, settled = false, releaseFirst!: () => void, releasePeers!: () => void, failed!: () => void;
+    const first = new Promise<void>(resolve => { releaseFirst = resolve; });
+    const peers = new Promise<void>(resolve => { releasePeers = resolve; });
+    const failure = new Promise<void>(resolve => { failed = resolve; });
+    const expected = new Error("fixture_independence_reader_failed");
+    vi.spyOn(filesystemIndependence, "hasValidNativeHarnessFilesystemIndependence").mockImplementation(async (...args) => {
+      const index = started++; active++;
+      try {
+        const valid = await validate(...args);
+        if (index === 0) { await first; failed(); if (outcome === "error") throw expected; return false; }
+        await peers; return valid;
+      } finally { active--; }
+    });
+    const scan = hasLocalNativeHarnessObligations(db);
+    void scan.then(() => { settled = true; }, () => { settled = true; });
+    try {
+      await vi.waitFor(() => expect(started).toBe(4));
+      releaseFirst(); await failure; await new Promise<void>(resolve => setImmediate(resolve));
+      expect(settled).toBe(false); expect(started).toBe(4); expect(active).toBe(3);
+      releasePeers();
+      if (outcome === "error") await expect(scan).rejects.toBe(expected);
+      else expect(await scan).toBe(true);
+      expect(active).toBe(0); expect(started).toBe(4);
+    } finally { releaseFirst(); releasePeers(); await scan.catch(() => undefined); }
+  });
+
   it("preserves WEB-only evidence with a succeeded source and unresolved external lease", async () => {
     const f = await controllerFixture(true, "deleted");
     if (!("runtimeContext" in f.execution)) throw new Error("fixture runtime context missing");