Record isolated native recovery load qualification and reproductions

Co-Authored-By: Paperclip <noreply@paperclip.ing>

doc/plans/2026-10-30-native-controller-checkpoints.md

@@ -515,3 +515,58 @@ the next replacement still continues the same provider history. Prepared-task
 resumed and failed-read/retry cases also pass for Codex and Claude fixtures.
 These are scripted-provider component results with remote workspace storage
 available, not live Daytona or distinct-release acceptance.
+
+
+### Bounded qualification campaign on October 30
+
+The follow-up campaign began at 18:33 UTC with an approximately one-hour budget.
+Three GPT-6.1 Sol workers own live Daytona, database load, and distinct-reader
+replacement/rollback; the coordinator owns integration, artifact builds and review.
+The exact native implementation under test is `a0ef3ce7493c8bb76c4766ff007e8a390118d145`.
+Qualification commit `da8c7f6e1837ddd896dfe83a7524812fee813487` adds measurement and
+installed-journey admission only; its native runtime, Runner, shared and database
+production files are byte-identical to that implementation.
+
+The independent Linux database campaign uses an eight-vCPU, 32 GiB disposable
+host, with PostgreSQL and its Node workload each limited to two CPUs and 1536 MiB.
+It preserves the existing 10/2/2 request/renewal/checkpoint pool caps per logical
+server. Steady and aligned-burst ABBA comparisons passed the existing absolute,
+correctness and 10% relative regression gates with 300 controllers and 10,000
+historical tasks. Neither changes snapshot/PRP transaction counts. The separate
+claim-capacity probe waits for natural lease expiry, invokes the production
+claim path for 300 synthetic executions, restores real encrypted checkpoints
+under empty controller directories, and measures current-owner checkpoint and
+renewal work. Synthetic provider references do not establish sandbox existence
+or authenticated provider reattachment.
+
+The closed, sanitized measurements and limits are recorded in
+`2026-10-30-native-controller-qualification-evidence.json`. The earlier failed,
+contended-host measurements remain in `2026-10-30-native-controller-load-evidence.json`.
+Lock measurements use calibrated PostgreSQL wait-start timestamps and sampled
+duration bounds. Observer traffic contributes to the measured database cost;
+short unsampled waits and full HTTP latency remain outside that measurement.
+
+Live qualification uses protected campaign
+[41108162900](https://github.com/paperclipai/paperclip/actions/runs/41108162900).
+Its initial catalog timeout occurred before provider allocation after Azure
+reported a dependency-cache egress limit. A scoped exact-entry cache invalidation
+and targeted retry forced a successful cold install. Only this campaign's
+ephemeral image publisher received a larger, delete-on-termination disk; image
+identity, source admission and provider authorization gates remain in force.
+
+Provider-free package builds pin first reader `59ff89a18fe1312b5f0bb908439bbdb1237b0252`
+and candidate `a0ef3ce7493c8bb76c4766ff007e8a390118d145` in runs
+[41108342354](https://github.com/paperclipai/paperclip/actions/runs/41108342354) and
+[41108352138](https://github.com/paperclipai/paperclip/actions/runs/41108352138).
+The source-process crash probe independently observed B waiting for A's live
+lease and claiming the same execution after natural expiry, with no lease edits.
+Its initial all-turns/single-provider-process assertion also crossed completion
+contract revision 1 to revision 2. The existing Runner attachment policy can
+retire a provider at a changed semantic contract while preserving its conversation;
+phase-specific observations distinguish that boundary from same-contract rollback.
+Earlier strict failures remain retained until that distinction is demonstrated.
+
+The final live, reader, installed-artifact, cleanup and CI outcomes are maintained
+in [PR #18343](https://github.com/paperclipai/paperclip/pull/18343), with their exact
+revisions and evidence limits. Campaign registration, successful artifact builds
+and this plan entry do not themselves certify that a live journey passed.

doc/plans/2026-10-30-native-controller-claim.patch

@@ -0,0 +1,170 @@
+--- scripts/benchmark-native-controller-recovery.ts
++++ scripts/benchmark-native-controller-recovery.ts
+@@ -14 +14 @@
+-const repoRoot = fileURLToPath(new URL("../", import.meta.url));
++const repoRoot = path.resolve(required("BENCH_CANDIDATE_ROOT"));
+@@ -241,0 +242,112 @@
++async function claimCapacityPhase(db: any, u: string, items: any[], monitor: any) {
++  const { claimNativeRestartRecoveries } = await load(repoRoot, "server/src/services/native-runtime/native-restart-recovery.ts");
++  const { loadNativeRunnerCheckpoint } = await load(repoRoot, "server/src/services/native-runtime/native-runner-checkpoint.ts");
++  const replacement = createDb(u, { maxConnections: 10, applicationName: "benchmark-claim-replacement" });
++  const successor = { bootId: randomUUID(), pid: process.pid, processStartedAt: new Date() };
++  const result: any = { schema: "paperclip.native-controller-claim-capacity/v1", ...provenance,
++    scope: "Production DB claim function against synthetic prepared-workspace references and real persisted snapshots/PRP records. Natural lease expiry; two server database clients. No provider attachment or external sandbox existence claim.",
++    errors: [], claims: [], reads: [], samples: [], sqlByClient: {}, renewedOwnerLosses: 0, restoredCheckpoints: 0,
++    retainedOldControllerLeaseMs: 60000, remoteReconnectGraceMs: 420000, checkpointPoolPerServer: 2 };
++  const clients = [
++    ["old_request", db.$client], ["old_liveness", livenessDb(db).$client], ["old_checkpoint", checkpointDb(db).$client],
++    ["new_request", replacement.$client], ["new_liveness", livenessDb(replacement).$client], ["new_checkpoint", checkpointDb(replacement).$client],
++  ];
++  for (const [name, client] of clients) { result.sqlByClient[name] = 0; client.options.debug = () => { result.sqlByClient[name]++; }; }
++  // Data is persisted by the ordinary snapshot port during the preconditioning phase.
++  const validity = await db.$client.unsafe("select count(*)::int as valid from heartbeat_runs r join native_run_finalizations f on f.run_id=r.id where r.status='running' and r.runner_profile_json->'nativeWorkspaceSync'->>'state'='prepared' and r.runner_profile_json->'sessionCheckpoint'->'identity'->>'runId'=r.id::text and exists(select 1 from native_runner_checkpoints c where c.run_id=r.id)");
++  assert.equal(validity[0].valid, 300, "All claim inputs need real snapshots/checkpoints and prepared references");
++  const before = pgResources(), started = performance.now(), requests = new Set<Promise<any>>(), renewals: any[] = [];
++  const observed = observeLockWaits(monitor);
++  let tick = 0, offered = 0, completed = 0;
++  const traffic = setInterval(() => {
++    for (let j = 0; j < 20; j++) {
++      const index = tick++, item = items[index % items.length], client = index % 2 ? replacement : db;
++      const at = performance.now(); offered++;
++      const task = loadAgentRunAuthentication(client, item).then(() => { completed++; result.reads.push(performance.now() - at); })
++        .catch((e: any) => result.errors.push({ source: "request", message: e.message })).finally(() => requests.delete(task));
++      requests.add(task);
++    }
++  }, 100); // 200 database authentication requests/second, split evenly between servers.
++  let sampling: Promise<any> | null = null;
++  const sample = setInterval(() => {
++    if (sampling) return;
++    sampling = monitor`select application_name,count(*)::int as connections,count(*) filter(where state='active')::int as active from pg_stat_activity where datname=current_database() and application_name!='benchmark-observer' group by application_name`
++      .then((rows: any) => result.samples.push({ elapsedMs: performance.now() - started, rows, postgresMemoryBytes: pgResources().memoryBytes }))
++      .catch((e: any) => result.errors.push({ source: "observer", message: e.message })).finally(() => { sampling = null; });
++  }, 1000);
++  try {
++    const [lease] = await db.$client.unsafe("select greatest(0,extract(epoch from(max(lease_expires_at)-clock_timestamp()))*1000)::float8 as remaining from native_run_finalizations where lease_owner is not null");
++    result.naturalExpiryWaitMs = Number(lease.remaining);
++    await sleep(result.naturalExpiryWaitMs + 100);
++    const [expired] = await db.$client.unsafe("select count(*)::int as expired from native_run_finalizations where lease_owner is not null and lease_expires_at<=clock_timestamp()");
++    assert.equal(expired.expired, 300, "Never rewrite or shorten a controller lease for the probe");
++    const claimStarted = performance.now();
++    const dispositions = await claimNativeRestartRecoveries({ db: replacement, controller: successor, restartKind: "hard",
++      runIds: items.map(item => item.runId), limit: 300,
++      onClaim: (claim: any, scope: any) => {
++        result.claims.push({ runId: claim.runId, kind: claim.kind, elapsedMs: performance.now() - claimStarted, generation: claim.controllerGeneration });
++        renewals.push(startNativeSessionExecutionLeaseRenewal({ db: replacement,
++          companyId: scope.companyId, issueId: scope.issueId, runId: claim.runId, leaseOwner: claim.leaseOwner,
++          attempt: claim.providerAttempt, controller: successor, remote: true,
++          onLost: async () => { result.renewedOwnerLosses++; } }));
++      },
++    });
++    result.claimDrainMs = performance.now() - claimStarted;
++    assert.equal(dispositions.length, 300);
++    assert(dispositions.every((claim: any) => claim.kind === "reattach_remote_runner"), "Every DB claim must independently validate its evidence");
++    // Restore persisted records into empty replacement directories, then persist a real new control and snapshot.
++    const work = [...dispositions];
++    await Promise.all(Array.from({ length: 4 }, async () => {
++      while (work.length) {
++        const claim: any = work.shift(), item = items.find(i => i.runId === claim.runId);
++        const checkpoint = await loadNativeRunnerCheckpoint(replacement, { companyId: item.companyId, issueId: item.issueId, runId: item.runId });
++        assert(checkpoint);
++        const directory = root + "/replacement-states/" + item.runId;
++        checkpoint.restoreSessionDirectory(directory);
++        const header = JSON.parse(checkpoint.records.get("header")!);
++        const core = new CandidateCore({ stateDirectory: directory, identity: header.state.identity,
++          expectedRunnerVersion: checkpoint.runnerVersion, expectedRunnerDigest: checkpoint.runnerDigest });
++        const binding = { companyId: item.companyId, issueId: item.issueId, runId: item.runId,
++          authorityRunId: item.runId, bootId: successor.bootId, generation: claim.controllerGeneration, leaseOwner: claim.leaseOwner };
++        core.setCheckpointPersistence(CandidateWriter(replacement, binding));
++        core.queueCommand("semantic_tool.result", { callId: "replacement-capacity", operationId: "fixture_read", result: { body: "x".repeat(4096) }, isError: false }, "replacement-control");
++        await core.flushCheckpoint();
++        const identity = item.snapshot.identity;
++        const port = new CandidatePort(replacement, { ...identity, sourceInstanceId: checkpoint.recoveryEvidence.runnerInstanceId,
++          controlPlaneSourceInstanceId: "replacement-controller", completionContractId: item.completionContractId, completionContractSha256: item.completionContractSha256 }, { nativeRecovery: {
++            compatibility: { configDigest: "sha256:" + "a".repeat(64), processConfigDigest: null }, owner: binding } });
++        await port.openRun({ identity, backendKind: "codex", sourceInstanceId: checkpoint.recoveryEvidence.runnerInstanceId });
++        await port.checkpointSession(item.snapshot);
++        result.restoredCheckpoints++;
++        await core.stop();
++      }
++    }));
++    result.claimAndRestoreDrainMs = performance.now() - claimStarted;
++    // Observe one natural 20-second renewal from the last replacement claim.
++    await sleep(Math.max(0, 21000 - (performance.now() - claimStarted - result.claimDrainMs)));
++    const [saved] = await replacement.$client.unsafe("select count(*)::int as owners,min(controller_generation)::int as min_generation,max(controller_generation)::int as max_generation,min(extract(epoch from(lease_expires_at-clock_timestamp()))*1000)::float8 as minimum_remaining_ms from native_run_finalizations where controller_boot_id=$1 and lease_owner is not null", [successor.bootId]);
++    result.postRenewalOwners = saved;
++    assert.equal(saved.owners, 300); assert.equal(saved.min_generation, 2); assert.equal(saved.max_generation, 2);
++    assert(saved.minimum_remaining_ms > 40000, "Replacement leases should have naturally renewed from the database clock");
++  } catch (e: any) { result.errors.push({ source: "claim-or-restore", message: e.message, stack: e.stack }); }
++  finally {
++    clearInterval(traffic); clearInterval(sample);
++    await Promise.allSettled([...requests]); await sampling;
++    await Promise.allSettled(renewals.map(r => r.stop()));
++    result.lockWait = await observed.stop();
++  }
++  result.elapsedMs = performance.now() - started;
++  result.requestsOffered = offered; result.requestsCompleted = completed; result.requestLatency = summary(result.reads); delete result.reads;
++  result.postgresCpuUsec = pgResources().cpuUsec - before.cpuUsec;
++  const totalSql = () => Object.values(result.sqlByClient).reduce((a: number, b: any) => a + Number(b), 0);
++  const idleStart = totalSql(); await sleep(1000); result.settledIdleSql = totalSql() - idleStart;
++  result.passed = result.errors.length === 0 && result.lockWait.errors.length === 0 && result.claims.length === 300
++    && result.restoredCheckpoints === 300 && completed === offered && result.renewedOwnerLosses === 0
++    && result.claimAndRestoreDrainMs < 420000 && result.requestLatency.p99 < 250 && result.settledIdleSql === 0;
++  await writeFile(root + "/claim-capacity.json", JSON.stringify(result, null, 2) + "\n");
++  console.log(JSON.stringify({ phase: "claim-capacity-complete", passed: result.passed, claims: result.claims.length,
++    claimDrainMs: result.claimDrainMs, claimAndRestoreDrainMs: result.claimAndRestoreDrainMs,
++    requests: result.requestLatency, errors: result.errors.slice(0, 3) }));
++  if (!result.passed) process.exitCode = 1;
++}
++
+@@ -429,0 +542 @@
++        runnerProfileJson: { nativeWorkspaceSync: { schema: "paperclip.native-workspace-sync/v3", state: "prepared", descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), finalHostSha256: null, workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: randomUUID(), remoteCwd: "/workspace", resourceDisposition: "destroy" } },
+@@ -544,0 +658,2 @@
++      completionContractId: contractId,
++      completionContractSha256: "fixture",
+@@ -905,0 +1021 @@
++  await claimCapacityPhase(db, u, items, monitor);
+@@ -917,6 +1033 @@
+-    const profile = [
+-      ["baseline", 300],
+-      ["candidate", 300],
+-      ["candidate", 300],
+-      ["baseline", 300],
+-    ] as const;
++    const profile = [["candidate", 300]] as const;
+@@ -953,35 +1064 @@
+-    const perOperation = (r: any, key: string) =>
+-      r[key] / (r.normalCompleted + r.normalWritesCompleted + r.controls + r.admissionsCompleted);
+-    const average = (values: number[]) => values.reduce((a, b) => a + b, 0) / values.length;
+-    const comparable =
+-      result.every((r) => r.correctnessPassed) &&
+-      result.every(
+-        (r) =>
+-          r.controls === result[0].controls &&
+-          r.normalCompleted === result[0].normalCompleted &&
+-          r.normalWritesCompleted === result[0].normalWritesCompleted,
+-      );
+-    const relative = {};
+-    for (const key of ["postgresCpuUsec", "walBytes", "requestP99"]) {
+-      const value = (r: any) => (key === "requestP99" ? r.normalReadSummary.p99 : perOperation(r, key));
+-      const baseline = average(result.filter((r) => r.variant === "baseline").map(value));
+-      const candidate = average(result.filter((r) => r.variant === "candidate").map(value));
+-      relative[key] = {
+-        baseline,
+-        candidate,
+-        increasePercent: comparable ? 100 * (candidate / baseline - 1) : null,
+-        passed: comparable ? candidate <= baseline * 1.1 : null,
+-      };
+-    }
+-    Object.assign(report, {
+-      relative,
+-      comparable,
+-      passed: result.every((r) => r.passed) && Object.values(relative).every((r: any) => r.passed),
+-      measurementLimits: [
+-        "Database request latency is not HTTP latency",
+-        "Pool queue includes client encoding; BEGIN response includes network time",
+-        "pg_stat_statements execution includes lock waits; lock episodes report sampled duration bounds, with short waits potentially missed",
+-        "Peak memory is sampled once a second",
+-        "Use an otherwise idle cluster and host; WAL and cgroup CPU are cluster-wide",
+-      ],
+-    });
++    Object.assign(report, { passed: result.every(r => r.passed), scope: "Preconditioning only for the separate claim-capacity.json report" });

doc/plans/2026-10-30-native-controller-overlap.patch

@@ -0,0 +1,76 @@
+--- scripts/benchmark-native-controller-recovery.ts
++++ scripts/benchmark-native-controller-recovery.ts
+@@ -0,0 +1 @@
++// Temporary qualification probe derived from frozen benchmark SHA256 abaa29d50eb95013836b54b8dacc94c72e080edf24744cc5d291cc5eb1ff2818.
+@@ -14 +15 @@
+-const repoRoot = fileURLToPath(new URL("../", import.meta.url));
++const repoRoot = path.resolve(required("BENCH_CANDIDATE_ROOT"));
+@@ -187 +188 @@
+-    const byPool = Object.fromEntries(["benchmark-application", "paperclip-controller-lease", "paperclip-runner-checkpoint"].map(pool => {
++    const byPool = Object.fromEntries(["benchmark-application", "benchmark-replacement-application", "paperclip-controller-lease", "paperclip-runner-checkpoint"].map(pool => {
+@@ -250,0 +252 @@
++  const replacementDb = createDb(u, { maxConnections: 10, applicationName: "benchmark-replacement-application" });
+@@ -256,0 +259 @@
++  const replacementOwner = { ...owner, bootId: randomUUID() };
+@@ -299,0 +303,3 @@
++    ["replacement_request", replacementDb.$client],
++    ["replacement_liveness", livenessDb(replacementDb).$client],
++    ["replacement_checkpoint", checkpointDb(replacementDb).$client],
+@@ -369,0 +376 @@
++  replacementDb.$client.options.debug = db.$client.options.debug;
+@@ -386,0 +394,2 @@
++  livenessDb(replacementDb).$client.options.debug = livenessDb(db).$client.options.debug;
++  checkpointDb(replacementDb).$client.options.debug = checkpointDb(db).$client.options.debug;
+@@ -389,0 +399,2 @@
++    const controllerDb = i % 2 ? replacementDb : db;
++    const controllerOwner = i % 2 ? replacementOwner : owner;
+@@ -443,3 +454,3 @@
+-        controllerBootId: owner.bootId,
+-        controllerPid: owner.pid,
+-        controllerProcessStartedAt: owner.processStartedAt,
++        controllerBootId: controllerOwner.bootId,
++        controllerPid: controllerOwner.pid,
++        controllerProcessStartedAt: controllerOwner.processStartedAt,
+@@ -467 +478 @@
+-      const writer = (variant === "candidate" ? CandidateWriter : BaselineWriter)(db, {
++      const writer = (variant === "candidate" ? CandidateWriter : BaselineWriter)(controllerDb, {
+@@ -472 +483 @@
+-        bootId: owner.bootId,
++        bootId: controllerOwner.bootId,
+@@ -511 +522 @@
+-      db,
++      controllerDb,
+@@ -531 +542 @@
+-                bootId: owner.bootId,
++                bootId: controllerOwner.bootId,
+@@ -544,0 +556,2 @@
++      db: controllerDb,
++      owner: controllerOwner,
+@@ -592 +605 @@
+-  for (let i = 0; i < 100; i++) await loadAgentRunAuthentication(db, items[i % n]);
++  for (let i = 0; i < 100; i++) await loadAgentRunAuthentication(items[i % n].db, items[i % n]);
+@@ -639 +652 @@
+-    const input = { db, ...item, attempt: 1, controller: owner };
++    const input = { ...item, attempt: 1, controller: item.owner };
+@@ -664 +677 @@
+-              const rows = await loadAgentRunAuthentication(db, item);
++              const rows = await loadAgentRunAuthentication(item.db, item);
+@@ -700 +713 @@
+-            await db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, item.issueId));
++            await item.db.update(issues).set({ updatedAt: new Date() }).where(eq(issues.id, item.issueId));
+@@ -713 +726 @@
+-            await db.transaction(async (tx: any) => {
++            await (metrics.admissionsOffered % 2 ? replacementDb : db).transaction(async (tx: any) => {
+@@ -917,6 +930 @@
+-    const profile = [
+-      ["baseline", 300],
+-      ["candidate", 300],
+-      ["candidate", 300],
+-      ["baseline", 300],
+-    ] as const;
++    const profile = [["baseline", 300], ["candidate", 300], ["candidate", 300], ["baseline", 300]] as const;
+@@ -929 +937 @@
+-        "Paired baseline/candidate, including real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.",
++        "Two logical servers in one client process, 150 controllers each, separate request/liveness/checkpoint pools at unchanged caps; same aggregate aligned-burst workload. Matched baseline/candidate ABBA pool-overlap component qualification; no takeover latency or provider reattachment claim. Includes real PaperclipControlPlanePort snapshot persistence on every control checkpoint. Database/control-plane component stress; real production checkpoint writer and natural remote lease timers. Same synthetic DB request load. Not full HTTP, harness or hosted capacity qualification.",
+@@ -930,0 +939 @@
++        logicalServers: 2,