Qualify retained native sessions after a second controller replacement

Keep the dedicated Daytona outage at five minutes and require a new useful turn from the retained provider session after completion and cache retirement.

Co-Authored-By: Paperclip <noreply@paperclip.ing>

doc/testing-native-remote-continuity.md

@@ -297,3 +297,59 @@ This is protocol and persistence evidence. The separate
 `everyday-workflows.runner-codex.daytona.recover-controller` Product E2E cell
 qualifies the actual Daytona/provider path, including completion and workspace
 finalization. Neither test substitutes for the other's failure boundary.
+
+The explicit `daytona-restart-continuity.runner-codex.daytona.recover-controller`
+cell fixes the outage at 300 seconds and adds a second replacement after
+completion. Its new turn executes the saved project through normal admission.
+The oracle compares hashed native/provider session identities and requires a new
+completed tool receipt and correct answer. An installed A/B/A artifact manifest
+uses the last replacement for this retained-session continuation/rollback check.
+The general everyday suite retains its 125-second default. These intervals are
+fault tests, not permitted HTTP deployment downtime. Both fixtures retain the
+workspace and remote sandbox; they do not prove whole-webserver disk loss.
+
+## Database cost qualification
+
+`scripts/benchmark-native-controller-recovery.ts` compares a pinned source
+baseline with the candidate's real PRP writer, control-plane snapshot port and
+lease renewal. It uses identical fixtures with 300 controllers and 10,000 prior
+tasks, in baseline/candidate/candidate/baseline order. `steady` distributes control
+checkpoints; `burst` aligns them every five seconds. `pilot` checks instrumentation
+with ten controllers. No provider is called.
+
+Run on an otherwise idle disposable Linux Postgres cluster with
+`pg_stat_statements` preloaded and its extension available. Give Postgres and the
+Node client two CPUs and 1536 MiB each. Set these explicit inputs:
+
+```sh
+BENCH_DISPOSABLE_POSTGRES=1 \
+BENCH_DATABASE_URL=postgres://postgres:fixture-password@127.0.0.1:5432/postgres \
+BENCH_BASELINE_ROOT=/absolute/path/to/pinned-baseline \
+BENCH_REPORT_ROOT=/absolute/path/to/new-report-directory \
+BENCH_POSTGRES_CGROUP=/absolute/path/to/postgres-cgroup \
+node --import ./cli/node_modules/tsx/dist/loader.mjs \
+  scripts/benchmark-native-controller-recovery.ts steady
+```
+
+The baseline needs the same installed dependency versions and built workspace
+packages. It supplies the baseline writer, snapshot port and PRP controller;
+the request fixtures, renewal path and pool policy are common controls. When
+running an exported tree without Git metadata, supply its exact
+`BENCH_BASELINE_REVISION`, `BENCH_CANDIDATE_REVISION` and
+`BENCH_CANDIDATE_DIFF`. Record the export source and verify its bytes separately.
+Use a fresh report directory for every attempt. Only new random databases are
+created; successful trial databases are dropped, and the owned template remains
+for inspection until the disposable cluster is removed.
+
+The report counts SQL across all three pools, actual snapshot/PRP transactions,
+changed records/bytes, WAL, database CPU, sampled memory, pool admission,
+transaction latency, lock-waiter samples and database request latency. Pool
+admission ends at connection assignment/BEGIN encoding; BEGIN response time is
+separate. Postgres execution time includes lock waits. This does **not** yet
+provide a lock-wait latency distribution or HTTP latency. Report those limits;
+do not call this whole-application load qualification.
+
+Correctness and absolute gates must pass before interpreting relative costs.
+Dropped operations invalidate paired comparisons. The report retains failures
+and refuses an overall pass if p99, drain, throughput, memory, CPU or the 10%
+relative regression gate fails. Do not change pool limits to conceal a failure.

tests/runner-e2e/README.md

@@ -11,12 +11,17 @@ qualification.
 The explicit-only `daytona-restart-continuity` suite runs the existing
 `recover-controller` journey with the recorded Codex and Claude staging models
 on fresh Daytona fixtures. After a completed task tool in an active revision
-turn, it keeps the owned development controller offline for 125 seconds, beyond
-the controller's two-minute reconnect budget. The existing independent oracle
+turn, it keeps the owned development controller offline for 300 seconds. This
+exceeds the 60-second ownership lease and remains within the Runner's 420-second
+reconnect grace. The existing independent oracle
 requires the original process, native session, sandbox lease and receipt prefix,
 one completion and successful workspace finalization, plus verified downloaded
-work. It permits one attempt per cell and retains failed evidence. This is
+work. After completion, it replaces the controller again with its native cache
+unavailable and asks a new turn to execute the saved project. The continuation
+oracle requires the same retained provider session and a new useful result.
+It permits one attempt per cell and retains failed evidence. This is
 development API-key acceptance, distinct from staging subscription recovery.
+The five-minute outage is a resilience test, not an HTTP downtime allowance.
 
 The explicit-only [GitHub review bot journey](GITHUB-REVIEW-JOURNEY.md) tests
 attended bot setup and an automatic PR review against an attached deployment.

tests/runner-e2e/catalog.ts

@@ -1160,12 +1160,12 @@ const repositoryModelProfiles = [
 export const runnerSuites: readonly RunnerSuiteFixture[] = [
   {
     id: "daytona-restart-continuity", label: "Daytona controller outage continuity", manualOnly: true,
-    description: "Keep the original active native runner, task tools and saved files through more than two minutes without a controller, then commit the result without replay.",
+    description: "Keep the original active native runner, task tools and saved files through five minutes without a controller, then continue the completed session after another replacement.",
     groups: [], profiles: repositoryModelProfiles.map(({ native }) => productionStoryProfile(native)),
     environments: runnerEnvironments.filter(environment => environment.id === "daytona"),
     tasks: everydayTasks.filter(task => task.id === "recover-controller")
-      .map(task => ({ ...task, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2,
-    definitionMetadata: { version: 2, scheduling: "explicit-only", controllerDowntimeMs: 125_000,
+      .map(task => ({ ...task, expectedRunCount: task.expectedRunCount + 1, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2,
+    definitionMetadata: { version: 3, scheduling: "explicit-only", controllerDowntimeMs: 300_000, retainedControllerContinuation: true,
       maximumAttemptsPerCell: 1, budgetMonthlyCents: 1000, instructions: "production", grading: "same-active-process-lease-native-prefix-and-workspace-receipt",
       authentication: "api-key", stagingAuthentication: "subscription", stagingQualification: false },
   },

tests/runner-e2e/controller-outage.test.ts

@@ -25,7 +25,7 @@ describe("controller outage qualification", () => {
     expect(executions).toHaveLength(2);
     for (const execution of executions) {
       const outage = controllerOutageForExecution(execution, {});
-      expect(outage).toEqual({ signal: "SIGKILL", holdMs: 125_000 });
+      expect(outage).toEqual({ signal: "SIGKILL", holdMs: 300_000 });
       expect(controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "300000" }))
         .toEqual({ signal: "SIGKILL", holdMs: 300_000 });
       expect(() => controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "1000" }))

tests/runner-e2e/everyday-flow.ts

@@ -26,6 +26,7 @@ import { hasPersistedSource, isSavedSourceCheckpoint } from "./everyday-interrup
 import { collectRunEvents } from "./run-observations.js";
 import { gradeRemoteRestart, observeRemoteRestartBoundary, projectRemoteRestartEvents, projectRemoteRestartRun, waitForRemoteRestartBoundary, type RemoteRestartEvidence } from "./everyday-remote-restart.js";
 import { controllerOutageForExecution, type ControllerOutageReceipt } from "./controller-outage.js";
+import { gradeRetainedControllerContinuation, retainedControllerRun } from "./retained-controller-continuation.js";
 import { setupAggregatorFixture } from "./aggregator-fixture.js";
 import { gradeProviderChoice, gradeProviderOutcome, requireProviderAccessCard } from "./connection-routing-evidence.js";
 import { setupConnectionReview } from "./connection-reviews.js";
@@ -1440,13 +1441,33 @@ export async function runEverydayFlow(input: Input) {
           artifactGradeModeForPhase("reviewed-delivery"),
           "reviewed-delivery",
         );
-    } else if (caseId === "recover-controller")
+    } else if (caseId === "recover-controller") {
       await download(
         parent!.id,
         artifactGradeModeForPhase("recovered-delivery"),
         "recovered-delivery",
       );
-    else if (caseId === "stop-redirect")
+      if (execution.suite.definitionMetadata?.retainedControllerContinuation === true) {
+        const prior = ev.runs.filter(run => run.status === "succeeded" && run.agentId === fixtures.agent.id)
+          .sort((a, b) => Date.parse(b.finishedAt ?? "") - Date.parse(a.finishedAt ?? ""))[0];
+        if (!prior) throw new Error("Retained continuation requires a completed source run");
+        const before = retainedControllerRun(prior), priorIds = new Set(ev.runs.map(run => run.id));
+        const outage = await input.restart();
+        await openParent();
+        const requestedAt = new Date().toISOString(), marker = `CONTINUED_${nonce}`;
+        await reply(`Run the existing delivered slugify.py on "  Queue--Ready!!  " with --separator _ and --max-length 7. Report its output after the marker ${marker}. Use the saved project; do not rebuild it.`);
+        await settled(`${marker} queue_r`);
+        const after = ev.runs.filter(run => !priorIds.has(run.id) && run.status === "succeeded" && run.agentId === fixtures.agent.id)
+          .sort((a, b) => Date.parse(b.finishedAt ?? "") - Date.parse(a.finishedAt ?? ""))[0];
+        if (!after) throw new Error("Retained continuation produced no new successful run");
+        const events = await collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${after.id}/events?afterSeq=${afterSeq}&limit=${limit}`));
+        const evidence = { before, after: retainedControllerRun(after), outage: outage || undefined, requestedAt,
+          completedTool: events.some(row => row.payload?.prpEvent?.eventType === "tool.execution.completed" && row.payload.prpEvent.payload?.status === "completed"),
+          usefulReply: settledAgentReply?.authorAgentId === fixtures.agent.id && String(settledAgentReply.body ?? "").includes(`${marker} queue_r`) };
+        ev.checks.push(...gradeRetainedControllerContinuation(evidence));
+        await input.evidence("retained-controller-continuation.json", evidence);
+      }
+    } else if (caseId === "stop-redirect")
       check(
         "new-direction-delivered",
         settledAgentReply?.authorAgentId === fixtures.agent.id &&

tests/runner-e2e/retained-controller-continuation.test.ts

@@ -0,0 +1,31 @@
+import { describe, expect, it } from "vitest";
+import { gradeRetainedControllerContinuation, retainedControllerRun } from "./retained-controller-continuation.js";
+
+function fixture() {
+  const before = retainedControllerRun({ id: "a", companyId: "company", agentId: "agent", nativeIssueId: "task",
+    runtimeMode: "native", status: "succeeded", nativeSessionId: "private-native-session", sessionIdAfter: "private-provider-session",
+    finishedAt: "2026-10-30T00:00:00Z" });
+  return { before, after: { ...before, runId: "b", finishedAt: "2026-10-30T00:01:00Z" },
+    outage: { schema: "paperclip.e2e.controller-outage/v1" as const, requestId: "retained", signal: "SIGKILL" as const,
+      holdMs: 1000, startedAt: "2026-10-30T00:00:01Z", previousPid: 1, replacementPid: 2,
+      stoppedAfterMs: 0, replacementStartedAfterMs: 1000, readyAfterMs: 2000, cacheDiscarded: true },
+    requestedAt: "2026-10-30T00:00:04Z", completedTool: true, usefulReply: true };
+}
+
+describe("retained controller continuation evidence", () => {
+  it("accepts a new useful turn after a completed source and cache retirement", () => {
+    expect(gradeRetainedControllerContinuation(fixture())[0]?.passed).toBe(true);
+    expect(JSON.stringify(fixture())).not.toContain("private-provider-session");
+  });
+  it.each(["provider", "native", "same-run", "before-completion", "no-cache-loss", "no-tool", "no-answer"])("rejects %s evidence", fault => {
+    const input = fixture();
+    if (fault === "provider") input.after.providerSessionSha256 = "different";
+    if (fault === "native") input.after.nativeSessionSha256 = null;
+    if (fault === "same-run") input.after.runId = input.before.runId;
+    if (fault === "before-completion") input.before.finishedAt = "2026-10-30T00:00:10Z";
+    if (fault === "no-cache-loss") input.outage.cacheDiscarded = false;
+    if (fault === "no-tool") input.completedTool = false;
+    if (fault === "no-answer") input.usefulReply = false;
+    expect(gradeRetainedControllerContinuation(input)[0]?.passed).toBe(false);
+  });
+});

tests/runner-e2e/retained-controller-continuation.ts

@@ -0,0 +1,36 @@
+import { createHash } from "node:crypto";
+import type { StoryCheck, StoryRun } from "./everyday-observations.js";
+import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js";
+
+const hash = (value: string | null | undefined) => value ? createHash("sha256").update(value).digest("hex") : null;
+
+/** Public evidence keeps provider/session identifiers hashed. */
+export function retainedControllerRun(run: StoryRun) {
+  return { runId: run.id, companyId: run.companyId, agentId: run.agentId, issueId: run.nativeIssueId,
+    status: run.status, runtimeMode: run.runtimeMode, finishedAt: run.finishedAt,
+    nativeSessionSha256: hash(run.nativeSessionId), providerSessionSha256: hash(run.sessionIdAfter) };
+}
+
+export function gradeRetainedControllerContinuation(input: {
+  before: ReturnType<typeof retainedControllerRun>;
+  after: ReturnType<typeof retainedControllerRun>;
+  outage: ControllerOutageReceipt | undefined;
+  requestedAt: string;
+  completedTool: boolean;
+  usefulReply: boolean;
+}): StoryCheck[] {
+  const { before, after, outage } = input;
+  const passed = validControllerOutageReceipt(outage)
+    && before.status === "succeeded" && after.status === "succeeded"
+    && before.runtimeMode === "native" && after.runtimeMode === "native"
+    && before.runId !== after.runId && before.companyId === after.companyId
+    && before.agentId === after.agentId && before.issueId === after.issueId
+    && Boolean(before.nativeSessionSha256) && before.nativeSessionSha256 === after.nativeSessionSha256
+    && Boolean(before.providerSessionSha256) && before.providerSessionSha256 === after.providerSessionSha256
+    && Date.parse(before.finishedAt ?? "") <= Date.parse(outage?.startedAt ?? "")
+    && Date.parse(input.requestedAt) >= Date.parse(outage?.startedAt ?? "") + (outage?.readyAfterMs ?? Infinity)
+    && Date.parse(after.finishedAt ?? "") >= Date.parse(input.requestedAt)
+    && input.completedTool && input.usefulReply;
+  return [{ id: "retained-controller-continuation", passed,
+    detail: "After completion and another empty-cache replacement, a new native turn must use the same retained provider session and execute the saved project for a useful reply." }];
+}

tests/runner-e2e/runner.spec.ts

@@ -1221,12 +1221,15 @@ for (const execution of executions) {
         const story = await runEverydayFlow({
           page, api, fixtures, execution, nonce, workspacePath, privateDir,
           deadlineAt: startedAtMs + deadlineMs,
-          restartCount: upgradeManifest ? upgradeManifest.sequence.length - 1 : 1,
+          restartCount: upgradeManifest ? upgradeManifest.sequence.length -
+            (execution.suite.definitionMetadata?.retainedControllerContinuation === true ? 2 : 1) : 1,
           restart: async (recoveryBinding, controllerDowntimeMs) => {
             let receipt: ControllerOutageReceipt | undefined;
             restartIndex += 1;
             await restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}-${restartIndex}`, deadlineAt: startedAtMs + deadlineMs,
-              discardControllerCache: controllerOutage !== undefined, controllerOutage, controllerDowntimeMs,
+              discardControllerCache: controllerOutage !== undefined,
+              controllerOutage: controllerOutage && !recoveryBinding ? { ...controllerOutage, holdMs: 1000 } : controllerOutage,
+              controllerDowntimeMs,
               ...(controllerOutage ? { recoveryBinding } : {}),
               onControllerOutage: async value => {
                 receipt = value;