From ea76687ab73bad8c0b2706d7f06deec40427b75d Mon Sep 17 00:00:00 2001 From: Dotta Date: Fri, 30 Oct 2026 11:48:46 -0500 Subject: [PATCH] Qualify retained native sessions after a second controller replacement Keep the dedicated Daytona outage at five minutes and require a new useful turn from the retained provider session after completion and cache retirement. Co-Authored-By: Paperclip --- doc/testing-native-remote-continuity.md | 56 +++++++++++++++++++ tests/runner-e2e/README.md | 11 +++- tests/runner-e2e/catalog.ts | 6 +- tests/runner-e2e/controller-outage.test.ts | 2 +- tests/runner-e2e/everyday-flow.ts | 25 ++++++++- .../retained-controller-continuation.test.ts | 31 ++++++++++ .../retained-controller-continuation.ts | 36 ++++++++++++ tests/runner-e2e/runner.spec.ts | 7 ++- 8 files changed, 163 insertions(+), 11 deletions(-) create mode 100644 tests/runner-e2e/retained-controller-continuation.test.ts create mode 100644 tests/runner-e2e/retained-controller-continuation.ts diff --git a/doc/testing-native-remote-continuity.md b/doc/testing-native-remote-continuity.md index 43c024faa4f..056421adcd3 100644 --- a/doc/testing-native-remote-continuity.md +++ b/doc/testing-native-remote-continuity.md @@ -297,3 +297,59 @@ This is protocol and persistence evidence. The separate `everyday-workflows.runner-codex.daytona.recover-controller` Product E2E cell qualifies the actual Daytona/provider path, including completion and workspace finalization. Neither test substitutes for the other's failure boundary. + +The explicit `daytona-restart-continuity.runner-codex.daytona.recover-controller` +cell fixes the outage at 300 seconds and adds a second replacement after +completion. Its new turn executes the saved project through normal admission. +The oracle compares hashed native/provider session identities and requires a new +completed tool receipt and correct answer. An installed A/B/A artifact manifest +uses the last replacement for this retained-session continuation/rollback check. +The general everyday suite retains its 125-second default. These intervals are +fault tests, not permitted HTTP deployment downtime. Both fixtures retain the +workspace and remote sandbox; they do not prove whole-webserver disk loss. + +## Database cost qualification + +`scripts/benchmark-native-controller-recovery.ts` compares a pinned source +baseline with the candidate's real PRP writer, control-plane snapshot port and +lease renewal. It uses identical fixtures with 300 controllers and 10,000 prior +tasks, in baseline/candidate/candidate/baseline order. `steady` distributes control +checkpoints; `burst` aligns them every five seconds. `pilot` checks instrumentation +with ten controllers. No provider is called. + +Run on an otherwise idle disposable Linux Postgres cluster with +`pg_stat_statements` preloaded and its extension available. Give Postgres and the +Node client two CPUs and 1536 MiB each. Set these explicit inputs: + +```sh +BENCH_DISPOSABLE_POSTGRES=1 \ +BENCH_DATABASE_URL=postgres://postgres:fixture-password@127.0.0.1:5432/postgres \ +BENCH_BASELINE_ROOT=/absolute/path/to/pinned-baseline \ +BENCH_REPORT_ROOT=/absolute/path/to/new-report-directory \ +BENCH_POSTGRES_CGROUP=/absolute/path/to/postgres-cgroup \ +node --import ./cli/node_modules/tsx/dist/loader.mjs \ + scripts/benchmark-native-controller-recovery.ts steady +``` + +The baseline needs the same installed dependency versions and built workspace +packages. It supplies the baseline writer, snapshot port and PRP controller; +the request fixtures, renewal path and pool policy are common controls. When +running an exported tree without Git metadata, supply its exact +`BENCH_BASELINE_REVISION`, `BENCH_CANDIDATE_REVISION` and +`BENCH_CANDIDATE_DIFF`. Record the export source and verify its bytes separately. +Use a fresh report directory for every attempt. Only new random databases are +created; successful trial databases are dropped, and the owned template remains +for inspection until the disposable cluster is removed. + +The report counts SQL across all three pools, actual snapshot/PRP transactions, +changed records/bytes, WAL, database CPU, sampled memory, pool admission, +transaction latency, lock-waiter samples and database request latency. Pool +admission ends at connection assignment/BEGIN encoding; BEGIN response time is +separate. Postgres execution time includes lock waits. This does **not** yet +provide a lock-wait latency distribution or HTTP latency. Report those limits; +do not call this whole-application load qualification. + +Correctness and absolute gates must pass before interpreting relative costs. +Dropped operations invalidate paired comparisons. The report retains failures +and refuses an overall pass if p99, drain, throughput, memory, CPU or the 10% +relative regression gate fails. Do not change pool limits to conceal a failure. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 61a87d5845c..52378d37d5c 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -11,12 +11,17 @@ qualification. The explicit-only `daytona-restart-continuity` suite runs the existing `recover-controller` journey with the recorded Codex and Claude staging models on fresh Daytona fixtures. After a completed task tool in an active revision -turn, it keeps the owned development controller offline for 125 seconds, beyond -the controller's two-minute reconnect budget. The existing independent oracle +turn, it keeps the owned development controller offline for 300 seconds. This +exceeds the 60-second ownership lease and remains within the Runner's 420-second +reconnect grace. The existing independent oracle requires the original process, native session, sandbox lease and receipt prefix, one completion and successful workspace finalization, plus verified downloaded -work. It permits one attempt per cell and retains failed evidence. This is +work. After completion, it replaces the controller again with its native cache +unavailable and asks a new turn to execute the saved project. The continuation +oracle requires the same retained provider session and a new useful result. +It permits one attempt per cell and retains failed evidence. This is development API-key acceptance, distinct from staging subscription recovery. +The five-minute outage is a resilience test, not an HTTP downtime allowance. The explicit-only [GitHub review bot journey](GITHUB-REVIEW-JOURNEY.md) tests attended bot setup and an automatic PR review against an attached deployment. diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index c419d76d1bf..247bf709e3e 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1160,12 +1160,12 @@ const repositoryModelProfiles = [ export const runnerSuites: readonly RunnerSuiteFixture[] = [ { id: "daytona-restart-continuity", label: "Daytona controller outage continuity", manualOnly: true, - description: "Keep the original active native runner, task tools and saved files through more than two minutes without a controller, then commit the result without replay.", + description: "Keep the original active native runner, task tools and saved files through five minutes without a controller, then continue the completed session after another replacement.", groups: [], profiles: repositoryModelProfiles.map(({ native }) => productionStoryProfile(native)), environments: runnerEnvironments.filter(environment => environment.id === "daytona"), tasks: everydayTasks.filter(task => task.id === "recover-controller") - .map(task => ({ ...task, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2, - definitionMetadata: { version: 2, scheduling: "explicit-only", controllerDowntimeMs: 125_000, + .map(task => ({ ...task, expectedRunCount: task.expectedRunCount + 1, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2, + definitionMetadata: { version: 3, scheduling: "explicit-only", controllerDowntimeMs: 300_000, retainedControllerContinuation: true, maximumAttemptsPerCell: 1, budgetMonthlyCents: 1000, instructions: "production", grading: "same-active-process-lease-native-prefix-and-workspace-receipt", authentication: "api-key", stagingAuthentication: "subscription", stagingQualification: false }, }, diff --git a/tests/runner-e2e/controller-outage.test.ts b/tests/runner-e2e/controller-outage.test.ts index d8bb79b9743..edb1c9a5718 100644 --- a/tests/runner-e2e/controller-outage.test.ts +++ b/tests/runner-e2e/controller-outage.test.ts @@ -25,7 +25,7 @@ describe("controller outage qualification", () => { expect(executions).toHaveLength(2); for (const execution of executions) { const outage = controllerOutageForExecution(execution, {}); - expect(outage).toEqual({ signal: "SIGKILL", holdMs: 125_000 }); + expect(outage).toEqual({ signal: "SIGKILL", holdMs: 300_000 }); expect(controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "300000" })) .toEqual({ signal: "SIGKILL", holdMs: 300_000 }); expect(() => controllerOutageForExecution(execution, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "1000" })) diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index 9f24264026f..96ddde1277f 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -26,6 +26,7 @@ import { hasPersistedSource, isSavedSourceCheckpoint } from "./everyday-interrup import { collectRunEvents } from "./run-observations.js"; import { gradeRemoteRestart, observeRemoteRestartBoundary, projectRemoteRestartEvents, projectRemoteRestartRun, waitForRemoteRestartBoundary, type RemoteRestartEvidence } from "./everyday-remote-restart.js"; import { controllerOutageForExecution, type ControllerOutageReceipt } from "./controller-outage.js"; +import { gradeRetainedControllerContinuation, retainedControllerRun } from "./retained-controller-continuation.js"; import { setupAggregatorFixture } from "./aggregator-fixture.js"; import { gradeProviderChoice, gradeProviderOutcome, requireProviderAccessCard } from "./connection-routing-evidence.js"; import { setupConnectionReview } from "./connection-reviews.js"; @@ -1440,13 +1441,33 @@ export async function runEverydayFlow(input: Input) { artifactGradeModeForPhase("reviewed-delivery"), "reviewed-delivery", ); - } else if (caseId === "recover-controller") + } else if (caseId === "recover-controller") { await download( parent!.id, artifactGradeModeForPhase("recovered-delivery"), "recovered-delivery", ); - else if (caseId === "stop-redirect") + if (execution.suite.definitionMetadata?.retainedControllerContinuation === true) { + const prior = ev.runs.filter(run => run.status === "succeeded" && run.agentId === fixtures.agent.id) + .sort((a, b) => Date.parse(b.finishedAt ?? "") - Date.parse(a.finishedAt ?? ""))[0]; + if (!prior) throw new Error("Retained continuation requires a completed source run"); + const before = retainedControllerRun(prior), priorIds = new Set(ev.runs.map(run => run.id)); + const outage = await input.restart(); + await openParent(); + const requestedAt = new Date().toISOString(), marker = `CONTINUED_${nonce}`; + await reply(`Run the existing delivered slugify.py on " Queue--Ready!! " with --separator _ and --max-length 7. Report its output after the marker ${marker}. Use the saved project; do not rebuild it.`); + await settled(`${marker} queue_r`); + const after = ev.runs.filter(run => !priorIds.has(run.id) && run.status === "succeeded" && run.agentId === fixtures.agent.id) + .sort((a, b) => Date.parse(b.finishedAt ?? "") - Date.parse(a.finishedAt ?? ""))[0]; + if (!after) throw new Error("Retained continuation produced no new successful run"); + const events = await collectRunEvents((afterSeq, limit) => api.get(`/api/heartbeat-runs/${after.id}/events?afterSeq=${afterSeq}&limit=${limit}`)); + const evidence = { before, after: retainedControllerRun(after), outage: outage || undefined, requestedAt, + completedTool: events.some(row => row.payload?.prpEvent?.eventType === "tool.execution.completed" && row.payload.prpEvent.payload?.status === "completed"), + usefulReply: settledAgentReply?.authorAgentId === fixtures.agent.id && String(settledAgentReply.body ?? "").includes(`${marker} queue_r`) }; + ev.checks.push(...gradeRetainedControllerContinuation(evidence)); + await input.evidence("retained-controller-continuation.json", evidence); + } + } else if (caseId === "stop-redirect") check( "new-direction-delivered", settledAgentReply?.authorAgentId === fixtures.agent.id && diff --git a/tests/runner-e2e/retained-controller-continuation.test.ts b/tests/runner-e2e/retained-controller-continuation.test.ts new file mode 100644 index 00000000000..76beee4cd42 --- /dev/null +++ b/tests/runner-e2e/retained-controller-continuation.test.ts @@ -0,0 +1,31 @@ +import { describe, expect, it } from "vitest"; +import { gradeRetainedControllerContinuation, retainedControllerRun } from "./retained-controller-continuation.js"; + +function fixture() { + const before = retainedControllerRun({ id: "a", companyId: "company", agentId: "agent", nativeIssueId: "task", + runtimeMode: "native", status: "succeeded", nativeSessionId: "private-native-session", sessionIdAfter: "private-provider-session", + finishedAt: "2026-10-30T00:00:00Z" }); + return { before, after: { ...before, runId: "b", finishedAt: "2026-10-30T00:01:00Z" }, + outage: { schema: "paperclip.e2e.controller-outage/v1" as const, requestId: "retained", signal: "SIGKILL" as const, + holdMs: 1000, startedAt: "2026-10-30T00:00:01Z", previousPid: 1, replacementPid: 2, + stoppedAfterMs: 0, replacementStartedAfterMs: 1000, readyAfterMs: 2000, cacheDiscarded: true }, + requestedAt: "2026-10-30T00:00:04Z", completedTool: true, usefulReply: true }; +} + +describe("retained controller continuation evidence", () => { + it("accepts a new useful turn after a completed source and cache retirement", () => { + expect(gradeRetainedControllerContinuation(fixture())[0]?.passed).toBe(true); + expect(JSON.stringify(fixture())).not.toContain("private-provider-session"); + }); + it.each(["provider", "native", "same-run", "before-completion", "no-cache-loss", "no-tool", "no-answer"])("rejects %s evidence", fault => { + const input = fixture(); + if (fault === "provider") input.after.providerSessionSha256 = "different"; + if (fault === "native") input.after.nativeSessionSha256 = null; + if (fault === "same-run") input.after.runId = input.before.runId; + if (fault === "before-completion") input.before.finishedAt = "2026-10-30T00:00:10Z"; + if (fault === "no-cache-loss") input.outage.cacheDiscarded = false; + if (fault === "no-tool") input.completedTool = false; + if (fault === "no-answer") input.usefulReply = false; + expect(gradeRetainedControllerContinuation(input)[0]?.passed).toBe(false); + }); +}); diff --git a/tests/runner-e2e/retained-controller-continuation.ts b/tests/runner-e2e/retained-controller-continuation.ts new file mode 100644 index 00000000000..0601c5815a3 --- /dev/null +++ b/tests/runner-e2e/retained-controller-continuation.ts @@ -0,0 +1,36 @@ +import { createHash } from "node:crypto"; +import type { StoryCheck, StoryRun } from "./everyday-observations.js"; +import { validControllerOutageReceipt, type ControllerOutageReceipt } from "./controller-outage.js"; + +const hash = (value: string | null | undefined) => value ? createHash("sha256").update(value).digest("hex") : null; + +/** Public evidence keeps provider/session identifiers hashed. */ +export function retainedControllerRun(run: StoryRun) { + return { runId: run.id, companyId: run.companyId, agentId: run.agentId, issueId: run.nativeIssueId, + status: run.status, runtimeMode: run.runtimeMode, finishedAt: run.finishedAt, + nativeSessionSha256: hash(run.nativeSessionId), providerSessionSha256: hash(run.sessionIdAfter) }; +} + +export function gradeRetainedControllerContinuation(input: { + before: ReturnType; + after: ReturnType; + outage: ControllerOutageReceipt | undefined; + requestedAt: string; + completedTool: boolean; + usefulReply: boolean; +}): StoryCheck[] { + const { before, after, outage } = input; + const passed = validControllerOutageReceipt(outage) + && before.status === "succeeded" && after.status === "succeeded" + && before.runtimeMode === "native" && after.runtimeMode === "native" + && before.runId !== after.runId && before.companyId === after.companyId + && before.agentId === after.agentId && before.issueId === after.issueId + && Boolean(before.nativeSessionSha256) && before.nativeSessionSha256 === after.nativeSessionSha256 + && Boolean(before.providerSessionSha256) && before.providerSessionSha256 === after.providerSessionSha256 + && Date.parse(before.finishedAt ?? "") <= Date.parse(outage?.startedAt ?? "") + && Date.parse(input.requestedAt) >= Date.parse(outage?.startedAt ?? "") + (outage?.readyAfterMs ?? Infinity) + && Date.parse(after.finishedAt ?? "") >= Date.parse(input.requestedAt) + && input.completedTool && input.usefulReply; + return [{ id: "retained-controller-continuation", passed, + detail: "After completion and another empty-cache replacement, a new native turn must use the same retained provider session and execute the saved project for a useful reply." }]; +} diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 080188c2cf7..a18da0c05e3 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1221,12 +1221,15 @@ for (const execution of executions) { const story = await runEverydayFlow({ page, api, fixtures, execution, nonce, workspacePath, privateDir, deadlineAt: startedAtMs + deadlineMs, - restartCount: upgradeManifest ? upgradeManifest.sequence.length - 1 : 1, + restartCount: upgradeManifest ? upgradeManifest.sequence.length - + (execution.suite.definitionMetadata?.retainedControllerContinuation === true ? 2 : 1) : 1, restart: async (recoveryBinding, controllerDowntimeMs) => { let receipt: ControllerOutageReceipt | undefined; restartIndex += 1; await restartIsolatedPaperclipServer({ api, requestId: `story-${nonce}-${restartIndex}`, deadlineAt: startedAtMs + deadlineMs, - discardControllerCache: controllerOutage !== undefined, controllerOutage, controllerDowntimeMs, + discardControllerCache: controllerOutage !== undefined, + controllerOutage: controllerOutage && !recoveryBinding ? { ...controllerOutage, holdMs: 1000 } : controllerOutage, + controllerDowntimeMs, ...(controllerOutage ? { recoveryBinding } : {}), onControllerOutage: async value => { receipt = value;