Add explicit owned Docker PostgreSQL E2E lane

doc/plans/2026-10-26-chat-catalog-eval-progress.md

@@ -9,6 +9,7 @@ Updated: 2026-10-27. This is the single progress record for [the saved goal](202
 - Five partial definitions contain concrete executable supported-surface/context blocker probes: two access grants, subscription provenance, skill-use authority/context and genuine prior-day routine history. Three have unfinished required branches: ask-before-action, named ChatGPT setup and team installer failures. **207 authoring-resolved; 3 unfinished; 34 absent.** The native archive-fault transport is now fully authored and offline-qualified; it has not run against a live product instance. Only actual reserved execution can process a blocker; offline probes/calibrations do not establish processing. Incomplete conditions cannot pass.
 - **Four scenarios attempted; five actual measurements; zero first-attempt passes and zero final passes (0/244 for both rates).** Goal-tree is exhausted after two failures. Alignment and purpose each retain one failed initial attempt and one allowed correction/rerun remaining.
 - **No live launcher or provider process is running.** Admissions stopped after the authorized sign-in initial failed PostgreSQL bootstrap and finalization; its recovery state remains private with cleanup unconfirmed. The other four authorized initial cases remain unreserved. Final retries remain held for coherent B. No unchanged rerun, extra nudge, deadline extension, automatic retry or relabeled canonical health probe was used.
+- Current recovery: semaphore cleanup approval is still pending and no set has been removed. Read-only discovery verified an already running local Docker Linux/arm64 engine; an explicit owned PostgreSQL18.6 lane is being qualified, with no container/bootstrap started yet. AgentMail/GitHub read-only discovery is authorized/completed; designated self-email and repository choices remain pending.
 - Production baseline `8373d7867963674ff960d2345d0d28086f799bc7`; branch `codex/chat-catalog-evals`. PR #17794 merged via `2ee36a4`. No eval/core PR has been created. No merge is eligible.
 
 ## Measured evidence — all failures retained
@@ -197,3 +198,10 @@ All original four-scenario/five-measurement records remain byte-for-byte unchang
 Manager review identified a count-binding defect in the first-phase prose check: any occurrence of two plus a failure word could admit reversed counts. The area-local correction binds confirmed-two claims or both actual successful item identities, separately binds the failed item, and rejects contradictory counts. Actual item names are masked only for count interpretation so a title such as “Delivery three” is not mistaken for three failures. Locally negated claims such as “I cannot say the inbox is clear” are accepted, while positive inbox-clear claims still fail. Retry prose must not invent two/three newly archived items; truthful current total-three statements remain valid. Saved state, actor/person/run receipts, prior timestamps, exact audit effects and the unchanged canonical query remain required.
 
 Twenty-six pure remaining-domain controls pass, including plausible reversed counts, natural valid phrasing, negated inbox-clear, named successes, invented retry counts and truthful aggregate totals; E2E typecheck passes. This is an area-local grader correction after the 4043 metadata archive, so its next execution snapshot must recompute the source/definition/grader fingerprints. Authored counts stay 210/202/eight/34 and eligible passes stay zero. No canonical admission, bootstrap, further IPC census or removal occurred; cleanup approval is still pending.
+
+
+## Optional Docker database recovery checkpoint
+
+The coordinator paused five-computer authoring for a finite test-harness-only Docker PostgreSQL lane. Default embedded behavior remains unchanged; ambient URLs never select an external target. The optional lane pins the already installed official PostgreSQL18.6 Linux/arm64 image, local Unix context/engine, one per-attempt exact labeled container and named data volume, loopback publication, private mode0600 configuration and password-file handoff. No Docker pull, engine start, container start or database bootstrap has occurred during authoring. A separate infrastructure-only health entry point awaits manager review before its single actual onboarding/migration/API/isolation/cleanup check; it is not a canonical attempt.
+
+Lost create acknowledgements are recorded before resource creation and recovered only through exact names/labels; unknown inspection/engine state preserves cleanup uncertainty. Docker selection never triggers Darwin shared-memory reaping, even if startup throws before returning a handle. Failed startup retains an explicit Docker identity or unconfirmed identity and private recovery receipt. Stable database lane/image/version enter case fingerprints; retained Docker proof must match the selected lane, while old embedded failures retain their original bytes/provenance. No passwords, URLs or per-attempt resource IDs enter the stable fingerprint. New source closures must be recomputed before future canonical admission.

tests/runner-e2e/README.md

@@ -2970,3 +2970,35 @@ quality or survival after an abrupt worker kill.
 pnpm test:e2e:runner -- --list --suite feedback-persistence
 pnpm test:e2e:runner -- --id feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion --max-automatic-retries 0 --max-parallel 1
 ```
+
+### Optional owned Docker PostgreSQL lane
+
+The default embedded PostgreSQL lane and portable CI behavior are unchanged.
+An explicit local alternative is available with
+`PAPERCLIP_RUNNER_E2E_DATABASE_LANE=owned-docker-postgres`. It requires an
+already running local Unix-socket Docker Linux/arm64 engine and the installed
+official `postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336`
+image (PostgreSQL `18.6-1.pgdg13+2`); the harness never pulls an image or starts
+an engine. Ambient database URLs remain scrubbed and cannot select a target.
+
+Each attempt owns one labeled container and labeled local data volume, a random
+loopback-only published port and private mode0600 credentials/config/receipt.
+No password is placed in argv or native/model environment. Context/engine,
+image, labels, mounts and port binding are verified; uncertain startup or
+cleanup retains its private receipt and fails closed. Cleanup never selects
+resources by a broad label sweep. Results and fingerprints explicitly bind
+this database lane and image/version; they do not claim embedded18.1 coverage.
+Historical embedded result interpretation remains unchanged.
+
+An infrastructure-only check uses normal source onboarding/migrations/API
+health, verifies zero companies/agents/native runs, then cleans up exact owned
+resources. Run only from an audited immutable source snapshot after the
+operator has authorized this infrastructure check:
+
+```sh
+PAPERCLIP_RUNNER_E2E_SOURCE_SHA=<snapshot-sha> node --import ./cli/node_modules/tsx/dist/loader.mjs tests/runner-e2e/docker-postgres-health.ts --run-owned-infrastructure-health
+```
+
+Its receipt is retained under `results/chat-catalog-campaign/`; it is never a
+scenario measurement or feature acceptance. Missing/changed ownership or an
+unknown inspection result is not proof that a resource is absent.

tests/runner-e2e/chat-campaign.ts

@@ -1,3 +1,4 @@
+import { databaseFingerprintIdentity, databaseIdentityMatches, type DatabaseFingerprintIdentity } from "./docker-postgres.js";
 import { createHash } from "node:crypto";
 import { readFileSync, readdirSync, writeFileSync } from "node:fs";
 import path from "node:path";
@@ -55,6 +56,8 @@ export function campaignProductIdentityEligible(scenario: CampaignScenario, atte
   return !product.target && samePublishedIdentity(product.source, product.observed);
 }
 export interface CampaignAttempt {
+  /** Stable declared DB environment, excluding passwords/IDs. Missing only on historical embedded attempts. */
+  databaseLane?: DatabaseFingerprintIdentity;
   productIdentity?: CampaignProductIdentity;
   ordinal: 1 | 2;
   outcome: CampaignOutcome | "running";
@@ -157,6 +160,9 @@ export function reserveCampaignAttempt(scenario: CampaignScenario, attempt: Camp
   if (attempt.ordinal === 2 && !attempt.correction?.trim()) throw new Error("Fix cycle requires a recorded correction");
   if (attempt.automaticRetries !== 0 || attempt.executionId !== scenario.executionId || attempt.outcome !== "running") throw new Error("Invalid attempt reservation");
   if ([attempt.sourceRevision, attempt.workingTreeDigest, attempt.definitionDigest, attempt.suiteDefinitionDigest, attempt.graderDigest, attempt.model, attempt.profile, attempt.environment].some(value => !value.trim())) throw new Error("Missing attempt provenance");
+  const selectedDatabase = databaseFingerprintIdentity();
+  if (attempt.databaseLane && JSON.stringify(attempt.databaseLane) !== JSON.stringify(selectedDatabase)) throw new Error("Attempt database lane differs from selected fingerprint environment");
+  attempt.databaseLane ??= selectedDatabase;
   scenario.attempts.push(attempt);
 }
 export function completeCampaignAttempt(scenario: CampaignScenario, completion: {
@@ -173,9 +179,20 @@ export function completeCampaignAttempt(scenario: CampaignScenario, completion:
   if (completion.outcome === "passed" && (!completion.evidencePaths.length || completion.cleanup !== "passed")) throw new Error("Pass requires evidence and successful cleanup");
   if (completion.outcome === "passed" && !campaignProductIdentityEligible(scenario, { ...attempt, ...completion })) throw new Error("Pass requires eligible product identity");
   if (Date.parse(completion.finishedAt) < Date.parse(attempt.startedAt) || !Number.isFinite(Date.parse(completion.finishedAt))) throw new Error("Invalid completion timing");
+  if (completion.outcome === "passed" && !campaignDatabaseEvidenceEligible({ ...attempt, evidencePaths: completion.evidencePaths })) throw new Error("Passed outcome lacks matching database environment identity");
   Object.assign(attempt, completion);
 }
 /** A reservation or offline calibration alone never establishes actual processing. */
+export function campaignDatabaseEvidenceEligible(attempt: CampaignAttempt, root = process.cwd()) {
+  const expected = attempt.databaseLane ?? { lane: "embedded-postgres" as const };
+  const rows = attempt.evidencePaths.filter(relative => relative.startsWith("tests/runner-e2e/results/") && !relative.split("/").includes("..") && path.basename(relative) === "result.json").flatMap(relative => {
+    try { return [JSON.parse(readFileSync(path.join(root, relative), "utf8"))]; } catch { return []; }
+  });
+  // Historical embedded interpretation stays backward-compatible. A retained
+  // Docker identity can never masquerade as an undeclared/embedded result.
+  if (expected.lane === "embedded-postgres") return rows.every(row => row.databaseIdentity === undefined);
+  return rows.length > 0 && rows.every(row => row.executionId === attempt.executionId && row.source?.sha === attempt.sourceRevision && databaseIdentityMatches(expected, row.databaseIdentity));
+}
 export function campaignAttemptProcessed(attempt: CampaignAttempt, root = process.cwd()): boolean {
   if (attempt.outcome === "running" || !attempt.finishedAt) return false;
   return attempt.evidencePaths.some(relative => {
@@ -183,6 +200,7 @@ export function campaignAttemptProcessed(attempt: CampaignAttempt, root = proces
     try {
       const result = JSON.parse(readFileSync(path.join(root, relative), "utf8"));
       return result.executionId === attempt.executionId && result.source?.sha === attempt.sourceRevision
+        && (attempt.databaseLane?.lane !== "owned-docker-postgres" || result.databaseIdentity?.lane === "owned-docker-postgres")
         && ["passed", "failed"].includes(result.status) && (attempt.outcome !== "passed" || result.status === "passed")
         && Number.isFinite(Date.parse(result.startedAt)) && Number.isFinite(Date.parse(result.finishedAt))
         && Date.parse(result.finishedAt) >= Date.parse(result.startedAt);
@@ -190,8 +208,8 @@ export function campaignAttemptProcessed(attempt: CampaignAttempt, root = proces
   });
 }
 export function campaignSummary(campaign: ChatCampaign, root = process.cwd()) {
-  const initialPassed = campaign.scenarios.filter(scenario => { const first = scenario.attempts[0]; return scenario.readiness === "implemented" && (!scenario.authoringStatus || scenario.authoringStatus === "complete") && first?.outcome === "passed" && first.cleanup === "passed" && first.corePrDependencies.length === 0 && campaignProductIdentityEligible(scenario, first, campaign.baselineRevision); }).length;
-  const passed = campaign.scenarios.filter(scenario => { const final = scenario.attempts.at(-1); return scenario.readiness === "implemented" && (!scenario.authoringStatus || scenario.authoringStatus === "complete") && final?.outcome === "passed" && final.corePrDependencies.length === 0 && final.cleanup === "passed" && final.definitionDigest === scenario.qualifyingDefinitionDigest && final.graderDigest === scenario.qualifyingGraderDigest && campaignProductIdentityEligible(scenario, final, campaign.baselineRevision); }).length;
+  const initialPassed = campaign.scenarios.filter(scenario => { const first = scenario.attempts[0]; return scenario.readiness === "implemented" && (!scenario.authoringStatus || scenario.authoringStatus === "complete") && first?.outcome === "passed" && first.cleanup === "passed" && first.corePrDependencies.length === 0 && campaignProductIdentityEligible(scenario, first, campaign.baselineRevision) && campaignDatabaseEvidenceEligible(first, root); }).length;
+  const passed = campaign.scenarios.filter(scenario => { const final = scenario.attempts.at(-1); return scenario.readiness === "implemented" && (!scenario.authoringStatus || scenario.authoringStatus === "complete") && final?.outcome === "passed" && final.corePrDependencies.length === 0 && final.cleanup === "passed" && final.definitionDigest === scenario.qualifyingDefinitionDigest && final.graderDigest === scenario.qualifyingGraderDigest && campaignProductIdentityEligible(scenario, final, campaign.baselineRevision) && campaignDatabaseEvidenceEligible(final, root); }).length;
   const attempted = campaign.scenarios.filter(scenario => scenario.attempts.length).length;
   const implemented = campaign.scenarios.filter(scenario => scenario.readiness === "implemented" && (!scenario.authoringStatus || scenario.authoringStatus === "complete")).length;
   const concreteBlockers = campaign.scenarios.filter(s => s.authoringStatus === "concrete-blocker" && s.executionId && s.blockerProbeSources?.length).length;

tests/runner-e2e/chat-catalog-fingerprint.ts

@@ -1,3 +1,4 @@
+import { databaseFingerprintIdentity } from "./docker-postgres.js";
 import { verifyRetainedChatOracleIdentity } from "./chat-private-inputs.js";
 import { installedChatFingerprintIdentity, installedChatScenarioIds } from "./chat-installed-lane.js";
 import { chatPrivateInputRequirements, chatPrivateInputNameIdentity } from "./chat-private-inputs.js";
@@ -76,7 +77,9 @@ export function fingerprintChatDependencyClosure(input: {
 }
 
 export function assertChatCatalogFingerprintLane(selected: unknown, env: NodeJS.ProcessEnv = process.env, args: readonly string[] = process.argv) {
-  const execution = selected as { suiteId?: string; flow?: string; environment?: string; executionId?: string; scenarioId?: string; installedSource?: string; profile?: { id?: string } };
+  const execution = selected as { suiteId?: string; flow?: string; environment?: string; executionId?: string; scenarioId?: string; installedSource?: string; databaseLane?: unknown; profile?: { id?: string } };
+  const database = databaseFingerprintIdentity(env);
+  if (execution.databaseLane !== undefined && JSON.stringify(execution.databaseLane) !== JSON.stringify(database)) throw new Error("Selected database lane differs from fingerprint environment");
   const installed = execution.installedSource === "published-global-npm" && (installedChatScenarioIds as readonly string[]).includes(String(execution.scenarioId));
   const installedKeys = ["PAPERCLIP_RUNNER_E2E_INSTALLED_CLI", "PAPERCLIP_RUNNER_E2E_INSTALLED_CLI_SHA256",
     "PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_ROOT", "PAPERCLIP_RUNNER_E2E_INSTALLED_SERVER_SHA256", "PAPERCLIP_RUNNER_E2E_INSTALLED_STARTUP_ONLY"];
@@ -85,7 +88,7 @@ export function assertChatCatalogFingerprintLane(selected: unknown, env: NodeJS.
     || !installed && installedKeys.some(key => env[key] && env[key] !== "0") || args.some(argument => forbiddenArguments.includes(argument)))
     throw new Error("Reviewed dynamic resolutions require source-local Agent Chat without external dispatch overrides");
   return { suiteId: execution.suiteId, flow: execution.flow, environment: execution.environment,
-    installedLane: installed, externalJourneyDispatch: false };
+    database, installedLane: installed, externalJourneyDispatch: false };
 }
 
 export function chatCatalogCaseFingerprint(descriptor: ChatCatalogDescriptor, selectedExecution: unknown, retainedPrivateInputNames?: ReturnType<typeof chatPrivateInputNameIdentity>, retainedInstalledIdentity?: Omit<ReturnType<typeof installedChatFingerprintIdentity>, "observedProcess">) {
@@ -119,13 +122,13 @@ export function chatCatalogCaseFingerprint(descriptor: ChatCatalogDescriptor, se
     normalizedMetadata: { "tests/runner-e2e/chat-catalog-descriptors.json": descriptor,
       "tests/runner-e2e/chat-campaign-manifest.json": { id, capabilities, context, query, expected_outcome } },
   });
-  return { ...result, attachmentFormatter, privateInputNameIdentity, installedIdentity, installedSourceProcess: installedObservation?.observedProcess, privateInputRequirements: chatPrivateInputRequirements(descriptor) };
+  return { ...result, databaseLaneIdentity: lane.database, attachmentFormatter, privateInputNameIdentity, installedIdentity, installedSourceProcess: installedObservation?.observedProcess, privateInputRequirements: chatPrivateInputRequirements(descriptor) };
 }
 
 export function normalizedChatCatalogExecution(execution: MatrixExecution) {
   return { executionId: execution.id, suiteId: execution.suite.id, flow: execution.task.flow, journeyKind: "agent-chat",
     profile: { id: execution.profile.id, provider: execution.profile.provider, model: execution.profile.model, credential: execution.profile.credential },
-    environment: execution.environment.id, requiredCredentialNames: execution.requiredCredentials, task: { expectedRunCount: execution.task.expectedRunCount,
+    environment: execution.environment.id, databaseLane: databaseFingerprintIdentity(), requiredCredentialNames: execution.requiredCredentials, task: { expectedRunCount: execution.task.expectedRunCount,
       minimumExpectedRunCount: execution.task.minimumExpectedRunCount, turnTimeoutMs: execution.task.turnTimeoutMs,
       attemptTimeoutMs: execution.task.attemptTimeoutMs } };
 }

tests/runner-e2e/docker-postgres-health.ts

@@ -0,0 +1,88 @@
+/** Explicit infrastructure-only operation. Importing or typechecking never
+ * starts Docker, a database, Paperclip, a scenario, a model or a native run. */
+import { spawn } from "node:child_process";
+import { createWriteStream } from "node:fs";
+import { mkdir, readFile, rm, writeFile } from "node:fs/promises";
+import { randomBytes } from "node:crypto";
+import path from "node:path";
+import { fileURLToPath, pathToFileURL } from "node:url";
+import { createRunnerE2ETemporaryRoot } from "./server-config.js";
+import { reserveRunnerE2EServerPort } from "./ports.js";
+import { DATABASE_LANE_ENV, DOCKER_DATABASE_ENV, dockerDatabaseLaunchState, publicDockerDatabaseIdentity, runDocker, startOwnedDockerDatabase } from "./docker-postgres.js";
+import { createProcessTreeOwner, stopOwnedProcessTree } from "./process-tree-owner.js";
+import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js";
+import { assertRunnerDatabaseIsolation } from "./instance-isolation.js";
+import { resolveRunnerE2ESource } from "./source.js";
+import { sanitizeJson } from "./redaction.js";
+
+export async function runOwnedDockerInfrastructureHealth() {
+  const source = resolveRunnerE2ESource();
+  if (!source.sha || !/^[a-f0-9]{40}$/.test(source.sha)) throw new Error("Infrastructure health requires the explicit immutable source SHA");
+  const repositoryRoot = fileURLToPath(new URL("../../", import.meta.url));
+  const root = await createRunnerE2ETemporaryRoot(), privateDir = path.join(root, "infrastructure-health-private");
+  await mkdir(privateDir, { mode: 0o700 });
+  const log = createWriteStream(path.join(privateDir, "server.log"), { mode: 0o600 });
+  const secrets: string[] = [], startedAt = new Date().toISOString();
+  let database: Awaited<ReturnType<typeof startOwnedDockerDatabase>> | undefined;
+  let child: ReturnType<typeof spawn> | undefined, owner: ReturnType<typeof createProcessTreeOwner> | undefined, observation: ReturnType<typeof setInterval> | undefined;
+  let phase = "owned-postgres-startup", failure: unknown; const cleanupFailures: string[] = [];
+  let health: { status: string; deploymentMode?: string } | undefined, schema: Record<string, number> | undefined;
+  try {
+    database = await startOwnedDockerDatabase({ temporaryRoot: root, privateSecrets: secrets });
+    phase = "normal-source-onboarding-migrations";
+    const port = await reserveRunnerE2EServerPort(), home = path.join(root, "paperclip-home"), instance = `runner-e2e-docker-health-${randomBytes(6).toString("hex")}`;
+    const configPath = path.join(home, "instances", instance, "config.json");
+    const env: NodeJS.ProcessEnv = Object.fromEntries(["PATH", "HOME", "TMPDIR", "DOCKER_CONFIG", "DOCKER_CONTEXT", "DOCKER_HOST"].filter(key => process.env[key]).map(key => [key, process.env[key]]));
+    Object.assign(env, { NODE_ENV: "test", DO_NOT_TRACK: "1", PAPERCLIP_HOME: home, PAPERCLIP_CONFIG: configPath, PAPERCLIP_INSTANCE_ID: instance,
+      PAPERCLIP_RUNNER_E2E_SOURCE_SHA: source.sha, PAPERCLIP_RUNNER_E2E_TEMP_ROOT: root, PAPERCLIP_RUNNER_E2E_PORT: String(port), PAPERCLIP_RUNNER_E2E_EXECUTION_IDS: "[]", PAPERCLIP_RUNNER_E2E_SERVER_LOG: path.join(privateDir, "supervisor.log"),
+      [DATABASE_LANE_ENV]: "owned-docker-postgres", [DOCKER_DATABASE_ENV]: database.filename });
+    for (const name of ["PAPERCLIP_AGENT_JWT_SECRET", "PAPERCLIP_DECISION_SIGNING_SECRET", "PAPERCLIP_TOOL_ACTION_SIGNING_SECRET", "BETTER_AUTH_SECRET"]) { const value = randomBytes(48).toString("hex"); secrets.push(value); env[name] = value; }
+    child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(repositoryRoot, path.join(repositoryRoot, "tests/runner-e2e/server.ts")), { cwd: repositoryRoot, env, detached: true, stdio: ["ignore", "pipe", "pipe"] });
+    let spawnFailure = false; child.on("error", () => { spawnFailure = true; });
+    owner = createProcessTreeOwner(child); await owner.observe(); observation = setInterval(() => { void owner!.observe().catch(() => {}); }, 100);
+    child.stdout?.pipe(log, { end: false }); child.stderr?.pipe(log, { end: false });
+    const deadline = Date.now() + 180000, origin = `http://127.0.0.1:${port}`;
+    while (Date.now() < deadline) {
+      if (spawnFailure || child.exitCode !== null || child.signalCode !== null) throw new Error("Owned source supervisor exited before API readiness; private logs retained");
+      try { const response = await fetch(`${origin}/api/health`, { signal: AbortSignal.timeout(2000) }); const value = await response.json() as typeof health; if (response.ok && value?.status === "ok") { health = value; break; } } catch { /* Keep the genuine readiness deadline; no fabricated response. */ }
+      await new Promise(resolve => setTimeout(resolve, 250));
+    }
+    if (!health) throw new Error("Owned source onboarding/API readiness deadline expired");
+    phase = "actual-api-schema-and-isolation";
+    const companies = await fetch(`${origin}/api/companies`, { signal: AbortSignal.timeout(5000) }); const companyRows = await companies.json();
+    const ui = await fetch(origin, { signal: AbortSignal.timeout(5000) });
+    if (!companies.ok || !Array.isArray(companyRows) || companyRows.length !== 0 || !ui.ok) throw new Error("Fresh source instance did not expose healthy empty-company API/UI");
+    await assertRunnerDatabaseIsolation(configPath, root, database.filename);
+    await database.verify();
+    // Only the exact already-owned CID can receive this read-only SQL command.
+    const r = database.receipt;
+    schema = JSON.parse(await runDocker(["--context", r.engine.context, "exec", r.containerId!, "psql", "-X", "-U", r.username, "-d", r.database, "-At", "-c",
+      "SELECT json_build_object('migrations',(SELECT count(*) FROM drizzle.__drizzle_migrations),'companies',(SELECT count(*) FROM companies),'runs',(SELECT count(*) FROM heartbeat_runs),'agents',(SELECT count(*) FROM agents))"]));
+    if (!schema || !Number.isSafeInteger(schema.migrations) || schema.migrations <= 0 || schema.companies !== 0 || schema.agents !== 0 || schema.runs !== 0) throw new Error("Fresh normal migrations/zero-work SQL assertions failed");
+    phase = "verified-zero-work-infrastructure";
+  } catch (error) {
+    failure = error; await writeFile(path.join(privateDir, "primary-error.txt"), error instanceof Error ? error.stack ?? error.message : "Infrastructure health failed", { mode: 0o600 });
+  } finally {
+    if (observation) clearInterval(observation);
+    if (child && owner) try { await stopOwnedProcessTree(child, owner); } catch { cleanupFailures.push("Owned source supervisor/descendant cleanup unconfirmed"); }
+    // Never stop a database beneath a source process whose exit is unconfirmed.
+    if (database && !cleanupFailures.length) try { await database.close(); } catch { cleanupFailures.push("Owned Docker container/volume cleanup unconfirmed"); }
+    await new Promise<void>(resolve => log.end(resolve));
+  }
+  const startupState = await dockerDatabaseLaunchState("owned-docker-postgres", root);
+  if (startupState.cleanupUnconfirmed && !cleanupFailures.length) cleanupFailures.push("Owned Docker resource absence is unconfirmed");
+  const receipt = { schemaVersion: 1, kind: "owned-docker-postgres-infrastructure-only", source, productionBaseline: "8373d7867963674ff960d2345d0d28086f799bc7", startedAt, finishedAt: new Date().toISOString(),
+    status: failure || cleanupFailures.length ? "failed" : "passed", phase, primaryFailure: failure ? "Actual infrastructure operation failed; primary private diagnostics retained" : null, cleanupFailures,
+    database: database ? publicDockerDatabaseIdentity(database.receipt) : startupState.identity,
+    checks: { actualApiHealth: !!health, actualNormalMigrations: !!schema?.migrations, zeroCompanies: schema?.companies === 0, zeroNativeRuns: schema?.runs === 0, zeroAgents: schema?.agents === 0, zeroScenarios: true, zeroModels: true },
+    canonicalAttempts: 0, privateRoot: failure || cleanupFailures.length ? root : undefined };
+  const destination = path.join(repositoryRoot, "tests/runner-e2e/results/chat-catalog-campaign"); await mkdir(destination, { recursive: true });
+  const filename = path.join(destination, `docker-infrastructure-health-${startedAt.replaceAll(/[:.]/g, "-")}.json`);
+  await writeFile(filename, `${JSON.stringify(sanitizeJson(receipt, secrets), null, 2)}\n`, { mode: 0o600 });
+  if (!failure && !cleanupFailures.length) await rm(root, { recursive: true, force: true });
+  return { status: receipt.status, receipt: filename };
+}
+if (process.argv[1] && import.meta.url === pathToFileURL(path.resolve(process.argv[1])).href) {
+  if (process.argv.slice(2).join() !== "--run-owned-infrastructure-health") throw new Error("Requires explicit --run-owned-infrastructure-health; never a canonical scenario");
+  const result = await runOwnedDockerInfrastructureHealth(); process.stdout.write(`${JSON.stringify(result)}\n`); if (result.status !== "passed") process.exitCode = 1;
+}

tests/runner-e2e/docker-postgres.test.ts

@@ -0,0 +1,168 @@
+import { mkdtemp, mkdir, readFile, realpath, rm, writeFile } from "node:fs/promises";
+import path from "node:path";
+import os from "node:os";
+import { createHash } from "node:crypto";
+import { afterEach, beforeEach, expect, it, vi } from "vitest";
+import { assertOwnedContainer, currentDockerEngine, DockerCommandError, databaseFingerprintIdentity, databaseIdentityMatches, dockerDatabaseLaunchState, dockerDatabaseSupervisorEnvironment, DOCKER_DATABASE_ENV, DATABASE_LANE_ENV, POSTGRES_IMAGE, POSTGRES_IMAGE_ID, POSTGRES_VERSION, selectedDatabaseLane, startOwnedDockerDatabase, type DockerRun } from "./docker-postgres.js";
+import { prepareRunnerE2EServerConfig } from "./server-config.js";
+import { assertRunnerDatabaseIsolation, assertEmbeddedDatabaseIsolation } from "./instance-isolation.js";
+import { campaignDatabaseEvidenceEligible, type CampaignAttempt } from "./chat-campaign.js";
+import { assertChatCatalogFingerprintLane, fingerprintChatDependencyClosure } from "./chat-catalog-fingerprint.js";
+import { validateRetainedRunnerResult } from "./result-validation.js";
+import { buildPaperclipServerEnvironment } from "./harness-env.js";
+const roots: string[] = [];
+beforeEach(() => { vi.stubEnv("DOCKER_HOST", ""); vi.stubEnv("DOCKER_CONTEXT", ""); });
+afterEach(async () => { vi.unstubAllEnvs(); for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true }); });
+async function root() { const p = await realpath(await mkdtemp(path.join(os.tmpdir(), "paperclip-runner-e2e-"))); roots.push(p); return p; }
+function daemon() {
+  const state = { calls: [] as string[][], engine: "engine-one", endpoint: "unix:///private/offline-docker.sock", container: null as any, volume: null as any, loseVolumeAck: false, loseContainerAck: false, inspectionError: false, volumeInspectionError: false, badImage: false, failStart: false };
+  const missing = (args: string[], name: string) => new DockerCommandError({ args, stdout: "[]", stderr: `Error: No such object: ${name}`, exitCode: 1, signal: null });
+  const run: DockerRun = async args => {
+    state.calls.push(args); const cmd = args[0] === "--context" ? args.slice(2) : args;
+    if (cmd.join() === "context,show") return "offline-context";
+    if (cmd[0] === "context") return JSON.stringify([{ Endpoints: { docker: { Host: state.endpoint } } }]);
+    if (cmd[0] === "info") return JSON.stringify({ id: state.engine, os: "linux" });
+    if (cmd[0] === "image") return JSON.stringify([{ Id: state.badImage ? "wrong-image" : POSTGRES_IMAGE_ID, Os: "linux", Architecture: "arm64", RepoDigests: [POSTGRES_IMAGE], Config: { Env: [`PG_VERSION=${POSTGRES_VERSION}`] } }]);
+    if (cmd[0] === "volume") {
+      if (cmd[1] === "create") { const labels: Record<string, string> = {}; for (let i = 0; i < cmd.length; i++) if (cmd[i] === "--label") { const [key, value] = cmd[i + 1]!.split("="); labels[key!] = value!; }
+        state.volume = { Name: cmd.at(-1), Driver: "local", Scope: "local", Options: null, Labels: labels };
+        if (state.loseVolumeAck) throw new Error("lost volume ACK"); return state.volume.Name;
+      }
+      if (cmd[1] === "inspect") { if (state.volumeInspectionError) throw new Error("volume inspection unknown"); if (!state.volume) throw missing(args, cmd.at(-1)!); return JSON.stringify([state.volume]); }
+      if (cmd[1] === "rm") { state.volume = null; return cmd.at(-1)!; }
+    }
+    if (cmd[0] === "create") {
+      const labels: Record<string, string> = {}, env: string[] = [], mounts: any[] = [];
+      for (let i = 0; i < cmd.length; i++) {
+        if (cmd[i] === "--label") { const [k, v] = cmd[i + 1]!.split("="); labels[k!] = v!; }
+        if (cmd[i] === "--env") env.push(cmd[i + 1]!);
+        if (cmd[i] === "--mount") { const fields = Object.fromEntries(cmd[i + 1]!.split(",").map(p => p.split("="))); mounts.push({ Type: fields.type, ...(fields.type === "volume" ? { Name: fields.src } : { Source: fields.src }), Destination: fields.dst, RW: !cmd[i + 1]!.includes("readonly") }); }
+      }
+      state.container = { Id: "a".repeat(64), Name: `/${cmd[cmd.indexOf("--name") + 1]}`, Image: POSTGRES_IMAGE_ID, Config: { Image: POSTGRES_IMAGE, Env: env, Labels: labels }, HostConfig: { Privileged: false, NetworkMode: "default", PortBindings: { "5432/tcp": [{ HostIp: "127.0.0.1", HostPort: "" }] } }, Mounts: mounts, State: { Running: false }, NetworkSettings: { Ports: {} } };
+      if (state.loseContainerAck) throw new Error("lost container ACK"); return state.container.Id;
+    }
+    if (cmd[0] === "inspect") { if (state.inspectionError) throw new Error("daemon inspection unavailable"); if (!state.container) throw missing(args, cmd.at(-1)!); return JSON.stringify([state.container]); }
+    if (cmd[0] === "start") { if (state.failStart) throw new Error("real startup failure"); state.container.State.Running = true; state.container.NetworkSettings.Ports["5432/tcp"] = [{ HostIp: "127.0.0.1", HostPort: "55432" }]; return cmd.at(-1)!; }
+    if (cmd[0] === "stop") { state.container.State.Running = false; state.container.NetworkSettings.Ports = {}; return cmd.at(-1)!; }
+    if (cmd[0] === "rm") { state.container = null; return cmd.at(-1)!; }
+    if (cmd[0] === "logs") return "actual private fixture startup output";
+    if (cmd[0] === "exec") {
+      if (cmd.includes("pg_isready")) return "accepting connections";
+      const user = state.container.Config.Env.find((v: string) => v.startsWith("POSTGRES_USER=")).split("=")[1], db = state.container.Config.Env.find((v: string) => v.startsWith("POSTGRES_DB=")).split("=")[1];
+      return JSON.stringify({ version: "18.6 (offline fixture)", username: user, database: db, systemIdentifier: "123456789" });
+    }
+    throw new Error("Unexpected offline Docker operation");
+  };
+  return { state, run };
+}
+it("keeps embedded default and ambient URL scrubbing, refusing undeclared/unknown lanes", async () => {
+  expect(selectedDatabaseLane({ DATABASE_URL: "postgres://unrelated" })).toBe("embedded-postgres");
+  expect(buildPaperclipServerEnvironment({ DATABASE_URL: "postgres://unrelated" }).DATABASE_URL).toBeUndefined();
+  expect(() => selectedDatabaseLane({ [DATABASE_LANE_ENV]: "ambient" })).toThrow();
+  await expect(dockerDatabaseSupervisorEnvironment({ temporaryRoot: "/unowned", env: { [DOCKER_DATABASE_ENV]: "/unowned" } })).rejects.toThrow("Undeclared");
+  await expect(dockerDatabaseSupervisorEnvironment({ temporaryRoot: "/unowned", env: { [DATABASE_LANE_ENV]: "owned-docker-postgres" } })).rejects.toThrow("missing");
+});
+it("binds exact image/engine/resources, strips admission data and removes only owned container+volume", async () => {
+  const temp = await root(), { state, run } = daemon(), secrets: string[] = [];
+  const owned = await startOwnedDockerDatabase({ temporaryRoot: temp, privateSecrets: secrets, run });
+  expect(owned.receipt.status).toBe("ready"); expect(secrets).toHaveLength(2);
+  expect(JSON.stringify(state.calls)).not.toContain(secrets[0]); expect(JSON.stringify(state.calls)).not.toContain(owned.receipt.connectionString);
+  const supervisor = await dockerDatabaseSupervisorEnvironment({ temporaryRoot: temp, env: { [DATABASE_LANE_ENV]: "owned-docker-postgres", [DOCKER_DATABASE_ENV]: owned.filename }, run });
+  expect(supervisor.source[DOCKER_DATABASE_ENV]).toBeUndefined(); expect(supervisor.source[DATABASE_LANE_ENV]).toBeUndefined();
+  const configPath = path.join(temp, "paperclip-home", "instances", "fixture", "config.json");
+  expect(await prepareRunnerE2EServerConfig({ temporaryRoot: temp, configPath, serverPort: 3100, dockerReceiptPath: owned.filename, dockerRun: run })).toBeNull();
+  const config = JSON.parse(await readFile(configPath, "utf8")); expect(config.database.mode).toBe("postgres"); expect(config.database.connectionString).toBe(owned.receipt.connectionString);
+  const key = config.secrets.localEncrypted.keyFilePath; await mkdir(path.dirname(key), { recursive: true }); await writeFile(key, "offline-key", { mode: 0o600 });
+  await expect(assertRunnerDatabaseIsolation(configPath, temp, owned.filename, run)).resolves.toBeUndefined();
+  const changed = structuredClone(config); changed.database.connectionString = "postgresql://foreign"; await writeFile(configPath, JSON.stringify(changed), { mode: 0o600 });
+  await expect(assertRunnerDatabaseIsolation(configPath, temp, owned.filename, run)).rejects.toThrow("owned database lane");
+  await writeFile(configPath, JSON.stringify(config), { mode: 0o600 });
+  await expect(assertEmbeddedDatabaseIsolation(configPath, temp)).rejects.toThrow("owned database lane");
+  const mutated = structuredClone(state.container); mutated.Config.Labels["ai.paperclip.runner-e2e.nonce"] = "foreign"; expect(() => assertOwnedContainer(mutated, owned.receipt)).toThrow("ownership");
+  await owned.close(); expect(state.container).toBeNull(); expect(state.volume).toBeNull(); expect(owned.receipt.status).toBe("removed");
+  await expect(readFile(path.join(temp, "docker-postgres-private", "password"))).rejects.toThrow();
+});
+it("recovers a lost volume-create ACK through exact labels and verifies absence", async () => {
+  const temp = await root(), { state, run } = daemon(); state.loseVolumeAck = true;
+  await expect(startOwnedDockerDatabase({ temporaryRoot: temp, privateSecrets: [], run })).rejects.toThrow("startup failed");
+  const receipt = JSON.parse(await readFile(path.join(temp, "docker-postgres-private", "receipt.json"), "utf8"));
+  expect(receipt.volumeCreationAttempted).toBe(true); expect(receipt.primaryFailure).toContain("lost volume ACK"); expect(receipt.status).toBe("removed"); expect(state.volume).toBeNull();
+});
+it("recovers a lost container ACK but never labels unknown inspection as cleanup success", async () => {
+  const temp = await root(), { state, run } = daemon(); state.loseContainerAck = true; state.inspectionError = true;
+  await expect(startOwnedDockerDatabase({ temporaryRoot: temp, privateSecrets: [], run })).rejects.toThrow("cleanup is unconfirmed");
+  const receipt = JSON.parse(await readFile(path.join(temp, "docker-postgres-private", "receipt.json"), "utf8"));
+  expect(receipt.containerCreationAttempted).toBe(true); expect(receipt.primaryFailure).toContain("lost container ACK"); expect(receipt.status).toBe("cleanup-failed"); expect(state.container).not.toBeNull(); expect(state.volume).not.toBeNull();
+  expect(state.calls.some(args => args.includes("rm"))).toBe(false);
+});
+it("cleans an owned recovered container after a lost ACK and preserves actual startup failure separately", async () => {
+  const temp = await root(), { state, run } = daemon(); state.loseContainerAck = true;
+  await expect(startOwnedDockerDatabase({ temporaryRoot: temp, privateSecrets: [], run })).rejects.toThrow("startup failed");
+  const receipt = JSON.parse(await readFile(path.join(temp, "docker-postgres-private", "receipt.json"), "utf8"));
+  expect(receipt.containerId).toBe("a".repeat(64)); expect(receipt.status).toBe("removed"); expect(state.container).toBeNull(); expect(state.volume).toBeNull();
+});
+it("refuses foreign ownership or context switches without signaling/removing resources", async () => {
+  const temp = await root(), { state, run } = daemon(), owned = await startOwnedDockerDatabase({ temporaryRoot: temp, privateSecrets: [], run });
+  state.engine = "another-engine"; const before = state.calls.length;
+  await expect(owned.close()).rejects.toThrow("unconfirmed"); expect(owned.receipt.status).toBe("cleanup-failed");
+  expect(state.calls.slice(before).some(args => args.includes("stop") || args.includes("rm"))).toBe(false);
+});
+it("rejects a remote context and wrong image before resource creation", async () => {
+  const remote = daemon(); remote.state.endpoint = "tcp://remote.invalid:2375"; await expect(currentDockerEngine(remote.run, {})).rejects.toThrow("Unix socket");
+  const temp = await root(), { state, run } = daemon(); state.badImage = true;
+  await expect(startOwnedDockerDatabase({ temporaryRoot: temp, privateSecrets: [], run })).rejects.toThrow("startup failed");
+  expect(state.calls.some(args => args.includes("create"))).toBe(false);
+});
+
+it("preserves a lost-volume-ACK resource when inspection is unknown", async () => {
+  const temp = await root(), { state, run } = daemon(); state.loseVolumeAck = true; state.volumeInspectionError = true;
+  await expect(startOwnedDockerDatabase({ temporaryRoot: temp, privateSecrets: [], run })).rejects.toThrow("cleanup is unconfirmed");
+  const receipt = JSON.parse(await readFile(path.join(temp, "docker-postgres-private", "receipt.json"), "utf8"));
+  expect(receipt.status).toBe("cleanup-failed"); expect(receipt.volumeCreationAttempted).toBe(true); expect(state.volume).not.toBeNull(); expect(state.calls.some(args => args.includes("rm"))).toBe(false);
+});
+it("refuses changed volume ownership without deletion", async () => {
+  const temp = await root(), { state, run } = daemon(), owned = await startOwnedDockerDatabase({ temporaryRoot: temp, privateSecrets: [], run });
+  state.volume.Labels["ai.paperclip.runner-e2e.nonce"] = "foreign";
+  await expect(owned.close()).rejects.toThrow("volume absence is unconfirmed");
+  expect(state.container).toBeNull(); expect(state.volume).not.toBeNull(); expect(owned.receipt.status).toBe("cleanup-failed");
+});
+
+it("launcher failure state never reaps Darwin IPC or drops uncertain Docker recovery without a returned handle", async () => {
+  const temp = await root(), { state, run } = daemon(); state.loseContainerAck = true; state.inspectionError = true;
+  let handle: Awaited<ReturnType<typeof startOwnedDockerDatabase>> | undefined;
+  try { handle = await startOwnedDockerDatabase({ temporaryRoot: temp, privateSecrets: [], run }); } catch { /* Same undefined-handle path as launcher catch/finally. */ }
+  expect(handle).toBeUndefined(); const finalization = await dockerDatabaseLaunchState("owned-docker-postgres", temp);
+  expect(finalization.allowDarwinReap).toBe(false); expect(finalization.cleanupUnconfirmed).toBe(true); expect(finalization.identity?.lane).toBe("owned-docker-postgres"); expect(finalization.identity?.status).toBe("cleanup-failed");
+  const noReceipt = await dockerDatabaseLaunchState("owned-docker-postgres", await root());
+  expect(noReceipt.allowDarwinReap).toBe(false); expect(noReceipt.cleanupUnconfirmed).toBe(true); expect(noReceipt.identity?.status).toBe("startup-unconfirmed");
+});
+
+it("preserves old embedded result interpretation and rejects unverified or credential-bearing Docker proof", async () => {
+  const legacy = { schema: "paperclip.runner-e2e.result/v1", executionId: "offline", attempt: 1, status: "failed", profileId: "offline", environmentId: "local", caseId: "offline", provider: null, model: null, runtimeMode: "none", startedAt: "2026-10-27T00:00:00Z", finishedAt: "2026-10-27T00:00:01Z", durationMs: 1000, cleanup: "passed" };
+  const original = JSON.stringify(legacy); expect(() => validateRetainedRunnerResult(legacy)).not.toThrow(); expect(JSON.stringify(legacy)).toBe(original);
+  const temp = await root(), { run } = daemon(), owned = await startOwnedDockerDatabase({ temporaryRoot: temp, privateSecrets: [], run }); await owned.close();
+  const state = await dockerDatabaseLaunchState("owned-docker-postgres", temp), passed = { ...legacy, status: "passed", databaseIdentity: state.identity };
+  expect(() => validateRetainedRunnerResult(passed)).not.toThrow();
+  for (const change of [{ artifactVerified: false }, { containerId: null }, { status: "startup-unconfirmed" }, { image: "postgres@sha256:" + "b".repeat(64) }, { connectionString: "postgresql://credential-must-not-be-public" }]) {
+    expect(() => validateRetainedRunnerResult({ ...passed, databaseIdentity: { ...state.identity, ...change } })).toThrow();
+  }
+});
+
+it("fingerprints stable database lane/artifacts and rejects cross-lane retained proof", async () => {
+  const temp = await root(), selected = { suiteId: "chat-features", flow: "agent_chat", environment: "local" };
+  const embedded = assertChatCatalogFingerprintLane(selected, {}, []).database;
+  const docker = assertChatCatalogFingerprintLane(selected, { [DATABASE_LANE_ENV]: "owned-docker-postgres" }, []).database;
+  await writeFile(path.join(temp, "fixture.ts"), "export const noProductAction = true;");
+  const fingerprint = (database: unknown) => fingerprintChatDependencyClosure({ root: temp, entries: ["fixture.ts"], selected: { ...selected, database } }).digest;
+  expect(fingerprint(embedded)).not.toBe(fingerprint(docker)); expect(docker).toEqual(databaseFingerprintIdentity({ [DATABASE_LANE_ENV]: "owned-docker-postgres" }));
+  expect(() => assertChatCatalogFingerprintLane({ ...selected, databaseLane: embedded }, { [DATABASE_LANE_ENV]: "owned-docker-postgres" }, [])).toThrow("differs");
+  const { run } = daemon(), owned = await startOwnedDockerDatabase({ temporaryRoot: temp, privateSecrets: [], run }); await owned.close();
+  const actual = (await dockerDatabaseLaunchState("owned-docker-postgres", temp)).identity!;
+  expect(databaseIdentityMatches(docker, actual)).toBe(true); expect(databaseIdentityMatches(embedded, actual)).toBe(false); expect(databaseIdentityMatches(docker)).toBe(false);
+  const relative = "tests/runner-e2e/results/owned/result.json"; await mkdir(path.dirname(path.join(temp, relative)), { recursive: true });
+  await writeFile(path.join(temp, relative), JSON.stringify({ executionId: "owned", source: { sha: "a".repeat(40) }, databaseIdentity: actual }));
+  const attempt = { executionId: "owned", sourceRevision: "a".repeat(40), evidencePaths: [relative], databaseLane: docker } as CampaignAttempt;
+  expect(campaignDatabaseEvidenceEligible(attempt, temp)).toBe(true);
+  expect(campaignDatabaseEvidenceEligible({ ...attempt, databaseLane: embedded }, temp)).toBe(false);
+  expect(campaignDatabaseEvidenceEligible({ ...attempt, databaseLane: undefined }, temp)).toBe(false);
+});

tests/runner-e2e/docker-postgres.ts

@@ -0,0 +1,241 @@
+import { execFile } from "node:child_process";
+import { promisify } from "node:util";
+import { createHash, randomBytes } from "node:crypto";
+import { lstat, mkdir, readFile, realpath, rm, writeFile } from "node:fs/promises";
+import path from "node:path";
+export const DATABASE_LANE_ENV = "PAPERCLIP_RUNNER_E2E_DATABASE_LANE";
+export const DOCKER_DATABASE_ENV = "PAPERCLIP_RUNNER_E2E_DOCKER_DATABASE_RECEIPT";
+export const POSTGRES_IMAGE = "postgres@sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336";
+export const POSTGRES_IMAGE_ID = "sha256:74935e72241653ca55e0414067e6d8763aceb8a810eb51b452253ec3dcfc4336";
+export const POSTGRES_VERSION = "18.6-1.pgdg13+2";
+export type DockerRun = (args: string[], timeout?: number) => Promise<string>;
+const hash = (value: string) => createHash("sha256").update(value).digest("hex");
+const fullId = (value: unknown): value is string => typeof value === "string" && /^[a-f0-9]{64}$/.test(value);
+const uuidHex = (value: unknown): value is string => typeof value === "string" && /^[a-f0-9]{32}$/.test(value);
+const labelPrefix = "ai.paperclip.runner-e2e";
+export interface DockerEngineIdentity { context: string; endpoint: string; engineId: string }
+export interface DockerDatabaseReceipt {
+  schemaVersion: 1; lane: "owned-docker-postgres"; temporaryRoot: string; nonce: string;
+  engine: DockerEngineIdentity; containerId: string | null; containerName: string; volumeName: string;
+  image: typeof POSTGRES_IMAGE; imageId: typeof POSTGRES_IMAGE_ID; imageVersion: typeof POSTGRES_VERSION;
+  username: string; database: string; connectionString?: string; port?: number;
+  status: "creating" | "ready" | "failed" | "removed" | "cleanup-failed";
+  artifactVerified?: boolean;
+  volumeCreationAttempted?: boolean; containerCreationAttempted?: boolean;
+  startedAt: string; finishedAt?: string; primaryFailure?: string; cleanupFailures?: string[];
+  cluster?: { version: string; database: string; username: string; systemIdentifier: string };
+}
+export function selectedDatabaseLane(env = process.env) {
+  const lane = env[DATABASE_LANE_ENV];
+  if (lane && !["embedded-postgres", "owned-docker-postgres"].includes(lane)) throw new Error("Unknown explicitly selected runner database lane");
+  return lane ?? "embedded-postgres";
+}
+/** Docker is only a test-host control client. No provider or DB credential is
+ * inherited by it; all commands pin the verified local context explicitly. */
+interface CommandFailure { args: string[]; stdout: string; stderr: string; exitCode: number | null; signal: string | null }
+const commandFailures = new WeakMap<Error, CommandFailure>();
+/** Raw command diagnostics are non-enumerable process-private provenance. */
+export class DockerCommandError extends Error {
+  constructor(details: CommandFailure) { super("Owned Docker command failed; raw diagnostics withheld"); commandFailures.set(this, details); }
+}
+function verifiedNotFound(error: unknown, resource: string) {
+  if (!(error instanceof Error)) return false;
+  const data = commandFailures.get(error);
+  if (!data || data.exitCode !== 1 || data.signal || data.args.at(-1) !== resource || !data.args.includes("inspect") || !["", "[]"].includes(data.stdout.trim())) return false;
+  const name = resource.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
+  return new RegExp(`^(?:Error(?: response from daemon)?: )?(?:No such (?:object|container|volume):? ${name}|get ${name}: no such volume)\\s*$`, "i").test(data.stderr.trim());
+}
+export const runDocker: DockerRun = async (args, timeout = 15000) => {
+  try {
+    const result = await promisify(execFile)("docker", args, { timeout, maxBuffer: 2 * 1024 * 1024,
+      env: { PATH: process.env.PATH ?? "", HOME: process.env.HOME ?? "", ...(process.env.DOCKER_CONFIG ? { DOCKER_CONFIG: process.env.DOCKER_CONFIG } : {}) } });
+    return (args.includes("logs") ? result.stdout + result.stderr : result.stdout).trim();
+  } catch (cause) {
+    const raw = cause as { stdout?: string; stderr?: string; code?: number; signal?: string };
+    throw new DockerCommandError({ args, stdout: raw.stdout ?? "", stderr: raw.stderr ?? "", exitCode: typeof raw.code === "number" ? raw.code : null, signal: raw.signal ?? null });
+  }
+};
+export async function currentDockerEngine(run: DockerRun = runDocker, env = process.env): Promise<DockerEngineIdentity> {
+  if (env.DOCKER_HOST && !env.DOCKER_HOST.startsWith("unix://")) throw new Error("Remote Docker hosts are not admitted");
+  const context = env.DOCKER_CONTEXT || await run(["context", "show"]);
+  if (!/^[A-Za-z0-9_.-]{1,100}$/.test(context)) throw new Error("Invalid local Docker context identity");
+  const rows = JSON.parse(await run(["context", "inspect", context]));
+  const endpoint = rows?.[0]?.Endpoints?.docker?.Host;
+  if (rows.length !== 1 || typeof endpoint !== "string" || !endpoint.startsWith("unix:///") || endpoint.includes("\n") || env.DOCKER_HOST && env.DOCKER_HOST !== endpoint) throw new Error("Docker context must resolve to the exact local Unix socket");
+  const info = JSON.parse(await run(["--context", context, "info", "--format", '{"id":{{json .ID}},"os":{{json .OSType}}}']));
+  if (info.os !== "linux" || typeof info.id !== "string" || !info.id.trim()) throw new Error("A running local Linux Docker engine is required");
+  return { context, endpoint, engineId: info.id };
+}
+async function pinnedEngine(expected: DockerEngineIdentity, run: DockerRun) {
+  const observed = await currentDockerEngine(run);
+  if (JSON.stringify(observed) !== JSON.stringify(expected)) throw new Error("Docker context or engine changed; owned operations refused");
+}
+function labels(receipt: DockerDatabaseReceipt) { return { [`${labelPrefix}.kind`]: "postgres", [`${labelPrefix}.nonce`]: receipt.nonce, [`${labelPrefix}.root`]: hash(receipt.temporaryRoot), [`${labelPrefix}.uid`]: String(process.getuid?.() ?? "unknown") }; }
+function hasLabels(actual: Record<string, string> | undefined, receipt: DockerDatabaseReceipt) { return !!actual && Object.entries(labels(receipt)).every(([k, v]) => actual[k] === v); }
+export function assertOwnedVolume(value: any, receipt: DockerDatabaseReceipt) {
+  if (!value || value.Name !== receipt.volumeName || value.Driver !== "local" || value.Scope !== "local" || Object.keys(value.Options ?? {}).length || !hasLabels(value.Labels, receipt)) throw new Error("Docker volume ownership/configuration mismatch");
+}
+export function assertOwnedContainer(value: any, receipt: DockerDatabaseReceipt, running?: boolean) {
+  const mounts = value?.Mounts, data = mounts?.find((m: any) => m.Destination === "/var/lib/postgresql"), password = mounts?.find((m: any) => m.Destination === "/run/secrets/postgres-password");
+  const ports = value?.HostConfig?.PortBindings?.["5432/tcp"], config = value?.Config;
+  if (!fullId(receipt.containerId) || value?.Id !== receipt.containerId || value.Name !== `/${receipt.containerName}` || value.Image !== POSTGRES_IMAGE_ID || config?.Image !== POSTGRES_IMAGE || !hasLabels(config.Labels, receipt)
+    || value.HostConfig.Privileged !== false || !["default", "bridge"].includes(value.HostConfig.NetworkMode) || !Array.isArray(mounts) || mounts.length !== 2
+    || data?.Type !== "volume" || data.Name !== receipt.volumeName || data.RW !== true || password?.Type !== "bind" || password.Source !== path.join(receipt.temporaryRoot, "docker-postgres-private", "password") || password.RW !== false
+    || !Array.isArray(ports) || ports.length !== 1 || ports[0].HostIp !== "127.0.0.1" || Object.keys(value.HostConfig.PortBindings).join() !== "5432/tcp"
+    || !config.Env.includes("POSTGRES_PASSWORD_FILE=/run/secrets/postgres-password") || !config.Env.includes(`POSTGRES_USER=${receipt.username}`) || !config.Env.includes(`POSTGRES_DB=${receipt.database}`) || config.Env.some((v: string) => /^POSTGRES_PASSWORD=/.test(v))
+    || running !== undefined && value.State?.Running !== running) throw new Error("Docker container ownership/isolation mismatch");
+  if (receipt.port !== undefined && value.State?.Running) {
+    const mapped = value.NetworkSettings?.Ports?.["5432/tcp"];
+    if (!Array.isArray(mapped) || mapped.length !== 1 || mapped[0].HostIp !== "127.0.0.1" || Number(mapped[0].HostPort) !== receipt.port) throw new Error("Docker loopback database mapping changed");
+  }
+}
+async function privateRoot(root: string) {
+  const s = await lstat(root);
+  if (!path.basename(root).startsWith("paperclip-runner-e2e-") || await realpath(root) !== root || !s.isDirectory() || s.mode & 0o077 || s.uid !== process.getuid?.()) throw new Error("Docker database requires the exact owned private attempt root");
+}
+function assertReceipt(value: DockerDatabaseReceipt, root: string) {
+  if (value?.schemaVersion !== 1 || value.lane !== "owned-docker-postgres" || value.temporaryRoot !== root || !uuidHex(value.nonce) || value.containerName !== `paperclip-e2e-pg-${value.nonce}` || value.volumeName !== `paperclip-e2e-pg-data-${value.nonce}`
+    || value.image !== POSTGRES_IMAGE || value.imageId !== POSTGRES_IMAGE_ID || value.imageVersion !== POSTGRES_VERSION || !/^[a-z][a-z0-9_]{1,60}$/.test(value.username) || !/^[a-z][a-z0-9_]{1,60}$/.test(value.database)) throw new Error("Invalid private Docker database receipt");
+}
+async function readPrivateDockerReceipt(filename: string, root: string) {
+  await privateRoot(root);
+  if (filename !== path.join(root, "docker-postgres-private", "receipt.json")) throw new Error("Docker receipt escaped its owned attempt");
+  const dir = await lstat(path.dirname(filename)), s = await lstat(filename);
+  if (!dir.isDirectory() || await realpath(path.dirname(filename)) !== path.dirname(filename) || dir.mode & 0o077 || dir.uid !== process.getuid?.() || !s.isFile() || s.isSymbolicLink() || (s.mode & 0o777) !== 0o600 || s.uid !== process.getuid?.() || s.size > 32768) throw new Error("Unsafe private Docker receipt");
+  const receipt = JSON.parse(await readFile(filename, "utf8")) as DockerDatabaseReceipt; assertReceipt(receipt, root);
+  return receipt;
+}
+export async function readDockerDatabaseReceipt(filename: string, root: string, run: DockerRun = runDocker) {
+  const receipt = await readPrivateDockerReceipt(filename, root);
+  if (receipt.status !== "ready" || !receipt.connectionString || !receipt.port || !fullId(receipt.containerId)) throw new Error("Owned Docker database is not ready");
+  const url = new URL(receipt.connectionString);
+  if (url.protocol !== "postgresql:" || url.hostname !== "127.0.0.1" || url.port !== String(receipt.port) || decodeURIComponent(url.username) !== receipt.username || url.pathname !== `/${receipt.database}` || !/^[a-f0-9]{64}$/.test(url.password) || url.search || url.hash) throw new Error("Docker database URL differs from owned receipt");
+  const passwordPath = path.join(root, "docker-postgres-private", "password"), passwordStat = await lstat(passwordPath);
+  if (!passwordStat.isFile() || passwordStat.isSymbolicLink() || (passwordStat.mode & 0o777) !== 0o600 || passwordStat.uid !== process.getuid?.() || await readFile(passwordPath, "utf8") !== url.password) throw new Error("Docker private password binding differs from receipt");
+  await pinnedEngine(receipt.engine, run);
+  assertOwnedContainer(JSON.parse(await run(["--context", receipt.engine.context, "inspect", receipt.containerId]))[0], receipt, true);
+  assertOwnedVolume(JSON.parse(await run(["--context", receipt.engine.context, "volume", "inspect", receipt.volumeName]))[0], receipt);
+  return receipt;
+}
+export interface DockerDatabaseIdentity {
+  lane: "owned-docker-postgres"; image: typeof POSTGRES_IMAGE; imageId: typeof POSTGRES_IMAGE_ID; imageVersion: typeof POSTGRES_VERSION;
+  artifactVerified: boolean; contextSha256: string | null; engineSha256: string | null; containerId: string | null;
+  cluster?: { version: string; systemIdentifier: string }; status: DockerDatabaseReceipt["status"] | "startup-unconfirmed";
+}
+export function publicDockerDatabaseIdentity(receipt: DockerDatabaseReceipt): DockerDatabaseIdentity { return { lane: receipt.lane, image: receipt.image, imageId: receipt.imageId, imageVersion: receipt.imageVersion, artifactVerified: receipt.artifactVerified === true, contextSha256: hash(`${receipt.engine.context}\n${receipt.engine.endpoint}`), engineSha256: hash(receipt.engine.engineId), containerId: receipt.containerId, cluster: receipt.cluster ? { version: receipt.cluster.version, systemIdentifier: receipt.cluster.systemIdentifier } : undefined, status: receipt.status }; }
+/** Also used when startup throws before returning a ready handle. Never confuse
+ * an unavailable receipt/daemon with proven absence or embedded fallback. */
+export async function dockerDatabaseLaunchState(lane: string, root: string) {
+  if (lane === "embedded-postgres") return { allowDarwinReap: true, cleanupUnconfirmed: false, identity: undefined };
+  const unknown: DockerDatabaseIdentity = { lane: "owned-docker-postgres", image: POSTGRES_IMAGE, imageId: POSTGRES_IMAGE_ID, imageVersion: POSTGRES_VERSION, artifactVerified: false, contextSha256: null, engineSha256: null, containerId: null, status: "startup-unconfirmed" };
+  try {
+    const receipt = await readPrivateDockerReceipt(path.join(root, "docker-postgres-private", "receipt.json"), root);
+    return { allowDarwinReap: false, cleanupUnconfirmed: receipt.status !== "removed" || !!receipt.cleanupFailures?.length, identity: publicDockerDatabaseIdentity(receipt) };
+  } catch { return { allowDarwinReap: false, cleanupUnconfirmed: true, identity: unknown }; }
+}
+export async function startOwnedDockerDatabase(input: { temporaryRoot: string; privateSecrets: string[]; run?: DockerRun }) {
+  const rawRun = input.run ?? runDocker, root = input.temporaryRoot; await privateRoot(root);
+  const startupDeadline = Date.now() + 120000; let cleaning = false;
+  const run: DockerRun = (args, timeout = 15000) => {
+    const remaining = cleaning ? timeout : Math.min(timeout, startupDeadline - Date.now());
+    if (remaining <= 0) throw new Error("Owned Docker startup deadline expired");
+    return rawRun(args, remaining);
+  };
+  const engine = await currentDockerEngine(run), nonce = randomBytes(16).toString("hex"), password = randomBytes(32).toString("hex"), directory = path.join(root, "docker-postgres-private"), filename = path.join(directory, "receipt.json");
+  await mkdir(directory, { mode: 0o700 });
+  const receipt: DockerDatabaseReceipt = { schemaVersion: 1, lane: "owned-docker-postgres", temporaryRoot: root, nonce, engine, containerId: null, containerName: `paperclip-e2e-pg-${nonce}`, volumeName: `paperclip-e2e-pg-data-${nonce}`, image: POSTGRES_IMAGE, imageId: POSTGRES_IMAGE_ID, imageVersion: POSTGRES_VERSION, username: `pc_${nonce.slice(0, 20)}`, database: `pc_${nonce.slice(0, 20)}`, status: "creating", startedAt: new Date().toISOString() };
+  input.privateSecrets.push(password); await writeFile(path.join(directory, "password"), password, { mode: 0o600, flag: "wx" });
+  const persist = () => writeFile(filename, `${JSON.stringify(receipt, null, 2)}\n`, { mode: 0o600 }); await persist();
+  const command = async (args: string[], timeout?: number) => { await pinnedEngine(engine, run); return run(["--context", engine.context, ...args], timeout); };
+  let closed = false;
+  const close = async () => {
+    if (closed) return; cleaning = true;
+    const failures: string[] = [];
+    let containerAbsent = !receipt.containerCreationAttempted;
+    if (receipt.containerCreationAttempted) {
+      const target = receipt.containerId ?? receipt.containerName;
+      try {
+        const value = JSON.parse(await command(["inspect", target]))[0];
+        const recovered = { ...receipt, containerId: receipt.containerId ?? value?.Id };
+        assertOwnedContainer(value, recovered); receipt.containerId = recovered.containerId; await persist();
+        if (value.State.Running) await command(["stop", "--time", "10", receipt.containerId!], 30000);
+        assertOwnedContainer(JSON.parse(await command(["inspect", receipt.containerId!]))[0], receipt, false);
+        await command(["rm", receipt.containerId!]);
+        // A successful rm acknowledgement is not the absence proof.
+        try { await command(["inspect", receipt.containerId!]); throw new Error("Owned container remained after removal"); }
+        catch (error) { if (!verifiedNotFound(error, receipt.containerId!)) throw error; }
+        containerAbsent = true;
+      } catch (error) {
+        if (verifiedNotFound(error, target) || receipt.containerId && verifiedNotFound(error, receipt.containerId)) containerAbsent = true;
+        else failures.push("Exact owned container absence is unconfirmed");
+      }
+    }
+    if (receipt.volumeCreationAttempted) {
+      if (!containerAbsent) failures.push("Owned volume retained because container absence is unconfirmed");
+      else try {
+        assertOwnedVolume(JSON.parse(await command(["volume", "inspect", receipt.volumeName]))[0], receipt);
+        await command(["volume", "rm", receipt.volumeName]);
+        try { await command(["volume", "inspect", receipt.volumeName]); throw new Error("Owned volume remained after removal"); }
+        catch (error) { if (!verifiedNotFound(error, receipt.volumeName)) throw error; }
+      } catch (error) { if (!verifiedNotFound(error, receipt.volumeName)) failures.push("Exact owned volume absence is unconfirmed"); }
+    }
+    receipt.status = failures.length ? "cleanup-failed" : "removed"; receipt.cleanupFailures = failures; receipt.finishedAt = new Date().toISOString(); await persist();
+    if (failures.length) throw new Error(failures.join("; "));
+    closed = true; await rm(path.join(directory, "password"), { force: true });
+  };
+  try {
+    const image = JSON.parse(await command(["image", "inspect", POSTGRES_IMAGE]))[0];
+    if (image?.Id !== POSTGRES_IMAGE_ID || image.Os !== "linux" || image.Architecture !== "arm64" || !image.RepoDigests?.includes(POSTGRES_IMAGE) || !image.Config.Env.includes(`PG_VERSION=${POSTGRES_VERSION}`)) throw new Error("Installed official PostgreSQL artifact differs from pinned identity; no pull permitted");
+    receipt.artifactVerified = true; await persist();
+    const labelArgs = Object.entries(labels(receipt)).flatMap(([k, v]) => ["--label", `${k}=${v}`]);
+    receipt.volumeCreationAttempted = true; await persist();
+    await command(["volume", "create", "--driver", "local", ...labelArgs, receipt.volumeName]);
+    assertOwnedVolume(JSON.parse(await command(["volume", "inspect", receipt.volumeName]))[0], receipt);
+    receipt.containerCreationAttempted = true; await persist();
+    const cid = await command(["create", "--cidfile", path.join(directory, "container.id"), "--pull=never", "--name", receipt.containerName, ...labelArgs, "--publish", "127.0.0.1::5432/tcp", "--memory", "1g", "--cpus", "2", "--pids-limit", "256", "--shm-size", "128m", "--mount", `type=volume,src=${receipt.volumeName},dst=/var/lib/postgresql`, "--mount", `type=bind,src=${path.join(directory, "password")},dst=/run/secrets/postgres-password,readonly`, "--env", "POSTGRES_PASSWORD_FILE=/run/secrets/postgres-password", "--env", `POSTGRES_USER=${receipt.username}`, "--env", `POSTGRES_DB=${receipt.database}`, POSTGRES_IMAGE]);
+    if (!fullId(cid)) throw new Error("Docker create returned no exact container ID"); receipt.containerId = cid; await persist();
+    assertOwnedContainer(JSON.parse(await command(["inspect", cid]))[0], receipt, false); await command(["start", cid], 20000);
+    const deadline = Date.now() + 60000; let ready = false;
+    while (Date.now() < deadline) { const value = JSON.parse(await command(["inspect", cid]))[0]; assertOwnedContainer(value, receipt); if (!value.State.Running) throw new Error("Owned PostgreSQL container stopped during initialization"); try { await command(["exec", cid, "pg_isready", "-h", "127.0.0.1", "-U", receipt.username, "-d", receipt.database], 5000); ready = true; break; } catch { await new Promise(resolve => setTimeout(resolve, 250)); } }
+    if (!ready) throw new Error("Owned PostgreSQL readiness deadline expired");
+    const observed = JSON.parse(await command(["inspect", cid]))[0], mapped = observed.NetworkSettings?.Ports?.["5432/tcp"];
+    if (!Array.isArray(mapped) || mapped.length !== 1 || mapped[0].HostIp !== "127.0.0.1" || !/^[0-9]+$/.test(mapped[0].HostPort) || Number(mapped[0].HostPort) < 1024 || Number(mapped[0].HostPort) > 65535) throw new Error("Owned PostgreSQL published port is not private loopback");
+    receipt.port = Number(mapped[0].HostPort); receipt.connectionString = `postgresql://${receipt.username}:${password}@127.0.0.1:${receipt.port}/${receipt.database}`; input.privateSecrets.push(receipt.connectionString);
+    const cluster = JSON.parse(await command(["exec", cid, "psql", "-X", "-U", receipt.username, "-d", receipt.database, "-At", "-c", "SELECT json_build_object('version',current_setting('server_version'),'database',current_database(),'username',current_user,'systemIdentifier',(SELECT system_identifier::text FROM pg_control_system()))"]));
+    if (!/^18\.6(?:\s|$)/.test(cluster.version) || cluster.database !== receipt.database || cluster.username !== receipt.username || !/^[0-9]+$/.test(cluster.systemIdentifier)) throw new Error("Owned PostgreSQL cluster identity/version mismatch");
+    receipt.cluster = cluster; receipt.status = "ready"; await persist(); await readDockerDatabaseReceipt(filename, root, run);
+    return { filename, receipt, verify: () => readDockerDatabaseReceipt(filename, root, run), close };
+  } catch (error) {
+    receipt.status = "failed"; receipt.primaryFailure = error instanceof Error ? error.message : "Owned Docker PostgreSQL startup failed";
+    const failure = error instanceof Error ? commandFailures.get(error) : undefined;
+    if (failure) await writeFile(path.join(directory, "startup-command-failure.json"), JSON.stringify(failure), { mode: 0o600 });
+    if (receipt.containerId) try { await writeFile(path.join(directory, "postgres.log"), await command(["logs", "--tail", "200", receipt.containerId]), { mode: 0o600 }); } catch { /* Preserve primary failure even if logs unavailable. */ }
+    await persist(); try { await close(); } catch { throw new Error("Owned Docker PostgreSQL startup failed and cleanup is unconfirmed; private receipt retained"); }
+    throw new Error("Owned Docker PostgreSQL startup failed; primary failure and diagnostics retained privately");
+  }
+}
+
+/** Supervisor consumes the private receipt, then removes both admission keys
+ * before constructing the actual Paperclip/native environment. */
+export async function dockerDatabaseSupervisorEnvironment(input: { temporaryRoot: string; env: NodeJS.ProcessEnv; run?: DockerRun }) {
+  const lane = selectedDatabaseLane(input.env), source = { ...input.env }, filename = source[DOCKER_DATABASE_ENV];
+  delete source[DOCKER_DATABASE_ENV]; delete source[DATABASE_LANE_ENV];
+  if (lane === "embedded-postgres") { if (filename) throw new Error("Undeclared Docker database receipt"); return { source, filename: undefined }; }
+  if (!filename) throw new Error("Selected Docker database lane is missing its owned receipt");
+  await readDockerDatabaseReceipt(filename, input.temporaryRoot, input.run);
+  for (const key of ["DOCKER_CONFIG", "DOCKER_HOST", "DOCKER_CONTEXT"]) delete source[key];
+  return { source, filename };
+}
+
+export function databaseFingerprintIdentity(env = process.env) {
+  return selectedDatabaseLane(env) === "owned-docker-postgres"
+    ? { lane: "owned-docker-postgres" as const, image: POSTGRES_IMAGE, imageId: POSTGRES_IMAGE_ID, version: POSTGRES_VERSION }
+    : { lane: "embedded-postgres" as const };
+}
+export type DatabaseFingerprintIdentity = ReturnType<typeof databaseFingerprintIdentity>;
+export function databaseIdentityMatches(expected: DatabaseFingerprintIdentity, actual?: DockerDatabaseIdentity) {
+  if (expected.lane === "embedded-postgres") return actual === undefined;
+  return !!actual && actual.lane === expected.lane && actual.image === expected.image && actual.imageId === expected.imageId && actual.imageVersion === expected.version
+    && actual.artifactVerified && !!actual.contextSha256 && !!actual.engineSha256 && fullId(actual.containerId) && !!actual.cluster
+    && /^18\.6(?:\s|$)/.test(actual.cluster.version) && /^[0-9]+$/.test(actual.cluster.systemIdentifier) && ["ready", "removed"].includes(actual.status);
+}

tests/runner-e2e/instance-isolation.ts

@@ -1,9 +1,12 @@
+import { readDockerDatabaseReceipt, type DockerRun } from "./docker-postgres.js";
 import path from "node:path";
 import { readFile, stat } from "node:fs/promises";
 
-export async function assertEmbeddedDatabaseIsolation(
+export async function assertRunnerDatabaseIsolation(
   configPath: string,
   temporaryRoot: string,
+  dockerReceiptPath?: string,
+  dockerRun?: DockerRun,
 ) {
   const configText = await readFile(configPath, "utf8");
   const config = JSON.parse(configText) as {
@@ -24,13 +27,9 @@ export async function assertEmbeddedDatabaseIsolation(
       localEncrypted?: { keyFilePath?: string };
     };
   };
-  if (
-    config.database?.mode !== "embedded-postgres" ||
-    config.database.connectionString
-  ) {
-    throw new Error(
-      "Runner E2E Paperclip instance did not use its embedded database",
-    );
+  const docker = dockerReceiptPath ? await readDockerDatabaseReceipt(dockerReceiptPath, temporaryRoot, dockerRun) : undefined;
+  if (docker ? config.database?.mode !== "postgres" || config.database.connectionString !== docker.connectionString : config.database?.mode !== "embedded-postgres" || !!config.database.connectionString) {
+    throw new Error("Runner E2E Paperclip instance did not use its explicitly owned database lane");
   }
   if (
     config.storage?.provider !== "local_disk" ||
@@ -42,8 +41,8 @@ export async function assertEmbeddedDatabaseIsolation(
     );
   }
   const isolatedPaths = {
-    database: config.database.embeddedPostgresDataDir,
-    backup: config.database.backup?.dir,
+    ...(docker ? {} : { database: config.database!.embeddedPostgresDataDir }),
+    backup: config.database!.backup?.dir,
     logs: config.logging?.logDir,
     storage: config.storage.localDisk?.baseDir,
     secretsKey: config.secrets.localEncrypted?.keyFilePath,
@@ -59,11 +58,15 @@ export async function assertEmbeddedDatabaseIsolation(
       );
     }
   }
-  const databasePath = path.resolve(isolatedPaths.database!);
-  const databaseStat = await stat(databasePath);
-  if (!databaseStat.isDirectory())
-    throw new Error("Embedded database path is not a directory");
+  if (!docker) {
+    const databasePath = path.resolve(isolatedPaths.database!);
+    const databaseStat = await stat(databasePath);
+    if (!databaseStat.isDirectory()) throw new Error("Embedded database path is not a directory");
+  }
   const keyStat = await stat(path.resolve(isolatedPaths.secretsKey!));
   if (!keyStat.isFile())
     throw new Error("Encrypted-secrets master key path is not a file");
 }
+
+/** Compatibility entry point remains embedded-only, never ambient external DB. */
+export async function assertEmbeddedDatabaseIsolation(configPath: string, temporaryRoot: string) { return assertRunnerDatabaseIsolation(configPath, temporaryRoot); }

tests/runner-e2e/launch.ts

@@ -1,3 +1,4 @@
+import { DATABASE_LANE_ENV, DOCKER_DATABASE_ENV, dockerDatabaseLaunchState, publicDockerDatabaseIdentity, selectedDatabaseLane, startOwnedDockerDatabase } from "./docker-postgres.js";
 import { ARCHIVE_CONTROL_ENV, ARCHIVE_SCENARIO, startArchiveFaultControl } from "./chat-archive-fault-control.js";
 import { abnormalFinalizationResult } from "./abnormal-finalization.js";
 import { CHAT_ORACLE_KEY_ENV } from "./chat-private-oracle.js";
@@ -51,7 +52,7 @@ import {
   resolvePaperclipRemoteRunnerBinaryForHarness,
   resolvePaperclipRunnerBinaryForHarness,
 } from "./harness-env.js";
-import { assertEmbeddedDatabaseIsolation } from "./instance-isolation.js";
+import { assertRunnerDatabaseIsolation } from "./instance-isolation.js";
 import {
   assertSecretFree,
   findSecretLeakInDirectory,
@@ -501,6 +502,7 @@ async function runAttempt(input: {
   const selectedDescriptor = execution.suite.id === "chat-features" ? findChatCatalogCase(execution.task.id) : undefined;
   let sourceEnvironment = { ...process.env };
   delete sourceEnvironment[CHAT_ORACLE_KEY_ENV];
+  const databaseLane = selectedDatabaseLane(sourceEnvironment);
   const installedPlugin = await verifyInstalledDaytonaPlugin(process.env, executions);
   const startedAtMs = Date.now();
   const sharedMemoryBaseline = snapshotDarwinSharedMemory();
@@ -512,6 +514,7 @@ async function runAttempt(input: {
   let attemptSecrets: string[] = [];
   let processCleanupFailed = false;
   let startupReceiptPath: string | undefined;
+  let dockerDatabase: Awaited<ReturnType<typeof startOwnedDockerDatabase>> | undefined;
   let archiveControl: Awaited<ReturnType<typeof startArchiveFaultControl>> | undefined;
   let gateway: Awaited<ReturnType<EvalOperations["start"]>> | undefined;
   const rawCleanupResults: Array<{ cleanup: string; synthetic: boolean; status: string }> = [];
@@ -566,6 +569,7 @@ async function runAttempt(input: {
     await mkdir(selectedPrivateInputsDirectory, { mode: 0o700 });
     const privateInputs = needsPrivateInputs ? stageChatPrivateInputs({ execution, descriptor: selectedDescriptor, privateDir: selectedPrivateInputsDirectory, env: process.env, secrets: credentials }) : undefined;
     if (selectedDescriptor?.scenarioId === ARCHIVE_SCENARIO) archiveControl = await startArchiveFaultControl({ selected: executions.map(row => row.id), temporaryRoot, backendOrigin: `http://127.0.0.1:${port}`, privateSecrets: credentials });
+    if (databaseLane === "owned-docker-postgres") dockerDatabase = await startOwnedDockerDatabase({ temporaryRoot, privateSecrets: credentials });
     const diagnosticRegistry = stagePrivateDiagnostics(temporaryRoot, executions.map(row => row.id), credentials);
     if (installedCli) await writeFile(path.join(privateDir, "installed-cli-admission.json"), `${JSON.stringify(installedCli, null, 2)}\n`, { mode: 0o600 });
     if (installedPlugin) await writeFile(path.join(privateDir, "installed-daytona-plugin-admission.json"), `${JSON.stringify(installedPlugin, null, 2)}\n`, { mode: 0o600 });
@@ -577,6 +581,8 @@ async function runAttempt(input: {
       ...buildRunnerE2EProcessEnvironment(sourceEnvironment, executions),
       [CHAT_PRIVATE_INPUTS_ENV]: privateInputs?.filename,
       [ARCHIVE_CONTROL_ENV]: archiveControl?.filename,
+      [DATABASE_LANE_ENV]: dockerDatabase ? "owned-docker-postgres" : undefined,
+      [DOCKER_DATABASE_ENV]: dockerDatabase?.filename,
       [PRIVATE_DIAGNOSTICS_ENV]: diagnosticRegistry,
       PATH: providerPath,
       PAPERCLIP_RUNNER_E2E_EXECUTION_IDS: JSON.stringify(
@@ -662,7 +668,7 @@ async function runAttempt(input: {
       await cp(privateDir, path.join(proofDir, "private"), { recursive: true });
       const probe = JSON.parse(await readFile(path.join(privateDir, "installed-startup-only.json"), "utf8"));
       if (processResult.exitCode !== 0 || processResult.timedOut || processResult.spawnError || processCleanupFailed || probe.status !== "health_ui_zero_company_passed" || probe.providerCalls !== 0 || probe.qualified !== false) throw new Error("Installed startup-only actual launch chain failed");
-      await assertEmbeddedDatabaseIsolation(configPath, temporaryRoot);
+      await assertRunnerDatabaseIsolation(configPath, temporaryRoot, dockerDatabase?.filename);
       startupReceiptPath = path.join(proofDir, "receipt.json");
       await writeFile(startupReceiptPath, `${JSON.stringify({ status: "installed_launch_chain_pending_scratch_cleanup", qualified: false, providerCalls: 0, credentialsLoaded: false, installedCli, processResult, probe }, null, 2)}\n`, { mode: 0o600 });
       return [];
@@ -698,6 +704,7 @@ async function runAttempt(input: {
           processFailureClass,
         );
         const result = await readResult(resultPath, fallback);
+        if (databaseLane === "owned-docker-postgres") result.databaseIdentity = dockerDatabase ? publicDockerDatabaseIdentity(dockerDatabase.receipt) : (await dockerDatabaseLaunchState(databaseLane, temporaryRoot)).identity;
         if (!result.workerTiming) result.workerTiming = await recoverWorkerTiming(path.join(path.dirname(resultPath), "snapshots", "worker-timing.json"),
           { executionId: candidate.id, attempt, startedAt: result.startedAt, finishedAt: result.finishedAt });
         if (result === fallback) {
@@ -717,7 +724,7 @@ async function runAttempt(input: {
     );
     let isolationError: unknown;
     try {
-      await assertEmbeddedDatabaseIsolation(configPath, temporaryRoot);
+      await assertRunnerDatabaseIsolation(configPath, temporaryRoot, dockerDatabase?.filename);
     } catch (error) {
       isolationError = error;
     }
@@ -915,6 +922,7 @@ async function runAttempt(input: {
       const fallback = syntheticResult(candidate, attempt, startedAtMs, "Attempt terminated before a worker result was retained");
       const original = await readResult(path.join(casePrivateDir, "result.json"), fallback);
       const result = abnormalFinalizationResult(original, error);
+      if (databaseLane === "owned-docker-postgres") result.databaseIdentity = dockerDatabase ? publicDockerDatabaseIdentity(dockerDatabase.receipt) : (await dockerDatabaseLaunchState(databaseLane, temporaryRoot)).identity;
       const uploadDir = path.join(resultsRoot, campaignId, candidate.suite.id, candidate.profile.id, candidate.environment.id, candidate.task.id, `attempt-${attempt}`);
       await mkdir(uploadDir, { recursive: true });
       const resultPath = path.join(uploadDir, "result.json");
@@ -933,9 +941,14 @@ async function runAttempt(input: {
     // runProcess has completed its owned Playwright/server process shutdown.
     let archiveCleanupError: unknown;
     try { await archiveControl?.close(); } catch (error) { archiveCleanupError = error; processCleanupFailed = true; }
+    let dockerCleanupError: unknown;
+    try { await dockerDatabase?.close(); } catch (error) { dockerCleanupError = error; processCleanupFailed = true; }
+    const databaseFinalization = await dockerDatabaseLaunchState(databaseLane, temporaryRoot);
+    if (databaseFinalization.cleanupUnconfirmed) processCleanupFailed = true;
+    if (databaseFinalization.identity) for (const result of publishedResults) result.databaseIdentity = databaseFinalization.identity;
     let gatewayCleanupError: unknown;
     try { await gateway?.close(); } catch (error) { gatewayCleanupError = error; }
-    if (!processCleanupFailed) reapNewDetachedDarwinSharedMemory(sharedMemoryBaseline);
+    if (!processCleanupFailed && databaseFinalization.allowDarwinReap) reapNewDetachedDarwinSharedMemory(sharedMemoryBaseline);
     let cleanupError: unknown;
     // Raw selected keys are never diagnostic/archive artifacts, even when recovery state is retained.
     try { await rm(selectedPrivateInputsDirectory, { recursive: true, force: true }); }
@@ -979,7 +992,7 @@ async function runAttempt(input: {
         `Refusing to remove unexpected temporary path ${temporaryRoot}`,
       );
     }
-    if (gatewayCleanupError || archiveCleanupError) cleanupError = new AggregateError([cleanupError, gatewayCleanupError, archiveCleanupError].filter(Boolean), "Eval owned gateway/transport cleanup failed");
+    if (gatewayCleanupError || archiveCleanupError || dockerCleanupError) cleanupError = new AggregateError([cleanupError, gatewayCleanupError, archiveCleanupError, dockerCleanupError].filter(Boolean), "Eval owned gateway/transport cleanup failed");
     if (startupReceiptPath) {
       const receipt = JSON.parse(await readFile(startupReceiptPath, "utf8"));
       receipt.status = cleanupError ? "installed_launch_chain_cleanup_failed" : "installed_launch_chain_passed";
@@ -1012,6 +1025,11 @@ async function runAttempt(input: {
       console.error(message);
       if (options.installedStartupOnly) throw new Error(message);
     }
+    if (databaseLane === "owned-docker-postgres") for (const result of publishedResults) {
+      const filename = publishedResultPaths.get(result.executionId); if (!filename) continue;
+      const safe = `${JSON.stringify(sanitizeJson(result, attemptSecrets), null, 2)}\n`;
+      assertSecretFree(safe, attemptSecrets, filename); await writeFile(filename, safe, "utf8");
+    }
     for (const result of publishedResults) {
       const resultPath = publishedResultPaths.get(result.executionId);
       const candidate = executions.find(candidate => candidate.id === result.executionId);

tests/runner-e2e/result-validation.ts

@@ -1,3 +1,4 @@
+import { POSTGRES_IMAGE, POSTGRES_IMAGE_ID } from "./docker-postgres.js";
 import { validateWorkerTiming } from "./worker-timing.js";
 import { connectionCheckpoints, connectionEvidencePasses, connectionRunSignals } from "./connection-evidence.js";
 import { githubJourneyCheckpoints, gradeGitHubReviewJourney } from "./github-review-evidence.js";
@@ -225,6 +226,12 @@ const matcher: Rule = (value, at) => {
   rule(value, at);
 };
 const fields = {
+  databaseIdentity: optional(closedShape({
+    lane: oneOf("owned-docker-postgres"), image: oneOf(POSTGRES_IMAGE), imageId: oneOf(POSTGRES_IMAGE_ID), imageVersion: oneOf("18.6-1.pgdg13+2"),
+    artifactVerified: boolean, contextSha256: nullable(matching(/^[a-f0-9]{64}$/)), engineSha256: nullable(matching(/^[a-f0-9]{64}$/)), containerId: nullable(matching(/^[a-f0-9]{64}$/)),
+    cluster: optional(closedShape({ version: matching(/^18\.6(?:\s|$)/), systemIdentifier: matching(/^[0-9]+$/) })),
+    status: oneOf("creating", "ready", "failed", "removed", "cleanup-failed", "startup-unconfirmed"),
+  })),
   accountingIdentity: optional(shape({ campaignId: string, attemptId: string })),
   publicMcp: optional(assistant),
   schema: oneOf(
@@ -403,6 +410,7 @@ export function validateRetainedRunnerResult(
 ): asserts value is RunnerE2EResult {
   result(value, "result");
   const checked = value as RunnerE2EResult;
+  if (checked.databaseIdentity && checked.status === "passed" && (!checked.databaseIdentity.artifactVerified || !checked.databaseIdentity.contextSha256 || !checked.databaseIdentity.engineSha256 || !checked.databaseIdentity.containerId || !checked.databaseIdentity.cluster || !["ready", "removed"].includes(checked.databaseIdentity.status))) invalid("result.databaseIdentity");
   const artifact = checked.productJourney?.artifact;
   if (artifact && (checked.suiteId !== "product-journeys" || checked.caseId !== "cloud-onboarding" ||
     checked.runIds && !checked.runIds.includes(artifact.originatingRunId))) invalid("result.productJourney.artifact");

tests/runner-e2e/server-config.ts

@@ -1,3 +1,4 @@
+import { readDockerDatabaseReceipt, type DockerRun } from "./docker-postgres.js";
 import { mkdir, mkdtemp, realpath, readFile, writeFile } from "node:fs/promises";
 import os from "node:os";
 import path from "node:path";
@@ -16,16 +17,18 @@ export async function prepareRunnerE2EServerConfig(input: {
   temporaryRoot: string;
   configPath: string;
   serverPort: number;
+  dockerReceiptPath?: string;
+  dockerRun?: DockerRun;
 }) {
+  const docker = input.dockerReceiptPath ? await readDockerDatabaseReceipt(input.dockerReceiptPath, input.temporaryRoot, input.dockerRun) : undefined;
   try {
-    await readFile(input.configPath, "utf8");
+    const existing = JSON.parse(await readFile(input.configPath, "utf8"));
+    if (docker ? existing.database?.mode !== "postgres" || existing.database.connectionString !== docker.connectionString : existing.database?.mode !== "embedded-postgres" || existing.database.connectionString) throw new Error("Existing runner database config differs from its explicit owned lane");
     return null;
   } catch (error) {
     if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;
   }
-  const databaseReservation = await reserveRunnerE2EDatabasePort(
-    input.serverPort,
-  );
+  const databaseReservation = docker ? null : await reserveRunnerE2EDatabasePort(input.serverPort);
   try {
     const instanceRoot = path.dirname(input.configPath);
     const config = paperclipConfigSchema.parse({
@@ -35,9 +38,10 @@ export async function prepareRunnerE2EServerConfig(input: {
         source: "onboard",
       },
       database: {
-        mode: "embedded-postgres",
+        mode: docker ? "postgres" : "embedded-postgres",
+        ...(docker ? { connectionString: docker.connectionString } : {}),
         embeddedPostgresDataDir: path.join(instanceRoot, "db"),
-        embeddedPostgresPort: databaseReservation.port,
+        embeddedPostgresPort: databaseReservation?.port ?? 54329,
         backup: {
           enabled: false,
           dir: path.join(input.temporaryRoot, "backups"),
@@ -71,7 +75,7 @@ export async function prepareRunnerE2EServerConfig(input: {
     });
     return databaseReservation;
   } catch (error) {
-    await databaseReservation.close();
+    await databaseReservation?.close();
     throw error;
   }
 }

tests/runner-e2e/server.ts

@@ -1,3 +1,4 @@
+import { dockerDatabaseSupervisorEnvironment } from "./docker-postgres.js";
 import { archiveServerEnvironment } from "./chat-archive-fault-control.js";
 import { CHAT_INSTALL_PLAN, refreshInstalledChatPins, publishInstalledChatReceipt } from "./chat-installed-lane.js";
 import { parseFinalizationCrashBinding, readFinalizationCrashBoundary, type FinalizationCrashBinding } from "./finalization-crash-boundary.js";
@@ -52,7 +53,8 @@ const restartTimeoutMs = 180_000;
 const gracefulStopTimeoutMs = 30_000;
 const installedRelease = process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI
   ? installedReleaseLaunch(process.env.PAPERCLIP_RUNNER_E2E_INSTALLED_CLI) : null;
-const archiveEnvironment = await archiveServerEnvironment({ selected: executionIds, temporaryRoot, backendOrigin: `http://127.0.0.1:${port}`, env: process.env });
+const dockerEnvironment = await dockerDatabaseSupervisorEnvironment({ temporaryRoot, env: process.env });
+const archiveEnvironment = await archiveServerEnvironment({ selected: executionIds, temporaryRoot, backendOrigin: `http://127.0.0.1:${port}`, env: dockerEnvironment.source });
 const serverEnvironment = buildPaperclipServerEnvironment(installedRelease
   ? installedReleaseEnvironment(archiveEnvironment.source, repositoryRoot, path.join(temporaryRoot, "provider-bin"))
   : archiveEnvironment.source, {
@@ -416,6 +418,7 @@ async function supervise() {
     temporaryRoot,
     configPath,
     serverPort: Number(port),
+    dockerReceiptPath: dockerEnvironment.filename,
   });
   // Postgres needs the socket itself; release immediately before child spawn.
   await databaseReservation?.close();

tests/runner-e2e/types.ts

@@ -290,6 +290,8 @@ export interface RunnerE2EBillingSummary {
 }
 
 export interface RunnerE2EResult {
+  /** Explicit optional owned Linux/arm64 Docker PG18.6 environment identity. */
+  databaseIdentity?: ReturnType<typeof import("./docker-postgres.js").publicDockerDatabaseIdentity>;
   schema: "paperclip.runner-e2e.result/v1" | "paperclip.runner-e2e.result/v2";
   accountingIdentity?: { campaignId: string; attemptId: string };
   executionId: string;