test: wait for native run settlement and retain recovered failures

tests/e2e/fixtures/native-ingestion-h5.md

@@ -1,10 +1,11 @@
-# H5b independent-agent observation registration (prospective; no execution release)
+# H5c canonical-settlement observation registration (prospective; no execution release)
 
 H4c remains frozen at `7d7e7a88d562a95a1d407b76474763b666c181f5`
 (`5db4129bbdc1fa38775121aff5d0e68a8a11762519e41cfdf36eacd90ba7256b`).
 Its observed 120-second streaming failures remain failures. Aborted or profiled
-attempts cannot be reused as complete timing arms. H5b uses one normal agent per concurrent remote participant. It repairs the
-experiment topology while preserving observation of original deadline outcomes.
+attempts cannot be reused as complete timing arms. H5c keeps one normal agent per
+concurrent remote participant and waits for the product's canonical settlement,
+including supported same-run recovery, within the same terminal deadline.
 It does not optimize the product or turn earlier failures into passes.
 
 The aligned fresh application baselines are B0 `bac53ec3bfeaf6546e7aaeaee516487b30534654`
@@ -16,12 +17,14 @@ or matching binary is presumed before those artifacts exist. The historical
 a365/d8cd/914 application failures remain evidence of their recorded revisions.
 
 Select H5 explicitly with
-`PAPERCLIP_INGESTION_OBSERVATION_MODE=h5b-observe-original-sla-independent-agents`.
-The new default H4d mode uses the same independent-agent geometry with the original
-120/180/900-second deadlines. Observation mode uses the explicit H5b protocol.
+`PAPERCLIP_INGESTION_OBSERVATION_MODE=h5c-observe-original-sla-independent-agents-settlement`.
+The new default H4e mode uses the same independent-agent geometry with the original
+120/180/900-second deadlines. Observation mode uses the explicit H5c protocol.
 Both modes hash the same nine source files, including the shared topology helper.
 Their source revision and measurement bytes must match for later confirmation.
-Old H4c/H5a protocols, geometry, arms and noise are ineligible for H5b.
+Old H4c/H5a protocols, geometry, arms and noise are ineligible for H5c. H5b179fe6
+and its H4d default proof use the premature terminal-status observer and are also
+ineligible. The new H4e proof must use the exact H5c source and geometry.
 
 Run `node scripts/describe-native-ingestion-h5.mjs` to inspect the registration.
 This is a dry run only: it starts no app, provider, database or browser and
@@ -77,7 +80,86 @@ A separate background presentation40P01 involved a run that succeeded. The
 retained log shows its coordinator→issue→run wait, but lacks PostgreSQL's full
 opposing lock cycle. Its cause and relationship to other contention are unproved.
 It is not evidence that the14 run failures came from a database deadlock. All
-historical failures remain separate; no failed H4c/H5a arm is reused as H5b noise.
+historical failures remain separate; no failed H4c/H5a arm is reused as H5c noise.
+
+## Retained H5b failure and canonical settlement correction
+
+H5b179fe6 on aligned B0bac53 admitted100 distinct normal agents before the common
+epoch. Its original120s poll failed with61/512 foreground markers; all512 became
+visible282.734s after the wait began. The observer stopped54.737s later on83
+succeeded and17 failed statuses. Independent PostgreSQL checks found all51,200
+lossless deltas and one accepted result per run. Qualification failed; history
+and Stop were not reached. This arm is not noise or performance evidence.
+
+All17 errors were provider_transport_failed with the generic message that the
+runner did not durably suspend before checkpoint. Existing run logs instead
+show runnerSuspended=true and providerDrained=false: the exact runner.drain
+receipt was still pending after its five-second drain window. Why the receipt
+was late remains unproved. These were not the earlier shared-agent lock errors.
+All17 finalizers were retryable_failure at attempt1 with a scheduled+30s workspace
+retry; application teardown finished before the earliest retry was due. ACTIVE
+leases and prepared workspace-sync receipts remained unfinished. Later recovery
+was not observed and is not presumed to pass.
+
+The exact B0bac53 run-failure-report.ts44–50 contract states:
+
+> A native run that waits for a same-run retry is stored as `failed` with
+> `nativePhase: "retryable_failure"`. That is not terminal: the retry or the
+> workspace copy-back retry can still finish the run as succeeded.
+
+Normal company-run lists omit
+runtimeMode/nativePhase. Normal GET /api/heartbeat-runs/:id exposes both through
+the existing full run columns. Inside the same single180_000ms terminal poll,
+resolve only failed list candidates through that detail GET and classify its
+current result. Keep failed/native/retryable_failure pending; permanent/unknown
+failures and GET errors reject. The observer sends no retry or provider command.
+
+Record bounded state changes per run with time, status, runtimeMode, nativePhase
+and errorCode. Missing list-only fields are null, not inferred. Cap at16 changes
+per run and fail on overflow rather than discard observations. Retain total
+unique retryable runs, recovered runs, nonretryable failures and per-error-class
+unique-run counts. Every observed retry must recover successfully within
+the unchanged180s window; a deadline miss remains censored failure, never a
+completed latency. All100 successful native runs, exact deltas/results, reload,
+lease cleanup, existing-history and local Stop gates remain required.
+
+After recorded settlement, acknowledgement, lease timing, reload and normal
+event verification, take one company-and-exact-run-set READ ONLY REPEATABLE READ
+snapshot through the existing owned PostgreSQL helper. Retain current error-code
+fields and the closed schema/errorCode/observedAt fields of
+result_json.recoveredExecutionFailure, plus descriptive finalization attempt.
+Require every actual run, exact identities, settled status, valid shapes and a
+closed connection before the next case. No event HTTP or database polling loop
+is added. The existing independent completion upper bound includes this check;
+the measured settlement and lease timestamps stay unchanged.
+
+The durable recovery marker is copied from the run's error fields when successful
+finalization clears them (native-run-projection.ts114–121 in B0bac53). It can
+therefore retain a failure recovered before terminal-window polling. Union its
+codes and current recorded errors with observed transition codes by run identity.
+The normal Stop path's closed cancelled code is retained as cancellation
+evidence (run-control.ts1345); other current Stop error codes remain failures.
+Use these all-recorded unique-ever-failed-run counts/rates for comparison, and
+retain the observed-only counters separately. This marker records the retained
+failure, not an exhaustive sequence of every error class or attempt. Do not claim
+omniscient error history. Finalization attempt is a claim counter that can be
+preserved across workspace repair (native-run-finalizer.ts237–240); it is not a
+failure count and supplies no rate gate.
+
+Successful committed ordinary runs must have both current error fields cleared,
+as required by native-run-projection.ts180–193. A still-current error fails the
+proof; a succeeded status alone cannot turn it into recovered failure evidence.
+
+Fresh A/A retains all histories and freezes, per case, total and per-error-class
+all-recorded unique-ever-failed-run rates plus the maximum absolute paired rate difference. Zero baseline
+counts are valid; no relative division is used. Each candidate may use only
+error classes observed in that baseline A/A. Every paired total/per-class rate
+increase and its median must stay within that frozen absolute noise envelope.
+A single-arm retry storm cannot hide behind the median. Recovery and its final
+backlog remain inside drainage timing; observing recovery does not mean the
+earlier errors disappeared. All recorded failures must have recovered, and every
+run still needs its exact independently persisted accepted result. No product
+retry policy or drain limit is changed.
 
 ## Fixed work and observation boundaries
 
@@ -95,7 +177,7 @@ is added. Fake-plugin streaming remains separate from real-cloud Stop proof.
 | Boundary | Original requirement | H5 observation |
 | --- | --- | --- |
 | Foreground markers | Original Playwright120s wait | Continue to an absolute600s ceiling from that same wait start |
-| All terminal runs |180s after foreground completion (and Stop click) | Same180s after eventual foreground completion; hard failure at the ceiling |
+| Canonically settled runs |180s after foreground completion (and Stop click) | Same single180s after eventual foreground completion; supported retries remain pending, permanent failures reject, deadline misses fail |
 | Stop acknowledgement and leases | Existing30s polls and ordering | Unchanged; full click→acknowledgement metric retained |
 | Invocation | Original900s test budget | Actual1200s observation timeout, separately disclosed |
 
@@ -131,7 +213,7 @@ status, errors and configured timeout. Its observed total duration is an
 independent conservative assessment of the original900s budget; hooks/teardown
 can be treated differently by the original timeout. It is not an exact original
 timer-expiration observation. A final candidate must have total duration<900s
-in every arm and later pass one unchanged H4d default-mode full invocation on the
+in every arm and later pass one unchanged H4e default-mode full invocation on the
 same final SUT revision and full qualified artifact inventory.
 
 ## Complete evidence and eligibility
@@ -167,7 +249,7 @@ not itself attest that the live filesystem stayed unchanged.
 | --- | --- | --- |
 | Fresh B0/B1 A/A arms | Known burst512 streaming120 miss; explicit conservative observed-duration900 miss | Complete finite correct observation; root-reviewed noise suitability |
 | B0→B1 cursor-only candidate | Only an exact registered known miss also present in the paired baseline: burst512 streaming120 or conservative observed-duration900 | No additional deadline failures; latency/control nonregression plus separate incremental SQL evidence; never claim SLA fixed |
-| Final B2 candidate in either comparison | None | Every arm meets original streaming/control checks and conservative900 gate; loaded improvements and later H4d proof |
+| Final B2 candidate in either comparison | None | Every arm meets original streaming/control checks and conservative900 gate; loaded improvements and later H4e proof |
 
 Only burst512's streaming miss is registered as a known workload SLA failure.
 New failures in other workloads require diagnosis. The existing Stop poll
@@ -190,13 +272,13 @@ improve; other fixed latency/control metrics must not regress outside noise.
 Do not infer p99 from sparse observations. There is no new fixed percentage A/A
 cutoff: root must reject unsuitable noisy/nonstationary calibration explicitly.
 
-Use `--observation-mode=h5b-observe-original-sla-independent-agents` for the comparator. Candidate
+Use `--observation-mode=h5c-observe-original-sla-independent-agents-settlement` for the comparator. Candidate
 comparisons also require `--candidate-policy=cursor-nonregression --expect=nonregression`
 or `--candidate-policy=final-b2`. Final comparisons expose `performancePassed`
-separately; `acceptancePassed` remains false until `--original-h4d-proof` and
-`--original-h4d-inventory` establish the later fresh H4d default-deadline functional pass.
-That H4d proof is full, unprofiled, retry0, default mode with original120/180/900s
-bounds, on the same frozen H5b harness source, nine-file measurement bytes and
+separately; `acceptancePassed` remains false until `--original-h4e-proof` and
+`--original-h4e-inventory` establish the later fresh H4e default-deadline functional pass.
+That H4e proof is full, unprofiled, retry0, default mode with original120/180/900s
+bounds, on the same frozen H5c harness source, nine-file measurement bytes and
 independent-agent geometry, and the exact final SUT revision/qualified inventory.
 It uses a new disposable company and starts after all candidate arms. Old7d/H4c
 proofs and H5a observations cannot confirm this geometry. Prior failed H4c attempts remain in the record.
@@ -206,7 +288,7 @@ ordering; no absolute click-to-acknowledgement deadline is added.
 Use the explicit H5 `--require-performance` gate for the two final comparison
 stages while that later proof is pending; this requires every performance and
 candidate original-SLA gate, but grants no final acceptance. After the one later
-H4d default-mode qualification, recompute both reports against their unchanged paired
+H4e default-mode qualification, recompute both reports against their unchanged paired
 manifests with the proof and `--require-acceptance`. This re-evaluates retained
 evidence; it neither reuses an arm in a different comparison nor reruns failures.
 

tests/e2e/fixtures/native-ingestion-h5.mjs

@@ -1,9 +1,9 @@
 import assert from 'node:assert/strict';
 import { createHash } from 'node:crypto';
 
-export const H5_MODE = 'h5b-observe-original-sla-independent-agents';
-export const H5_PROTOCOL = 'v5b-independent-agents-complete-observation-original-sla';
-export const H4D_PROTOCOL = 'v4d-independent-agents-original-deadlines';
+export const H5_MODE = 'h5c-observe-original-sla-independent-agents-settlement';
+export const H5_PROTOCOL = 'v5c-independent-agents-complete-settlement-original-sla';
+export const H4E_PROTOCOL = 'v4e-independent-agents-settlement-original-deadlines';
 export const H5_TOPOLOGY = Object.freeze({ mode: 'one-normal-agent-per-concurrent-participant',
   fullRemoteAgentPoolSize: 100, agentSetupConcurrency: 8, maxConcurrentRuns: 50,
   poolReuse: 'only-across-sequential-cases' });
@@ -12,6 +12,7 @@ export const H5_BASELINE_REVISIONS = Object.freeze({ B0: 'bac53ec3bfeaf6546e7aae
 export const ORIGINAL_BUDGETS = Object.freeze({ providerReadinessMs: 300_000, fullInvocationMs: 900_000,
   streamingMs: 120_000, terminalDrainageMs: 180_000, stopAcknowledgementMs: 30_000, remoteLeaseCleanupMs: 30_000 });
 export const H5_LIMITS = Object.freeze({ streamingMs: 600_000, terminalDrainageMs: 180_000, fullInvocationMs: 1_200_000 });
+export const SETTLEMENT_MAX_TRANSITIONS_PER_RUN = 16;
 export const H5_WORKLOADS = Object.freeze([
   { name: 'paced', runs: 1, events: 128, intervalMs: 25, prelude: 0, stop: false },
   { name: 'burst16', runs: 1, events: 16, intervalMs: 0, prelude: 0, stop: false },
@@ -22,14 +23,19 @@ export const H5_WORKLOADS = Object.freeze([
 ]);
 export const H5_REGISTRATION = Object.freeze({ mode: H5_MODE, protocolVersion: H5_PROTOCOL,
   baselineRevisions: H5_BASELINE_REVISIONS,
-  topology: H5_TOPOLOGY, originalDeadlineProtocol: H4D_PROTOCOL,
+  topology: H5_TOPOLOGY, originalDeadlineProtocol: H4E_PROTOCOL,
+  settlement: { classification: 'Failed native retryable_failure remains pending within the original terminal window',
+    detailReads: 'Only failed run-list candidates use the normal detail GET; no event polling or retry dispatch',
+    maxTransitionsPerRun: SETTLEMENT_MAX_TRANSITIONS_PER_RUN,
+    retryRatePolicy: 'No new recorded error class after frozen A/A; every paired and median total/per-class unique-ever-failed-run rate increase must stay within absolute paired A/A noise',
+    recordedFailures: 'Post-timing exact-run-set read-only snapshot unions current and durable recovered error codes with observed transitions; retained recovery metadata is last-failure evidence, not exhaustive per-class history' },
   originalBudgets: ORIGINAL_BUDGETS, observationLimits: H5_LIMITS, workloads: H5_WORKLOADS,
   pairs: 5, order: 'baseline-first on even pair index; candidate-first on odd pair index',
   stages: ['B0 A/A', 'B1 A/A', 'B0→B1 nonregression', 'B1→B2 improvement', 'B0→B2 improvement'],
   knownBaselineMisses: ['burst512:streaming120', 'invocation:observed-total-duration900'],
-  baselineEligibility: 'Finite complete correct observation; SLA misses remain failures of the H4d original deadlines',
+  baselineEligibility: 'Finite complete correct observation; SLA misses remain failures of the H4e original deadlines',
   cursorCandidateEligibility: 'Nonregression; only the paired baseline exact registered known misses may be inherited',
-  finalCandidateEligibility: 'Every original SLA passes in every arm; independently assessed total duration <900s; later H4d default-mode functional PASS on the same harness source and independent-agent geometry required',
+  finalCandidateEligibility: 'Every original SLA passes in every arm; independently assessed total duration <900s; later H4e default-mode functional PASS on the same harness source and independent-agent geometry required',
   noise: 'Fresh B0 and B1 A/A frozen before any further product implementation; no old abort/profile samples',
   performance: 'Loaded p95 emission/intended-to-visible and terminal drainage improve beyond frozen noise in median and at least4/5 pairs; all other metrics nonregress',
 });
@@ -156,11 +162,178 @@ export async function observeStreaming({ readCount, poll, target, now = Date.now
 
 export function originalTerminal(streaming, terminal) {
   return streaming.original.status === 'passed'
-    ? { status: terminal.complete ? 'passed' : 'failed', reason: terminal.complete ? null : 'original_terminal_deadline',
+    ? { status: terminal.complete ? 'passed' : 'failed', reason: terminal.complete ? null
+      : terminal.censored === false && terminal.error ? 'original_terminal_failure' : 'original_terminal_deadline',
       startedAtMs: terminal.startedAtMs, deadlineMs: terminal.deadlineMs, assessedAtMs: terminal.assessedAtMs }
     : { status: 'not-reached', reason: 'original_streaming_deadline', startedAtMs: null, deadlineMs: null, assessedAtMs: null };
 }
 
+/** The run list omits nativePhase. Resolve failed rows through the normal run
+ * detail API before deciding whether the existing terminal window has drained. */
+export async function observeRunSettlement({ readRuns, readRun, poll, expectedRuns, now = Date.now, onOutcome = _value => {} }) {
+  const startedAtMs = now();
+  const outcome = { startedAtMs, deadlineMs: startedAtMs + ORIGINAL_BUDGETS.terminalDrainageMs,
+    assessedAtMs: 0, lastObservedAtMs: 0, observedRuns: 0, terminalRuns: 0, pendingRetryRuns: 0,
+    lastStatuses: [], transitions: [], uniqueRetryableRuns: 0, recoveredRuns: 0, nonretryableFailures: 0,
+    retryableErrorClasses: {}, complete: false, censored: false };
+  const previous = new Map();
+  let runs = [];
+  onOutcome(outcome);
+  try {
+    await poll(async () => {
+      runs = [];
+      for (const listed of await readRuns()) {
+        const run = listed.status === 'failed' ? await readRun(listed.id) : listed;
+        assert.equal(run.id, listed.id, 'Run detail belongs to a different run');
+        if (listed.status === 'failed') {
+          assert.ok(typeof run.runtimeMode === 'string', 'Run detail must expose runtimeMode');
+          if (run.runtimeMode === 'native') assert.ok(Object.hasOwn(run, 'nativePhase'), 'Run detail must expose nativePhase');
+        }
+        runs.push(run);
+      }
+      assert.ok(runs.length <= expectedRuns && new Set(runs.map(run => run.id)).size === runs.length,
+        'Terminal observation contains duplicate or additional runs');
+      const observedAtMs = now();
+      const statuses = runs.map(run => ({ runId: run.id, status: run.status,
+        runtimeMode: run.runtimeMode ?? null, nativePhase: run.nativePhase ?? null, errorCode: run.errorCode ?? null }));
+      for (const state of statuses) {
+        const signature = JSON.stringify(state);
+        if (previous.get(state.runId) !== signature) {
+          assert.ok(outcome.transitions.filter(row => row.runId === state.runId).length < SETTLEMENT_MAX_TRANSITIONS_PER_RUN,
+            'Terminal state transition evidence overflow; no observation may be dropped');
+          outcome.transitions.push({ observedAtMs, ...state });
+        }
+        previous.set(state.runId, signature);
+      }
+      const pending = run => run.status === 'failed' && run.runtimeMode === 'native' && run.nativePhase === 'retryable_failure';
+      const settled = run => ['succeeded', 'cancelled', 'failed'].includes(run.status) && !pending(run);
+      Object.assign(outcome, { lastObservedAtMs: observedAtMs, observedRuns: runs.length,
+        terminalRuns: runs.filter(settled).length, pendingRetryRuns: runs.filter(pending).length, lastStatuses: statuses,
+        ...summarizeSettlementTransitions(outcome.transitions) });
+      onOutcome(outcome);
+      assert.ok(!runs.some(run => run.status === 'timed_out' || run.status === 'failed' && !pending(run)),
+        'Terminal run failed; automatic retry is not pending');
+      return runs.length === expectedRuns && runs.every(settled);
+    }, ORIGINAL_BUDGETS.terminalDrainageMs);
+    assert.equal(outcome.recoveredRuns, outcome.uniqueRetryableRuns, 'Every observed retryable run must recover successfully');
+    outcome.assessedAtMs = now(); outcome.complete = true; onOutcome(outcome);
+    return { runs, outcome };
+  } catch (error) {
+    outcome.assessedAtMs = now();
+    outcome.censored = error?.matcherResult?.name === 'toBe' && error.matcherResult.pass === false
+      && error.message.includes(`Timeout ${ORIGINAL_BUDGETS.terminalDrainageMs}ms exceeded while waiting on the predicate`);
+    outcome.error = String(error); onOutcome(outcome); throw error;
+  }
+}
+
+export function summarizeSettlementTransitions(transitions) {
+  const retryable = new Set(), recovered = new Set(), permanent = new Set(), classes = new Map();
+  for (const row of transitions) {
+    if (row.status === 'failed' && row.runtimeMode === 'native' && row.nativePhase === 'retryable_failure') {
+      assert.ok(typeof row.errorCode === 'string' && row.errorCode.length > 0, 'Retryable failure needs its observed error class');
+      retryable.add(row.runId);
+      const runs = classes.get(row.errorCode) ?? new Set(); runs.add(row.runId); classes.set(row.errorCode, runs);
+    } else if (row.status === 'failed' || row.status === 'timed_out') permanent.add(row.runId);
+    if (row.status === 'succeeded' && retryable.has(row.runId)) recovered.add(row.runId);
+  }
+  return { uniqueRetryableRuns: retryable.size, recoveredRuns: recovered.size, nonretryableFailures: permanent.size,
+    retryableErrorClasses: Object.fromEntries([...classes].sort(([a], [b]) => a.localeCompare(b)).map(([code, runs]) => [code, runs.size])) };
+}
+
+export function assertSettlementObservation(value, item) {
+  assert.ok(value, 'Missing canonical run settlement observation');
+  assert.equal(value.complete, true); assert.equal(value.censored, false);
+  assert.ok(Number.isSafeInteger(value.startedAtMs) && value.startedAtMs > 0);
+  assert.equal(value.deadlineMs, value.startedAtMs + ORIGINAL_BUDGETS.terminalDrainageMs);
+  assert.equal(value.assessedAtMs, item.settledAtMs);
+  assert.ok(value.lastObservedAtMs >= value.startedAtMs && value.lastObservedAtMs <= value.assessedAtMs
+    && value.assessedAtMs <= value.deadlineMs, 'Settlement must fit the original terminal window');
+  assert.equal(value.observedRuns, item.runs); assert.equal(value.terminalRuns, item.runs); assert.equal(value.pendingRetryRuns, 0);
+  assert.deepEqual(value.lastStatuses.map(row => row.runId).sort(), [...item.runIds].sort());
+  assert.ok(value.lastStatuses.every(row => row.status === (item.stop ? 'cancelled' : 'succeeded')), 'Failed runs are not settled success');
+  assert.ok(Array.isArray(value.transitions) && value.transitions.length >= item.runs
+    && value.transitions.length <= item.runs * SETTLEMENT_MAX_TRANSITIONS_PER_RUN, 'Missing or overflowing bounded state history');
+  const last = new Map(), counts = new Map(); let priorTime = value.startedAtMs;
+  for (const row of value.transitions) {
+    assert.ok(item.runIds.includes(row.runId), 'State history belongs to another run');
+    assert.ok(Number.isSafeInteger(row.observedAtMs) && row.observedAtMs >= priorTime
+      && row.observedAtMs <= value.lastObservedAtMs, 'Invalid state history timestamp');
+    priorTime = row.observedAtMs;
+    for (const field of ['runtimeMode', 'nativePhase', 'errorCode']) assert.ok(row[field] === null || typeof row[field] === 'string');
+    const count = (counts.get(row.runId) ?? 0) + 1; counts.set(row.runId, count);
+    assert.ok(count <= SETTLEMENT_MAX_TRANSITIONS_PER_RUN, 'State history overflow');
+    last.set(row.runId, row);
+  }
+  for (const row of value.lastStatuses) {
+    const { observedAtMs: _at, ...state } = last.get(row.runId) ?? {};
+    assert.deepEqual(state, row, 'Final state must match retained state history');
+  }
+  const summary = summarizeSettlementTransitions(value.transitions);
+  for (const [field, expected] of Object.entries(summary)) assert.deepEqual(value[field], expected, 'Retry ledger must match retained transitions');
+  assert.equal(summary.nonretryableFailures, 0, 'Nonretryable failures cannot qualify');
+  assert.equal(summary.recoveredRuns, summary.uniqueRetryableRuns, 'Every observed retryable run must recover successfully');
+  return summary;
+}
+
+export function assertRecordedFailures(proof, item, companyId) {
+  const observed = assertSettlementObservation(item.h5?.terminalObservation ?? item.terminalObservation, item);
+  assert.equal(proof?.kind, 'recorded-run-failure-codes', 'Missing post-timing recorded failure proof');
+  assert.equal(proof.transaction, 'READ ONLY REPEATABLE READ'); assert.equal(proof.observedReadOnly, 'on');
+  assert.equal(proof.observedIsolation, 'repeatable read');
+  const after = Math.max(item.settledAtMs, item.remoteLeasesCheckedAtMs ?? 0, item.stopAcknowledgedAtMs ?? 0,
+    item.localStopProof?.checkedAtMs ?? 0);
+  assert.ok(Number.isSafeInteger(proof.queryStartedAtMs) && proof.queryStartedAtMs >= after
+    && Number.isSafeInteger(proof.checkedAtMs) && proof.checkedAtMs >= proof.queryStartedAtMs
+    && Number.isSafeInteger(proof.connectionClosedAtMs) && proof.connectionClosedAtMs >= proof.checkedAtMs,
+  'Close the post-timing failure snapshot after settlement and lease timing');
+  assert.equal(proof.rows?.length, item.runs, 'Recorded failure proof needs every actual run');
+  assert.deepEqual(proof.rows.map(row => row.run_id).sort(), [...item.runIds].sort(), 'Recorded failure run set mismatch');
+  assert.equal(new Set(proof.rows.map(row => row.run_id)).size, item.runs);
+  const classes = new Map(), ever = new Set(), recovered = new Set();
+  const add = (runId, code) => {
+    assert.ok(typeof code === 'string' && code.length > 0 && code.length <= 128, 'Missing closed recorded error code');
+    ever.add(runId); const ids = classes.get(code) ?? new Set(); ids.add(runId); classes.set(code, ids);
+  };
+  const transitions = (item.h5?.terminalObservation ?? item.terminalObservation).transitions;
+  for (const row of transitions) if (row.status === 'failed') add(row.runId, row.errorCode);
+  for (const row of proof.rows) {
+    assert.equal(row.company_id, companyId, 'Recorded failure company mismatch');
+    assert.equal(row.agent_id, item.controllerAuthority.rows.find(authority => authority.run_id === row.run_id)?.agent_id,
+      'Recorded failure agent mismatch');
+    assert.equal(row.runtime_mode, 'native');
+    assert.equal(row.status, item.stop ? 'cancelled' : 'succeeded', 'Recorded run is not successfully settled');
+    if (!item.stop) assert.equal(row.native_phase, 'committed', 'Recorded native finalization is not committed');
+    else assert.ok(row.native_phase === null || typeof row.native_phase === 'string', 'Invalid recorded cancellation phase');
+    assert.ok(typeof row.current_has_error === 'boolean');
+    assert.ok(row.current_error_code === null || typeof row.current_error_code === 'string');
+    if (!item.stop) {
+      assert.equal(row.current_has_error, false, 'Successful committed runs must clear their current execution error');
+      assert.equal(row.current_error_code, null, 'Successful committed runs must clear their current execution error code');
+    }
+    // The normal Stop path writes the closed code "cancelled". Retain it as
+    // cancellation evidence; every other recorded code remains failure evidence.
+    if ((row.current_has_error || row.current_error_code !== null) && !(item.stop && row.current_error_code === 'cancelled'))
+      add(row.run_id, row.current_error_code);
+    if (row.recovered_type === null || row.recovered_type === 'null') {
+      for (const key of ['recovered_schema', 'recovered_error_code', 'recovered_observed_at']) assert.equal(row[key], null);
+    } else {
+      assert.equal(row.recovered_type, 'object'); assert.equal(row.recovered_schema, 'paperclip.recovered_execution_failure.v1');
+      assert.ok(typeof row.recovered_observed_at === 'string' && Number.isFinite(Date.parse(row.recovered_observed_at))
+        && Date.parse(row.recovered_observed_at) <= proof.checkedAtMs, 'Invalid recovered failure observation time');
+      add(row.run_id, row.recovered_error_code);
+    }
+    assert.ok(row.finalization_attempt === null || Number.isSafeInteger(row.finalization_attempt) && row.finalization_attempt >= 0,
+      'Invalid descriptive finalization attempt');
+    if (ever.has(row.run_id) && row.status === 'succeeded') recovered.add(row.run_id);
+  }
+  assert.equal(recovered.size, ever.size, 'Every recorded failed run must have recovered successfully');
+  assert.ok(item.durable.every(row => row.resultAcceptedEvents === (item.stop ? 0 : 1)), 'Every run needs its exact independently persisted result');
+  if (item.h5?.independentDrain) assert.ok(item.h5.independentDrain.verifiedAtMs >= proof.connectionClosedAtMs,
+    'Independent drainage must include the closed recorded-failure snapshot');
+  return { ...observed, uniqueEverFailedRuns: ever.size, allRecordedRecoveredRuns: recovered.size,
+    recordedErrorClasses: Object.fromEntries([...classes].sort(([a], [b]) => a.localeCompare(b)).map(([code, ids]) => [code, ids.size])) };
+}
+
 export function assessH5Arm(report, owner) {
   assert.equal(report.protocolVersion, H5_PROTOCOL);
   assert.equal(report.observationMode, H5_MODE);
@@ -209,6 +382,7 @@ export function assessH5Arm(report, owner) {
       assert.equal(streaming.error, null);
       assert.equal(value.terminalOriginal.status, 'passed');
     }
+    assertSettlementObservation(value.terminalObservation, item);
     assert.equal(value.terminalObservation.complete, true, 'H5 terminal observation must drain');
     assert.equal(value.terminalObservation.censored, false);
     assert.equal(value.terminalObservation.deadlineMs, value.terminalObservation.startedAtMs + H5_LIMITS.terminalDrainageMs);

tests/e2e/native-event-ingestion.journey.ts

@@ -7,8 +7,8 @@ import { hostname } from "node:os";
 import path from "node:path";
 import { expect, test } from "@playwright/test";
 import { json } from "./agent-chat.shared";
-import { h5Enabled, H5_MODE, H5_PROTOCOL, H4D_PROTOCOL, H5_TOPOLOGY, H5_LIMITS, H5_WORKLOAD_SHA256, ORIGINAL_BUDGETS,
-  observeStreaming, originalTerminal, prepareParticipantAgents, assertParticipantTopology } from "./fixtures/native-ingestion-h5.mjs";
+import { h5Enabled, H5_MODE, H5_PROTOCOL, H4E_PROTOCOL, H5_TOPOLOGY, H5_LIMITS, H5_WORKLOAD_SHA256, ORIGINAL_BUDGETS,
+  observeStreaming, observeRunSettlement, originalTerminal, prepareParticipantAgents, assertParticipantTopology, assertRecordedFailures } from "./fixtures/native-ingestion-h5.mjs";
 
 type SeenMarker = { markerCode: string; index: number; visibleMs: number };
 type Emission = { nonce: string; markerCode: string; index: number; intendedMs: number; emittedMs: number; receivedAtMs: number };
@@ -44,7 +44,7 @@ const stats = (values: number[]) => {
 
 async function runtimeAuthority(input: {
   appRoot: string; companyId: string; environmentId: string; issueIds: string[]; transport: "local" | "sandbox-remote";
-  afterStop?: boolean;
+  afterStop?: boolean; recordedFailureRunIds?: string[];
 }) {
   const configuration = process.env.PAPERCLIP_E2E_SERVER_CONFIG;
   if (!configuration) throw new Error("Missing owned server configuration");
@@ -68,6 +68,21 @@ async function runtimeAuthority(input: {
         current_setting('transaction_isolation') as isolation`;
       expect(settings[0].read_only).toBe("on");
       expect(settings[0].isolation).toBe("repeatable read");
+      if (input.recordedFailureRunIds) {
+        const queryStartedAtMs = Date.now();
+        const rows = await tx`select r.id as run_id, r.company_id, r.agent_id, r.runtime_mode, r.status, r.native_phase,
+          r.error_code as current_error_code, r.error is not null as current_has_error,
+          jsonb_typeof(r.result_json->'recoveredExecutionFailure') as recovered_type,
+          r.result_json->'recoveredExecutionFailure'->>'schema' as recovered_schema,
+          r.result_json->'recoveredExecutionFailure'->>'errorCode' as recovered_error_code,
+          r.result_json->'recoveredExecutionFailure'->>'observedAt' as recovered_observed_at,
+          f.attempt as finalization_attempt
+          from heartbeat_runs r left join native_run_finalizations f on f.run_id = r.id and f.company_id = r.company_id
+          where r.company_id = ${input.companyId} and r.id = any(${input.recordedFailureRunIds}::uuid[])`;
+        return { kind: "recorded-run-failure-codes", queryStartedAtMs, checkedAtMs: Date.now(),
+          transaction: "READ ONLY REPEATABLE READ", observedReadOnly: settings[0].read_only,
+          observedIsolation: settings[0].isolation, rows };
+      }
       const rows = await tx`select r.id as run_id, r.company_id, r.agent_id, r.native_issue_id as issue_id, r.runtime_mode,
         r.native_session_id, r.runner_instance_id, r.status, r.process_pid, r.process_group_id, r.process_started_at,
         r.runner_profile_json->'nativeExecutionInput'->'binding'->>'executionWorkspaceId' as admitted_execution_workspace_id,
@@ -115,7 +130,7 @@ async function runtimeAuthority(input: {
   } finally { await sql.end({ timeout: 5 }); }
   const connectionClosedAtMs = Date.now();
   let localJournal;
-  if (input.transport === "local" && !input.afterStop) {
+  if (input.transport === "local" && !input.afterStop && !input.recordedFailureRunIds) {
     expect(input.issueIds).toHaveLength(1);
     const base = await realpath(path.join(path.dirname(configuration), "runtime/paperclip-runner/durable-sessions"));
     expect(base.startsWith(`${ownedHome}${path.sep}`)).toBe(true);
@@ -275,7 +290,7 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
   const browserRecording = { trace: captureMode(info.project.use.trace), video: captureMode(info.project.use.video),
     screencast: captureMode(info.project.use.trace) !== "off" || captureMode(info.project.use.video) !== "off" };
   const report: Record<string, unknown> = {
-    protocolVersion: h5 ? H5_PROTOCOL : H4D_PROTOCOL, transport, bodyStartedAtMs,
+    protocolVersion: h5 ? H5_PROTOCOL : H4E_PROTOCOL, transport, bodyStartedAtMs,
     agentTopology: { ...H5_TOPOLOGY, remoteAgentPoolSize: Math.max(...cases.map(item => item.runs)) },
     ...(h5 ? { observationMode: H5_MODE, workloadSha256: H5_WORKLOAD_SHA256, observationLimits: H5_LIMITS,
       observationComplete: false, actualPlaywrightTimeoutMs: info.timeout,
@@ -555,25 +570,20 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
       }
       active.phase = "terminal-drainage";
       let runs: any[] = [];
-      const terminalStartedAtMs = h5 ? Date.now() : undefined;
-      const terminalObservation = h5 ? { startedAtMs: terminalStartedAtMs!, deadlineMs: terminalStartedAtMs! + H5_LIMITS.terminalDrainageMs,
-        assessedAtMs: 0, lastObservedAtMs: 0, observedRuns: 0, terminalRuns: 0,
-        lastStatuses: [] as Array<{ runId: string; status: string }>, complete: false, censored: false } : undefined;
-      if (h5) active.terminalObservation = terminalObservation;
-      try { await expect.poll(async () => {
-        runs = (await json(await request.get(`/api/companies/${company.id}/heartbeat-runs?limit=1000`))).filter((run: { issueId?: string; nativeIssueId?: string; contextSnapshot?: { issueId?: string; taskId?: string } }) => issueIds.includes(run.issueId ?? run.nativeIssueId ?? run.contextSnapshot?.issueId ?? run.contextSnapshot?.taskId));
-        if (terminalObservation) Object.assign(terminalObservation, { lastObservedAtMs: Date.now(), observedRuns: runs.length,
-          terminalRuns: runs.filter(run => ["succeeded", "cancelled", "failed"].includes(run.status)).length,
-          lastStatuses: runs.map(run => ({ runId: run.id, status: run.status })) });
-        return runs.length === issueIds.length && runs.every(run => ["succeeded", "cancelled", "failed"].includes(run.status));
-      }, { timeout: 180_000 }).toBe(true); }
+      let settlement;
+      try { settlement = await observeRunSettlement({ expectedRuns: issueIds.length,
+        readRuns: async () => (await json(await request.get(`/api/companies/${company.id}/heartbeat-runs?limit=1000`))).filter((run: { issueId?: string; nativeIssueId?: string; contextSnapshot?: { issueId?: string; taskId?: string } }) => issueIds.includes(run.issueId ?? run.nativeIssueId ?? run.contextSnapshot?.issueId ?? run.contextSnapshot?.taskId)),
+        readRun: async (runId: string) => json(await request.get(`/api/heartbeat-runs/${runId}`)),
+        poll: (read: () => Promise<boolean>, milliseconds: number) => expect.poll(read, { timeout: milliseconds }).toBe(true),
+        onOutcome: (value: Record<string, unknown>) => { active.terminalObservation = value; },
+      }); }
       catch (error) {
-        if (terminalObservation) { terminalObservation.assessedAtMs = Date.now(); terminalObservation.censored = true;
-          active.terminalOriginal = originalTerminal(streamingObservation!, terminalObservation); }
+        if (h5) active.terminalOriginal = originalTerminal(streamingObservation!, active.terminalObservation);
         throw error;
       }
-      const settledAt = Date.now(), seenMarkers = await observed();
-      if (terminalObservation) { terminalObservation.assessedAtMs = settledAt; terminalObservation.complete = true; }
+      const terminalObservation = settlement.outcome;
+      runs = settlement.runs;
+      const settledAt = terminalObservation.assessedAtMs, seenMarkers = await observed();
       active.settledAt = settledAt; active.seenMarkers = seenMarkers;
       const details = [];
       for (const run of runs) details.push(await json(await request.get(`/api/heartbeat-runs/${run.id}`)));
@@ -651,8 +661,9 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
         ...(h5 ? { h5: { streaming: streamingObservation, terminalObservation,
           terminalOriginal: originalTerminal(streamingObservation!, terminalObservation!),
           visibleCountByOriginalPaintDeadline: seenMarkers.filter(marker => marker.visibleMs <= streamingObservation!.original.deadlineMs).length,
-          deadlineCountScope: "Original poll last count/time retained separately; paint-deadline count derives from unchanged2RAF timestamps" } } : {}),
-        ...(h5 ? { terminalStatuses: runs.map(run => ({ runId: run.id, runtimeMode: run.runtimeMode, status: run.status })) } : {}),
+          deadlineCountScope: "Original poll last count/time retained separately; paint-deadline count derives from unchanged2RAF timestamps" } } : { terminalObservation }),
+        terminalStatuses: runs.map(run => ({ runId: run.id, runtimeMode: run.runtimeMode, status: run.status,
+          nativePhase: run.nativePhase ?? null, errorCode: run.errorCode ?? null })),
         setupActivationConcurrency, setupReadinessStartedAtMs, readinessDeadlineMs, allReadyObservedAtMs, activations, activationWaves,
         nonce, markerCode, epochMs, settledAtMs: settledAt, stopRequestedAtMs: stopRequestedAt,
         stopAcknowledgedAtMs: acknowledgedAt, preludeProof, controllerAuthority, clockEvidence, interChunkGaps,
@@ -689,7 +700,7 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
           if (!rows.length) break;
           for (const row of rows) {
             const event = row.payload?.prpEvent;
-            if (h5 && (row.eventType === "run.result.accepted" || event?.eventType === "run.result.accepted")) resultAcceptedEvents++;
+            if (row.eventType === "run.result.accepted" || event?.eventType === "run.result.accepted") resultAcceptedEvents++;
             if (event?.eventType !== "item.delta") continue;
             for (const match of String(event.payload?.text ?? "").matchAll(new RegExp(`\\[${code}([a-z0-9]{1,3})\\]`, "g"))) {
               const index = Number.parseInt(match[1], 36);
@@ -712,16 +723,20 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
         expect(new Set(deltas.map(item => item.identity)).size).toBe(deltas.length);
         expect(deltas.map(item => item.index)).toEqual(Array.from({ length: stop ? deltas.length : workload.events }, (_, index) => index));
         if (stop) for (const item of visible) expect(deltas.some(delta => delta.index === item.index)).toBe(true);
-        if (h5) expect(resultAcceptedEvents).toBe(stop ? 0 : 1);
+        expect(resultAcceptedEvents).toBe(stop ? 0 : 1);
         durable.push({ runId: run.id, token, markerCode: code, individualDeltas: deltas.length, uniqueSourceIdentities: deltas.length,
-          matchedSourceClocks: deltas.length, ordered: true, ...(h5 ? { resultAcceptedEvents } : {}) });
+          matchedSourceClocks: deltas.length, ordered: true, resultAcceptedEvents });
       }
       await expect.poll(() => [...state.streams.values()].every(stream => stream.ended || stream.aborted), { timeout: 30_000 }).toBe(true);
       expect(state.errors).toEqual([]);
       expect([...state.streams.values()].every(stream => !stream.error)).toBe(true);
+      const recordedFailures = await runtimeAuthority({ appRoot, companyId: company.id,
+        environmentId: caseEnvironment.id, issueIds, transport: caseTransport, recordedFailureRunIds: runs.map(run => run.id) });
+      active.recordedFailures = recordedFailures;
+      assertRecordedFailures(recordedFailures, { ...result, durable, runIds: runs.map(run => run.id) }, company.id);
       if (h5) {
         const verifiedAtMs = Date.now();
-        Object.assign(result.h5!, { independentDrain: { verifiedAtMs, elapsedFromEpochMs: verifiedAtMs - epochMs,
+        Object.assign((result as any).h5, { independentDrain: { verifiedAtMs, elapsedFromEpochMs: verifiedAtMs - epochMs,
           deltaCount: durable.reduce((count, row) => count + row.individualDeltas, 0),
           resultAcceptedCount: durable.reduce((count, row) => count + (row.resultAcceptedEvents ?? 0), 0),
           clockStreamsClosed: true,
@@ -732,6 +747,7 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
         hello: stream.hello, ended: stream.ended, aborted: stream.aborted, error: stream.error, partialBytes: stream.partialBytes,
         providerReadyAtMs: stream.providerReadyAtMs, clockHelloAtMs: stream.clockHelloAtMs, readinessObservedAtMs: stream.readinessObservedAtMs,
         emissions: [...stream.records.values()].sort((a, b) => a.index - b.index) })));
+      Object.assign(result, { recordedFailures });
       (report.cases as unknown[]).push({ ...result, durable, persistedAfterReload: true });
       await info.attach(`${workload.name}${stop ? "-stop" : ""}-metrics`, { body: JSON.stringify(result, null, 2), contentType: "application/json" });
       pending.delete(key); clocks.delete(key); delete report.activeCase;