diff --git a/doc/benchmarks/2026-11-04-native-event-ingestion-application.md b/doc/benchmarks/2026-11-04-native-event-ingestion-application.md new file mode 100644 index 00000000000..8497b91e945 --- /dev/null +++ b/doc/benchmarks/2026-11-04-native-event-ingestion-application.md @@ -0,0 +1,294 @@ +# Native ingestion application acceptance + +Status at 2026-11-04 10:00 UTC: **application acceptance remains open**. Source +append measurements passed their registered gates. No production-application +A/A calibration, paired browser improvement, or deployed improvement is claimed. +Both delivery PRs remain drafts. + +This report records the application inputs and qualification evidence. The +[source report](2026-11-03-native-event-ingestion-interleaved.md) measures active +append-service work; it does not establish application burst drainage or browser +latency. The dependent batching PR owns incremental and combined results. + +## Matched application builds + +All three inputs normally merge upstream +`4a83ba4050f6e6d334fe4cb6e3ace050097530ce`. They use independent normal locked +application installs and production builds. The browser driver has a separate +frozen install; the application checkouts' test files are not the executed driver. + +| Arm | Immutable application revision | Role | +| --- | --- | --- | +| B0 | `bac53ec3bfeaf6546e7aaeaee516487b30534654` | Matched baseline | +| B1 | `07061a601cdc20603c8c7a46f9ddbad7ac5f5939` | Cursor change | +| B2 base | `ca7cba171cd5198ed569d75012edc858041cbc44` | Unoptimized batching base | + +All three normal builds exited zero. Their shared native inputs have Rust tree +`bbe29bba574ce8d5cfddf2f7461ae0eb5983df07` and binary SHA256 +`abb65bdc073991e428bd37f9809603ee7564543cca0ed0305dc231137c93070f`. +Native target reuse was limited to the same 15 native build inputs. Each +application has its own Node dependencies and compiled output. + +The qualified-build receipt SHA256 is +`a1259ba760f3cbbe44817e333f9f620bcf3769fb0a6c16f9ce2f0f5b2ed60d94`. +The inventory-summary SHA256 is +`d3766fc4206d569d10fd02224433f1e9a2caa6992a7c62216e80af368aa4e44d`. +Inventories cover `server/dist`, `ui/dist`, `packages/db/src`, and +`packages/shared/src`: 12,884 / 12,898 / 12,906 files respectively. Every measured +arm must validate its actual files, preferred-UI-shadow absence, native binary, +clean source and compiled revision against the qualified inventory again. +Source/build-input comparisons match the frozen B1/B2 to the delivery branches; +this excludes test/tooling differences rather than executing the delivery lock +as the browser driver. + +These are build qualifications, not performance results. A further Rust change +needs its own immutable revision, normal build, binary and qualified inventory. +The existing B2 base cannot stand in for that candidate. + +## Retained full-load qualification failures + +Earlier H4c application attempts missed their original streaming deadline; +[their report](2026-11-03-native-event-ingestion-h4c.md) remains retained. Later +qualification failures are also excluded from paired performance samples. + +| Driver registration | Application baseline | Workload authority | Original 120s streaming observation | Eventual foreground completion | Final observation before teardown | +| --- | --- | --- | --- | --- | --- | +| H5a, `7388e24d` | Historical `a36567f9` | Two agents, 50 runs each | 54 of 512 foreground markers | 512 at 296.555s from poll start | 86 succeeded; 14 retryable failures | +| H5b, `179fe6fa6da1b24ca08c1162301538d3e3788af6` | Matched `bac53ec3` | 100 independent agents, one run each | 61 of 512 foreground markers | 512 at 282.734s from poll start | 83 succeeded; 17 retryable failures | + +Both attempts independently verified all 51,200 durable deltas against original +emission identities, payloads and clocks, and 100 accepted results. Those counts +do not establish final settlement or lease cleanup. Existing-history and Stop +cases were not reached. + +H5a's 14 failures encountered 30-second merge locks on the two shared agent +directories. The later independent-agent experiment does not establish a product +fix for that topology. A separate PostgreSQL deadlock on a succeeded run does +not establish the failed runs' wait cycle. + +H5b's 17 failures were different: their logs recorded suspension but no timely +provider-drain receipt after the five-second wait. The reason the receipts were +late remains unproved. The observer treated failed native `retryable_failure` +runs as terminal and stopped the application before their first supported retry. +The first retry was due at 05:40:32 UTC; teardown occurred at 05:40:26 UTC. +The global scheduler was enabled. Eligibility and recovery execution were not +observed, so this failure does not establish either recovery success or failure. + +The complete raw attempts, stopped-database read-only checks and source logs +remain sealed privately. Public summaries include only the fields above; raw +process, configuration and provider logs are not wholesale publication inputs. + +## H5c qualification + +The frozen driver is +[`e431ec63d41fe90c6fd68bd2fc2bb969feeca917`](https://github.com/paperclipai/paperclip/tree/e431ec63d41fe90c6fd68bd2fc2bb969feeca917), +tag `native-ingestion-h5c`. Its nine-file measurement SHA256 is +`2265aabe9c7344f5d4b9c8b08a5c6bbd5571a6e2de9ae7b72a8d1733250e609f`. +The [registered protocol](../../tests/e2e/fixtures/native-ingestion-h5.md) and +[exact execution helpers](../../scripts/native-ingestion/h5c/README.md) provide +reproduction commands, dependency provenance, bounded resource ownership and +pass/fail gates. + +H5c observes retryable finalization inside the same single 180-second terminal +window. It records observed transitions and current/retained recovery errors. +Final success, complete events/results, and lease cleanup remain mandatory. +The retained recovery marker contains the last failure; it is not an exhaustive +history of every retry. Comparisons reject new recorded error classes and retry +rates outside freshly frozen baseline noise. No manual retry or new provider +turn is introduced. + +The original 120-second streaming and 900-second invocation limits remain. +Extended observation preserves original deadline misses as failures. The final +candidate must pass every original limit in every arm and later pass a separate +default-mode original-SLA invocation with the same source and qualified build. + +The SMALL B0 functional qualification passed one Playwright attempt without +retry: 49.6 seconds including hooks, 33.0 seconds in the test body. Three remote +fixture runs retained 48 deltas with unique identities and original clocks, +three accepted results, reload persistence, and successful lease cleanup. A +separate normal local Stop run retained eight canonical deltas, zero accepted +results, native cancellation acknowledgement, independent runner/group/provider +absence and a released lease. Both cases recorded zero retries. This small case +does not exercise recovery under the 100-run load and does not qualify real +cloud Stop. + +The small report SHA256 is +`1c1ec8a0163e09fd75cabf3616374e8b185165a23c425f57794840c72b40782c`; +the 25-file seal manifest is +`6db1d2d41375a965fefa266a1d2bd47ddf9ad6f8d07773d7bde8a7bb8786b959`; +the sealed archive is +`50837389f0d2737a7f7ec249e75760da0aee366eeea881a9ce58f99025f20f83`. +Independent lead review verified those bytes, the owning attempt's attached +report, original clocks, recorded-failure proofs and external process/port +cleanup. The legacy raw `performanceAcceptanceEligible` flag denotes an +unprofiled run; `qualificationOnly: true` is independently rejected by the frozen +performance assessor. No flag or raw report was rewritten to change eligibility. + +## H5c full failure and independently verified persistence + +Full H5c B0 qualification failed at its unchanged 1,200-second observation +limit during independent durable-event verification. The foreground showed +66 of 512 markers near the original 120-second deadline and all 512 at +283.391 seconds after the epoch. All 100 runs succeeded within 75.926 seconds +of the original single 180-second terminal window. Fourteen transient provider +transport failures recovered; no nonretryable failure remained. + +Independent lead review verified the owning report, 56 raw files and eight +read-only stopped-PostgreSQL supplement files. All 51,200 durable deltas match +the original emission identities, payloads, ordering and clocks; all 100 results +were accepted, finalizers committed and leases released with successful cleanup. +This does not establish that those events committed before the original +streaming deadline. The owning Playwright attempt timed out after verifying +only 33 of 100 runs through its API oracle. Existing-history and Stop cases +were not reached. API cleanup encountered a disposed request context; separate +physical-quiet and port checks passed. The attempt remains failed and ineligible. + +The failed report SHA256 is +`4946558e5a04377650661cbe2c0d5d45b755ef77f5064399e4df8889c3f7b595`; +the raw seal manifest is +`2adb8afed5a10c6dc2091d7946a2e2d3bfc8ad6677613b46c20b1190d5ce47c8`. + +## H5d prospective correction + +The separately frozen driver is +[`2df9897e1ce2ca90be29fa097d776c3766e217ad`](https://github.com/paperclipai/paperclip/tree/2df9897e1ce2ca90be29fa097d776c3766e217ad), +tag `native-ingestion-h5d`. Its nine-file measurement SHA256 is +`e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4`; +the comparator SHA256 is +`09daef50fad5eef4dbda4e63a191749d0d167d3ae9d38bfd22df8721e2aac4a3`. + +Plain assertions replace individual Playwright steps in bulk readback while +preserving payload, identity, original-clock, ordering, result and Stop-prefix +checks. Workloads, browser observation, APIs and deadlines stay fixed. A +separate no-application diagnostic measures oracle cost; it establishes no +product performance gain. H5d has now completed its own small and full functional observation. Fresh +A/A calibration remains separate; qualification attempts are not reused. + +Before observing A/A data, the comparator registers suitability limits for +paired difference and chronological/role drift. It freezes actual absolute and +relative noise; its two-millisecond timestamp representation floor never pads +a regression allowance. Short measurements cannot support subfloor, percentage +speedup or p99 claims. Fully verified completion now has its own quantitative +gate: faster browser delivery or terminal status cannot hide slower final +drainage, missing readback or unclosed leases. Its conservative completion rate +includes verification, reload and lease observation; it is not the exact last +commit time. All comparison arms must be fresh, and every final-candidate arm +must meet the original limits as well as a later same-source original-SLA run. + +## H5d qualification and calibration status + +SMALL passed one owning attempt: 48 normal deltas across three remote fixture +runs, three accepted results, reload persistence, and a separate local Stop with +nine canonical deltas, zero results, acknowledged cancellation, Runner PID/group +and provider PID absence, a bound server retirement receipt and released lease. +Lead review verified all 40 sealed files and the exact owning attachment. + +FULL passed one owning test attempt without retry in 853.339 seconds; the whole +command took 870.711 seconds. Independent readback completed all six cases: + +| Case | Measured deltas | Accepted results | Fully verified drainage from epoch | +| --- | ---: | ---: | ---: | +| Paced single stream | 128 | 1 | 8.662s | +| 16-event burst, one run | 16 | 1 | 6.589s | +| 128-event burst, ten runs | 1,280 | 10 | 15.287s | +| 512-event burst, 100 runs | 51,200 | 100 | 402.584s | +| Foreground history under ten concurrent streams | 1,280 | 10 | 17.088s | +| Local Stop | 9 | 0 | 3.948s | + +The history case is **one foreground run with 1,000 persisted prelude events +under nine fresh concurrent streams**. The frozen workload explicitly gives +background streams no prelude. A normal event-API read verified the foreground's +1,000 unique prelude events and retained global head 1062 before the common +epoch. It does not establish that all ten runs have 1,000-event histories. + +The baseline still failed the original loaded-streaming limit: 41 of 512 +foreground markers were visible at the last observation near 120 seconds. +Foreground completion took 287.750 seconds; terminal settlement then took +103.326 seconds, followed by 12.483 seconds of verification. Forty-three +recorded provider transport failures recovered; all 100 runs succeeded and their +leases closed successfully. The original 900-second owning-test gate passed. +This is complete functional observation with a retained responsiveness failure, +not passing application performance acceptance. + +The full report SHA256 is +`d12ad9128d6a30bc335a69761d6004b2b3618fc275a07a84ef5434cfdb00b1e1`. +Lead review verified 38 sealed files, exact owning attachment, all source-clock +prefix joins, counts, results and leases, and reran the frozen assessment on +those bytes. An independent postreader initially assumed the foreground was the +first element of an API run list. The actual foreground was element 9. The +preserved failure was corrected in a separate offline reader by joining the +foreground token, durable proof, provider clock, activation and authority row; +event-count and timing checks remain. The owning attempt, raw seal, measurement +source and workload were unchanged. No failed application attempt was regraded. + +A separate no-application diagnostic confirmed assertion-framework overhead for +synthetic 1,000-event inputs: five samples each gave medians of 3,859.190ms for +the old Playwright assertions and 1.954ms for plain assertions. Playwright +samples increased chronologically from 1,843.546 to 5,655.339ms; plain assertions +ranged from 1.363 to 3.017ms. This is neither a stable product-cost distribution +nor an exact attribution of the earlier full-run timeout. + +Fresh B0 A/A began at 09:30 UTC. Two of ten invocations have completed and the +third is running. No noise envelope or paired application gain has passed. + +## Blocked-write Stop qualification + +The separate companion emits its own 1,000-event prelude through normal +ingestion, confirms a real append blocked by an owned run-row lock, then presses +Stop through the UI. During a fixed hold after the actual HTTP request it +requires no committed cancellation intent or audit, a pending response, ongoing +emissions and both birth-identified process groups alive. Public Stop must +commit its own intent under the same run lock; postrelease measurements retain +that durable boundary rather than treating a blocked intent as dispatch delay. + +Two qualification failures remain excluded. V1 failed before the measured +epoch because it incorrectly assumed the provider shared the Runner process +group. Its cleanup left live processes; separately recorded manual cleanup +retired those owned processes. Manual cleanup is not successful Stop evidence. +V2 measured the held-write boundary, its own acknowledged intent and audit +records, and physical exit of both independent groups. It then failed an +unsupported unconditional `native.local_process_stopped` assertion. Transcript, +reload and lease checks after that assertion were not reached. Normal cleanup, +port and strict physical-quiet checks passed without manual signals. + +The product supports two termination-proof states: retained original process +identities with verified physical absence, or cleared identities with a +server-authored retirement event bound to the latest launch/identity. The +retirement event is authoritative evidence for identity clearing; immediate +cancellation acknowledgement does not universally require it. The separately frozen +v3 oracle checks this distinction in a coherent authority snapshot after the +existing lease poll, without adding a wait budget. It also retains the own-intent, +audit, physical-group, lease, transcript and reload requirements. V2 cannot be +regraded or reused as an accepted arm. + +V2 source is `6e5349cdbd5e3a197877f1082f84a17e866c03e7`; its failed report +SHA256 is `0ab6c383a07c57f5eb16f226d0e458c70b6d6800e3d9dce051e3d2ca53bba20e` +and 42-file manifest is +`f81168c2d0c2218d658e1c6422b074fe80d10de9d88ad752c0b04fc54b1cde65`. +Independent lead review verified those bytes and the owning attempt's attachment. +Those two failed trials remain excluded. V3 at +`bc33f09cae26404352848169d92a20c89480a5f5` subsequently passed one independent +qualification. Fifteen held samples covered 1,010ms after the actual Stop POST +with no committed intent/audit, response pending, and both original groups +alive. After release it verified the same own native intent and audits, both +identified PIDs and both independent process groups absent, cleared mutable +identity backed by the latest bound server receipt, and the exact local lease +released without failed cleanup. The 1,000-event prelude, twelve canonical Stop +deltas, zero results and persisted reload were verified. API cleanup and strict +physical quiet passed without manual signals. + +Lead review independently verified all 49 files, the ten arm-artifact seals, +the exact owning Playwright attachment, and the frozen functional oracle. V3 raw +report SHA256 is +`66ebb86acfb5de5a3140ec4a5db104775265fc296c8e831d2964d065359a739b`. +This qualifies that functional path; control-latency acceptance still needs its +own fresh baseline noise and paired comparisons, distinct from the main matrix. + +## Remaining acceptance + +No application A/A or paired browser improvement has passed. No proposed Rust +optimization is implemented. Remaining gates are fresh B0/B1 application and companion noise; all three paired comparisons; loaded browser +and complete-drainage improvement; sparse/control nonregression; complete-window +resource accounting; original-SLA confirmation; affected live-provider and real +cloud Stop journeys on the final runtime; and current-head checks/review. +Local evidence establishes no deployed improvement. diff --git a/scripts/compare-native-ingestion-browser.ts b/scripts/compare-native-ingestion-browser.ts index 53fbb6ff03d..f9e3f206586 100644 --- a/scripts/compare-native-ingestion-browser.ts +++ b/scripts/compare-native-ingestion-browser.ts @@ -107,6 +107,10 @@ const workloads = [ ]; const names = workloads.map(item => item.name); const median = (values: number[]) => [...values].sort((a, b) => a - b)[Math.floor(values.length / 2)]!; +const pooledMedian = (values: number[]) => { + const sorted = [...values].sort((a, b) => a - b), middle = Math.floor(sorted.length / 2); + return sorted.length % 2 ? sorted[middle]! : (sorted[middle - 1]! + sorted[middle]!) / 2; +}; const first = pairs[0]; const sha256 = (value: unknown) => typeof value === "string" && /^[a-f0-9]{64}$/.test(value); const uuid = (value: unknown) => typeof value === "string" && /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i.test(value); @@ -463,27 +467,112 @@ if (h5 && phase === "compare" && candidatePolicy === "final-b2") { } } const metric = (item: any, key: string): number => key.split(".").reduce((value, part) => value[part], item); +// Prospective main-campaign rule. The 2ms floor is timestamp contrast +// representation only: it never pads observed noise or a regression gate. +const browserAaRule = Object.freeze({ pairCount: 5, pairedFraction: 0.20, medianDriftFraction: 0.10, + contrastRepresentationFloorMs: 2, representationLimitedMedianMs: 20 }); +function assertArmChronology(rows: any[]) { + assert.equal(rows?.length, 10, "Browser A/A/comparison requires ten actual owning invocations"); + const sorted = [...rows].sort((a,b) => a.startedAtMs - b.startedAtMs); + const companies = new Set(), runIds = new Set(), owners = new Set(); + for (const [index, row] of sorted.entries()) { + assert.ok(Number.isSafeInteger(row.startedAtMs) && row.startedAtMs > 0 && Number.isFinite(row.durationMs) + && row.durationMs > 0, "Missing finite actual owning timestamp/duration"); + assert.equal(row.finishedAtMs, row.startedAtMs + row.durationMs); + if (index) assert.ok(sorted[index - 1].finishedAtMs <= row.startedAtMs, "Browser owning invocations overlap"); + assert.ok(sha256(row.ownerReportSha256) && !owners.has(row.ownerReportSha256), "Reused owning report"); owners.add(row.ownerReportSha256); + assert.ok(uuid(row.companyId) && !companies.has(row.companyId), "Each browser arm needs its fresh owned company"); companies.add(row.companyId); + assert.ok(Array.isArray(row.runIds) && row.runIds.length > 0); + for (const runId of row.runIds) { + assert.ok(uuid(runId) && !runIds.has(runId), "Browser arms reuse an owned run"); runIds.add(runId); + } + const pairIndex = Math.floor(index / 2), firstArm = pairIndex % 2 ? "candidate" : "baseline"; + assert.equal(row.pairIndex, pairIndex, "Owner chronology must match registered pair order"); + assert.equal(row.arm, index % 2 ? firstArm === "baseline" ? "candidate" : "baseline" : firstArm, + "Browser pairs must alternate baseline-first then candidate-first"); + } + return sorted; +} +const armChronology = h5 ? assertArmChronology(pairs.flatMap((pair, pairIndex) => (["baseline", "candidate"] as const).map(arm => { + const report = pair[arm], owner = report.owningPlaywright, startedAtMs = Date.parse(owner.startedAt); + return { pairIndex, arm, startedAtMs, durationMs: owner.durationMs, finishedAtMs: startedAtMs + owner.durationMs, + ownerReportSha256: owner.reportSha256, companyId: report.companyId, runIds: report.cases.flatMap((item: any) => item.runIds) }; +}))) : undefined; +function assessAaMetric(samples: any[], chronology: any[]) { + assert.equal(samples.length, 5); + for (const sample of samples) { + assert.ok(Number.isFinite(sample.baseline) && sample.baseline > 0 && Number.isFinite(sample.candidate) && sample.candidate >= 0); + assert.equal(sample.improvementMs, sample.baseline - sample.candidate, "A/A paired absolute contrast was forged"); + assert.equal(sample.improvementRelative, sample.improvementMs / sample.baseline, "A/A paired relative contrast was forged"); + } + const pairedDifferencesMs = samples.map(sample => Math.abs(sample.improvementMs)); + const pairedLimitsMs = samples.map(sample => Math.max(browserAaRule.pairedFraction * sample.baseline, browserAaRule.contrastRepresentationFloorMs)); + const baselineMedianMs = median(samples.map(sample => sample.baseline)), candidateMedianMs = median(samples.map(sample => sample.candidate)); + const chronologicalValues = chronology.map(row => samples[row.pairIndex][row.arm]); + const firstFiveMedianMs = median(chronologicalValues.slice(0,5)), lastFiveMedianMs = median(chronologicalValues.slice(5)); + const chronologicalMedianDriftMs = Math.abs(lastFiveMedianMs - firstFiveMedianMs); + const chronologicalDriftLimitMs = Math.max(browserAaRule.medianDriftFraction * firstFiveMedianMs, browserAaRule.contrastRepresentationFloorMs); + const roleMedianDriftMs = Math.abs(candidateMedianMs - baselineMedianMs); + const roleDriftLimitMs = Math.max(browserAaRule.medianDriftFraction * baselineMedianMs, browserAaRule.contrastRepresentationFloorMs); + const pooledMedianMs = pooledMedian(samples.flatMap(sample => [sample.baseline, sample.candidate])); + return { sampleCount: 10, pairCount: 5, pairedDifferencesMs, pairedLimitsMs, baselineMedianMs, candidateMedianMs, + pooledMedianMs, firstFiveMedianMs, lastFiveMedianMs, chronologicalMedianDriftMs, chronologicalDriftLimitMs, + roleMedianDriftMs, roleDriftLimitMs, representationLimited: pooledMedianMs <= browserAaRule.representationLimitedMedianMs, + observedNoiseAbsoluteMs: Math.max(...pairedDifferencesMs), + observedNoiseRelative: Math.max(...samples.map(sample => Math.abs(sample.improvementRelative))), + passed: pairedDifferencesMs.every((value,index) => value <= pairedLimitsMs[index]) + && chronologicalMedianDriftMs <= chronologicalDriftLimitMs && roleMedianDriftMs <= roleDriftLimitMs }; +} const measurements = names.flatMap(name => { const observed = first.baseline.cases.find((item: any) => item.name === name); const keys = ["firstVisibleAfterEmissionMs", "emissionToVisible.medianMs", "drainThroughTerminalMs", ...(name === "paced" ? ["interChunkGaps.maxMs"] : []), ...(observed.emissionToVisible.p95Ms === undefined ? [] : ["emissionToVisible.p95Ms", "intendedToVisible.p95Ms"]), - ...(observed.stop ? ["stopThroughTerminalMs", "stopThroughAcknowledgementMs"] : [])]; + ...(observed.stop ? ["stopThroughTerminalMs", "stopThroughAcknowledgementMs"] : []), + ...(h5 && !observed.stop ? ["h5.independentDrain.elapsedFromEpochMs"] : [])]; return keys.map(key => { const samples = pairs.map(pair => { const baseline = metric(pair.baseline.cases.find((item: any) => item.name === name), key); const candidate = metric(pair.candidate.cases.find((item: any) => item.name === name), key); assert.ok(Number.isFinite(baseline) && baseline > 0 && Number.isFinite(candidate) && candidate >= 0, `Invalid latency ${name}.${key}`); - return { baseline, candidate, improvementRelative: (baseline - candidate) / baseline }; + return { baseline, candidate, improvementMs: baseline - candidate, improvementRelative: (baseline - candidate) / baseline }; }); + const pooledMedianMs = pooledMedian(samples.flatMap(sample => [sample.baseline, sample.candidate])); + const representationLimited = h5 && pooledMedianMs <= browserAaRule.representationLimitedMedianMs; + const aaSuitability = h5 && phase === "aa" ? assessAaMetric(samples, armChronology!) : undefined; return { name, metric: key, baselineMedianMs: median(samples.map(sample => sample.baseline)), candidateMedianMs: median(samples.map(sample => sample.candidate)), - medianPairedImprovementRelative: median(samples.map(sample => sample.improvementRelative)), + medianPairedImprovementRelative: representationLimited ? null : median(samples.map(sample => sample.improvementRelative)), + ...(h5 ? { pooledMedianMs, representationLimited, percentageGainClaimAllowed: !representationLimited, + sub2msOrP99GainClaimAllowed: false, medianPairedImprovementMs: median(samples.map(sample => sample.improvementMs)), + aaSuitability } : {}), candidateWins: samples.filter(sample => sample.improvementRelative > 0).length, aaNoiseEnvelopeRelative: phase === "aa" ? Math.max(...samples.map(sample => Math.abs(sample.improvementRelative))) : undefined, + ...(h5 ? { aaNoiseEnvelopeAbsoluteMs: phase === "aa" ? Math.max(...samples.map(sample => Math.abs(sample.improvementMs))) : undefined } : {}), samples }; }); }); +// A descriptive rate from the independently verified completion upper bound. +// Gate duration against its own A/A noise; invent no separate rate threshold. +const verifiedCompletionRates = h5 ? names.filter(name => !first.baseline.cases.find((item: any) => item.name === name).stop).map(name => { + const samples = pairs.map(pair => { + const baseline = pair.baseline.cases.find((item: any) => item.name === name); + const candidate = pair.candidate.cases.find((item: any) => item.name === name); + const counts = [baseline, candidate].map(item => item.h5.independentDrain.deltaCount); + assert.equal(counts[0], counts[1], "Verified completion rates require equal matched measured-delta counts"); + assert.ok([baseline, candidate].every(item => item.h5.independentDrain.elapsedFromEpochMs > 0), + "Verified completion rate needs positive observed duration"); + return { measuredDeltas: counts[0], baselineElapsedMs: baseline.h5.independentDrain.elapsedFromEpochMs, + candidateElapsedMs: candidate.h5.independentDrain.elapsedFromEpochMs, + baselineDeltasPerSecond: 1000 * counts[0] / baseline.h5.independentDrain.elapsedFromEpochMs, + candidateDeltasPerSecond: 1000 * counts[1] / candidate.h5.independentDrain.elapsedFromEpochMs }; + }); + return { name, sampleCount: samples.length, samples, + baselineMedianDeltasPerSecond: median(samples.map(sample => sample.baselineDeltasPerSecond)), + candidateMedianDeltasPerSecond: median(samples.map(sample => sample.candidateDeltasPerSecond)), + gatedDurationMetric: "h5.independentDrain.elapsedFromEpochMs", + scope: "Matched measured normalized deltas / independently verified completion upper bound; includes oracle, reload, screenshot, API and lease verification; not exact last-commit throughput" }; +}) : undefined; const retryObservations = names.map(name => { const samples = pairs.map(pair => Object.fromEntries((["baseline", "candidate"] as const).map(arm => { const item = pair[arm].cases.find((value: any) => value.name === name); @@ -525,6 +614,9 @@ const result: Record = { comparatorSha256: createHash("sha256").update(await readFile(new URL(import.meta.url))).digest("hex"), manifestSha256: createHash("sha256").update(await readFile(manifestPath)).digest("hex"), measurements, primaryLoadedWorkload: "burst512", complete: true, retryObservations, + ...(h5 ? { verifiedCompletionRates, armChronology, + ...(phase === "aa" ? { aaSuitability: { rule: browserAaRule, metricCount: measurements.length, + passed: measurements.every(item => item.aaSuitability?.passed) } } : {}) } : {}), ...(h5 ? { observationMode: H5_MODE, workloadSha256: H5_WORKLOAD_SHA256, candidatePolicy, h5Assessments, artifactInventories: { baseline: first.baseline.armInventory, candidate: first.candidate.armInventory }, originalH4eConfirmation, originalProtocolAcceptance: "Baseline SLA misses are retained; H5c complete observation is not an original-deadline PASS", @@ -550,16 +642,55 @@ if (phase === "compare") { assert.equal(noise.gatesSha256, result.gatesSha256, "Acceptance gates changed after A/A calibration"); assert.equal(noise.comparatorSha256, result.comparatorSha256, "Comparator changed after A/A calibration"); assert.ok(noise.pairCount >= 5); + if (h5) { + assert.equal(noise.pairCount, 5); assert.deepEqual(noise.aaSuitability?.rule, browserAaRule, "Browser A/A suitability rule changed"); + const chronology = assertArmChronology(noise.armChronology); + assert.deepEqual(chronology, noise.armChronology, "Frozen A/A chronology was reordered"); + assert.ok(armChronology![0].startedAtMs >= chronology.at(-1).finishedAtMs, + "Comparison must follow all owning calibration invocations"); + const priorOwners = new Set(chronology.map(row => row.ownerReportSha256)); + const priorCompanies = new Set(chronology.map(row => row.companyId)); + const priorRuns = new Set(chronology.flatMap(row => row.runIds)); + for (const arm of armChronology!) { + assert.ok(!priorOwners.has(arm.ownerReportSha256) && !priorCompanies.has(arm.companyId) + && arm.runIds.every((runId: string) => !priorRuns.has(runId)), "Comparison reuses a calibration owner/company/run"); + } + assert.equal(noise.measurements?.length, measurements.length, "Frozen A/A is missing a predeclared metric"); + assert.equal(noise.aaSuitability.metricCount, measurements.length); + for (const item of measurements) { + const controls = noise.measurements.filter((entry: any) => entry.name === item.name && entry.metric === item.metric); + assert.equal(controls.length, 1, "Frozen A/A needs exactly one calibration per metric"); + const control = controls[0], actual = assessAaMetric(control.samples, chronology); + assert.equal(actual.passed, true, "Unsuitable browser A/A cannot freeze noise"); + assert.deepEqual(control.aaSuitability, actual, "Browser A/A suitability was forged"); + assert.equal(control.aaNoiseEnvelopeAbsoluteMs, actual.observedNoiseAbsoluteMs, "Browser absolute noise was forged"); + assert.equal(control.aaNoiseEnvelopeRelative, actual.observedNoiseRelative, "Browser relative noise was forged"); + assert.equal(control.representationLimited, actual.representationLimited, "Browser representation flag was forged"); + } + assert.equal(noise.aaSuitability.passed, true, "Unsuitable browser A/A cannot freeze noise"); + } result.gates = measurements.map(item => { const control = noise.measurements.find((entry: any) => entry.name === item.name && entry.metric === item.metric); assert.ok(control); const improvementRequired = expectation === "improvement" && item.name === "burst512" - && ["emissionToVisible.p95Ms", "intendedToVisible.p95Ms", "drainThroughTerminalMs"].includes(item.metric); - const noRegression = -item.medianPairedImprovementRelative <= control.aaNoiseEnvelopeRelative; - const paired = noiseExceedingPairedWins(item.samples.map(sample => sample.improvementRelative), control.aaNoiseEnvelopeRelative); + && ["emissionToVisible.p95Ms", "intendedToVisible.p95Ms", "drainThroughTerminalMs", + "h5.independentDrain.elapsedFromEpochMs"].includes(item.metric); + const representationLimited = h5 && (control.representationLimited || item.representationLimited); + if (representationLimited) { + item.representationLimited = true; item.percentageGainClaimAllowed = false; item.medianPairedImprovementRelative = null; + } + const medianImprovementRelative = median(item.samples.map(sample => sample.improvementRelative)); + const medianImprovementMs = median(item.samples.map(sample => sample.improvementMs)); + const noRegression = representationLimited ? -medianImprovementMs <= control.aaNoiseEnvelopeAbsoluteMs + : -medianImprovementRelative <= control.aaNoiseEnvelopeRelative; + const paired = noiseExceedingPairedWins(item.samples.map(sample => h5 && sample.improvementMs <= 2 ? 0 : sample.improvementRelative), control.aaNoiseEnvelopeRelative); const improvement = paired.pairedWinRequirementPassed - && item.medianPairedImprovementRelative > control.aaNoiseEnvelopeRelative; + && medianImprovementRelative > control.aaNoiseEnvelopeRelative && (!h5 || medianImprovementMs > 2); return { name: item.name, metric: item.metric, improvementRequired, noiseEnvelopeRelative: control.aaNoiseEnvelopeRelative, + ...(h5 ? { noiseEnvelopeAbsoluteMs: control.aaNoiseEnvelopeAbsoluteMs, representationLimited, + percentageGainClaimAllowed: !representationLimited, sub2msOrP99GainClaimAllowed: false, + nonregressionScale: representationLimited ? "observed-absolute-ms" : "observed-relative", + requiredAbsoluteImprovementAboveMs: improvementRequired ? 2 : undefined } : {}), noiseExceedingPairedWins: paired.noiseExceedingPairedWins, requiredPairedWins: paired.requiredPairedWins, passed: improvementRequired ? improvement : noRegression }; }); @@ -588,6 +719,8 @@ if (phase === "compare") { } if (arg("output")) await writeFile(arg("output")!, JSON.stringify(result, null, 2)); console.log(JSON.stringify(result, null, 2)); +if (h5 && phase === "aa") assert.equal(result.aaSuitability.passed, true, + "Browser A/A suitability failed; retain all ten arms, diagnose and review a fresh calibration"); if (process.argv.includes("--require-performance")) { assert.equal(h5, true, "H5c performance-only gating is explicit; default H4e keeps original deadlines"); assert.equal(phase, "compare"); diff --git a/scripts/native-ingestion-h5.test.mjs b/scripts/native-ingestion-h5.test.mjs index 0c04e043eca..a54364447d3 100644 --- a/scripts/native-ingestion-h5.test.mjs +++ b/scripts/native-ingestion-h5.test.mjs @@ -3,23 +3,20 @@ import { test } from 'node:test'; import { createRequire } from 'node:module'; import { readFile } from 'node:fs/promises'; import { h5Enabled, H5_MODE, observeStreaming, originalTerminal, requireH5Candidate, - H5_LIMITS, ORIGINAL_BUDGETS, prepareParticipantAgents, observeRunSettlement } from '../tests/e2e/fixtures/native-ingestion-h5.mjs'; + H5_LIMITS, ORIGINAL_BUDGETS, prepareParticipantAgents, observeRunSettlement, collectDurableRunEvents, assertDurableRunEvents, + assertPreludeEvents, assertVisibleMarkerClocks } from '../tests/e2e/fixtures/native-ingestion-h5.mjs'; const timeout = milliseconds => Object.assign(new Error(`Timeout ${milliseconds}ms exceeded while waiting on the predicate`), { matcherResult: { name: 'toBeGreaterThanOrEqual', pass: false } }); -test('the actual paginated result counter counts accepted results in default H4e and observation H5c', async () => { - const source = await readFile(new URL('../tests/e2e/native-event-ingestion.journey.ts', import.meta.url), 'utf8'); - const statements = source.split('\n').filter(line => line.includes('resultAcceptedEvents++')); - assert.equal(statements.length, 1, 'Exercise the actual event-loop counter statement'); - const count = new Function('h5', 'rows', `let resultAcceptedEvents = 0; - for (const row of rows) { const event = row.payload?.prpEvent; ${statements[0]} } return resultAcceptedEvents;`); - for (const h5 of [false, true]) { - assert.equal(count(h5, [{ eventType: 'run.result.accepted', payload: { prpEvent: { eventType: 'run.result.accepted' } } }]), 1); - assert.equal(count(h5, [{ payload: { prpEvent: { eventType: 'run.result.accepted' } } }]), 1); - assert.equal(count(h5, [{ eventType: 'run.result.proposed' }, { payload: { prpEvent: { eventType: 'item.delta' } } }]), 0); - assert.equal(count(h5, []), 0); - } +test('the actual durable collector counts each accepted row once in default H4e and observation H5c', () => { + for (const row of [ + { eventType: 'run.result.accepted', payload: { prpEvent: { eventType: 'run.result.accepted' } } }, + { payload: { prpEvent: { eventType: 'run.result.accepted' } } }, + { eventType: 'run.result.accepted' }]) assert.equal(collectDurableRunEvents([row], 'q7').resultAcceptedEvents, 1); + assert.equal(collectDurableRunEvents([{ eventType: 'run.result.proposed' }, + { payload: { prpEvent: { eventType: 'item.delta' } } }], 'q7').resultAcceptedEvents, 0); + assert.equal(collectDurableRunEvents([], 'q7').resultAcceptedEvents, 0); }); test('real frozen Playwright tiny poll exposes the matcher deadline shape used by H5', async () => { @@ -231,3 +228,87 @@ test('cursor inherits only paired exact known misses; final candidate inherits n assert.throws(() => requireH5Candidate('final-b2', base, base), /every original SLA/); assert.throws(() => requireH5Candidate('final-b2', slowInvocation, slowInvocation), /every original SLA/); }); + + +const durableFixture = ({ stop = false, count = 4 } = {}) => { + const runId = 'run-1', token = 'provider-1', markerCode = 'q7'; + const rows = Array.from({ length: count }, (_, index) => ({ seq: index+1, payload: { prpEvent: { + eventType: 'item.delta', runId, sourceInstanceId: 'source-1', sourceEventId: `event-${index}`, sourceSeq: index+1, + payload: { text: `[${markerCode}${index.toString(36)}] ` } } } })); + if (!stop) rows.push({ seq: count+1, eventType: 'run.result.accepted' }); + return { rows, input: { runId, token, markerCode, events: 4, stop, requireNativeIdentity: true, + sourceClocks: new Map(Array.from({ length: 4 }, (_, index) => [index, { nonce: token, markerCode, index, + intendedMs: 1000+index*25, emittedMs: 1000+index*25, receivedAtMs: 1001+index*25 }])), + visible: Array.from({ length: stop ? count : 4 }, (_, index) => ({ index, markerCode, visibleMs: 1003+index*25 })) } }; +}; +const checkDurable = fixture => assertDurableRunEvents({ ...fixture.input, + collected: collectDurableRunEvents(fixture.rows, fixture.input.markerCode) }); + +test('durable oracle retains complete per-run evidence in H4e and H5c and permits only the canonical Stop prefix', () => { + for (const requireNativeIdentity of [false,true]) { + const fixture = durableFixture(); fixture.input.requireNativeIdentity = requireNativeIdentity; + assert.deepEqual(checkDurable(fixture), { runId: 'run-1', token: 'provider-1', markerCode: 'q7', individualDeltas: 4, + uniqueSourceIdentities: 4, matchedSourceClocks: 4, ordered: true, resultAcceptedEvents: 1 }); + } + const stopped = durableFixture({ stop: true, count: 2 }); + assert.equal(checkDurable(stopped).individualDeltas, 2); + assert.equal(checkDurable(stopped).resultAcceptedEvents, 0); + assert.equal(stopped.input.sourceClocks.size, 4, 'Raw emissions outside the durable Stop prefix remain permitted'); +}); +test('durable oracle rejects corrupted payloads, coalescing, replay identity conflicts, gaps and foreign runs', () => { + const changes = [ + fixture => { fixture.rows[0].payload.prpEvent.payload.text = '[q70] corruption'; }, + fixture => { fixture.rows[0].payload.prpEvent.payload.text = '[q70] [q71] '; }, + fixture => { fixture.rows[1].payload.prpEvent.sourceEventId = 'event-0'; fixture.rows[1].payload.prpEvent.sourceSeq = 1; }, + fixture => { fixture.rows.splice(1,1); }, fixture => { fixture.rows.splice(1,0,fixture.rows[0]); }, + fixture => { fixture.rows.reverse(); }, fixture => { fixture.rows[0].payload.prpEvent.runId = 'foreign-run'; }, + fixture => { fixture.rows[0].payload.prpEvent.sourceInstanceId = ''; }, + fixture => { fixture.rows[0].payload.prpEvent.sourceEventId = ''; }, + fixture => { fixture.rows[0].payload.prpEvent.sourceSeq = 0; }, + fixture => { fixture.rows[0].payload.prpEvent.sourceSeq = '1'; }, + ]; + for (const change of changes) { + const fixture = durableFixture(); change(fixture); + assert.throws(() => checkDurable(fixture), /Durable run run-1/, 'Failure must identify its exact run'); + } +}); +test('durable oracle rejects wrong or missing original clock joins, wrong source counts and result counts', () => { + for (const change of [ + fixture => { fixture.input.sourceClocks.delete(0); }, + fixture => { fixture.input.sourceClocks.get(0).nonce = 'foreign-source'; }, + fixture => { fixture.input.sourceClocks.get(0).markerCode = 'z9'; }, + fixture => { fixture.input.sourceClocks.get(0).index = 1; }, + fixture => { fixture.input.sourceClocks.set(9, { nonce: 'provider-1', markerCode: 'q7', index: 9 }); }, + fixture => { fixture.rows.pop(); }, fixture => { fixture.rows.push({ eventType: 'run.result.accepted' }); }, + ]) { const fixture = durableFixture(); change(fixture); assert.throws(() => checkDurable(fixture), /Durable run run-1/); } +}); +test('durable Stop oracle rejects visible markers outside the ordered durable prefix or an accepted result', () => { + const missing = durableFixture({ stop: true, count: 2 }); missing.input.visible.push({ index: 2 }); + assert.throws(() => checkDurable(missing), /visible Stop marker 2/); + const gap = durableFixture({ stop: true, count: 2 }); gap.rows.shift(); assert.throws(() => checkDurable(gap), /canonical Stop prefix/); + const result = durableFixture({ stop: true, count: 2 }); result.rows.push({ eventType: 'run.result.accepted' }); + assert.throws(() => checkDurable(result), /exact accepted result count/); +}); +test('prelude oracle preserves exact setup payloads, source tuple uniqueness, count and prelude-only selection', () => { + const rows = [0,1,2].map(index => ({ payload: { prpEvent: { eventType: 'item.delta', itemId: 'prelude-turn', + sourceInstanceId: 'source-1', sourceEventId: `event-${index}`, sourceSeq: index+1, + payload: { text: index === 2 ? ' PRELUDE_provider-1_READY ' : '.' } } } })); + const check = selected => assertPreludeEvents({ rows: selected, runId: 'run-1', token: 'provider-1', expectedEvents: 3 }); + assert.deepEqual(check([...rows,{ payload: { prpEvent: { eventType: 'item.delta', itemId: 'measured-turn' } } }]), + { individualDeltas: 3, uniqueSourceIdentities: 3 }); + assert.throws(() => check(rows.slice(1)), /Prelude run run-1/); + const bad = structuredClone(rows); bad[0].payload.prpEvent.payload.text = '..'; assert.throws(() => check(bad), /complete normalized payload/); + const replay = structuredClone(rows); Object.assign(replay[1].payload.prpEvent, { sourceEventId: 'event-0', sourceSeq: 1 }); + assert.throws(() => check(replay), /unique source identity tuples/); +}); +test('visible-clock oracle preserves original two-frame timestamps and rejects missing clocks or incomplete unique output', () => { + const fixture = durableFixture(), { visible: seenMarkers, sourceClocks } = fixture.input; + const check = (seen = seenMarkers, clocks = sourceClocks, stop = false) => assertVisibleMarkerClocks({ + seenMarkers: seen, sourceClocks: clocks, markerCode: 'q7', expectedEvents: 4, stop }); + const visible = check(); assert.deepEqual(visible.map(row => row.visibleMs), seenMarkers.map(row => row.visibleMs)); + assert.deepEqual(visible.map(row => row.emittedMs), [...sourceClocks.values()].map(row => row.emittedMs)); + assert.throws(() => check(seenMarkers, new Map()), /Missing original provider clock/); + assert.throws(() => check(seenMarkers.slice(1)), /complete unique foreground count/); + assert.throws(() => check([seenMarkers[0],seenMarkers[0],seenMarkers[2],seenMarkers[3]]), /complete unique foreground count/); + assert.equal(check(seenMarkers.slice(0,2), sourceClocks, true).length, 2); +}); diff --git a/scripts/native-ingestion-performance-gates.test.mjs b/scripts/native-ingestion-performance-gates.test.mjs index 0c0d75ee1ac..6b818188106 100644 --- a/scripts/native-ingestion-performance-gates.test.mjs +++ b/scripts/native-ingestion-performance-gates.test.mjs @@ -234,7 +234,7 @@ test("H5 assessor retains an original SLA failure assessed before its nominal st assert.throws(() => requireH5Candidate("final-b2", outcome, outcome), /every original SLA/); }); -test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires complete owning evidence", async () => { +test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires complete owning evidence", async context => { const directory = await mkdtemp(path.join(os.tmpdir(), "native-ingestion-h5-gates-")); const script = new URL("./compare-native-ingestion-browser.ts", import.meta.url); const save = async (name, value) => { const file = path.join(directory, name); await writeFile(file, JSON.stringify(value)); return file; }; @@ -271,11 +271,17 @@ test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires for (let index = 0; index < 5; index++) { const entry = {}; for (const arm of ["baseline", "candidate"]) { - const report = h5Report(arm === "baseline" || sameRevision ? baselineRevision : candidateRevision, arm === "baseline" ? 180000 : candidateLatency); + const ordinal = 2 * index + (arm === "baseline" ? index % 2 : index % 2 ? 0 : 1); + const startedAtMs = Date.parse(sameRevision ? "2026-11-02T00:00:00.000Z" : "2026-11-03T00:00:00.000Z") + ordinal * 2000000; + const ownedIdPrefix = createHash("sha256").update(prefix).digest("hex").slice(0,6) + ordinal.toString(16).padStart(2,"0"); + const report = JSON.parse(JSON.stringify(h5Report(arm === "baseline" || sameRevision ? baselineRevision : candidateRevision, + arm === "baseline" ? 180000 : candidateLatency)).replaceAll("00000000-", `${ownedIdPrefix}-`), (key, value) => + typeof value === "number" && (key.endsWith("AtMs") || ["epochMs", "deadlineMs", "readinessDeadlineMs", "visibleMs", "intendedMs", "emittedMs"].includes(key)) ? value + startedAtMs : value); + report.bodyFinishedAtMs = startedAtMs + duration - 1; const owner = { timeout: 1200000, expectedStatus: "passed", results: [{ status: "passed", retry: 0, - duration, startTime: "2026-11-03T00:00:00.000Z", errors: [], attachments: [] }] }; + duration, startTime: new Date(startedAtMs).toISOString(), errors: [], attachments: [] }] }; if (index === 0 && arm === "candidate") mutate?.(report, owner); - mutateArm?.(report, { index, arm }); + mutateArm?.(report, { index, arm, owner }); owner.results[0].attachments.push({ name: "native-ingestion-evidence", contentType: "application/json", body: Buffer.from(JSON.stringify(report)).toString("base64") }); entry[`${arm}File`] = await save(`${prefix}-${index}-${arm}.json`, report); @@ -299,11 +305,90 @@ test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires const originalInventory = await inventory(proof, originalProof, "original-h4c"); const finalProofArgs = [`--original-h4e-proof=${originalProof}`, `--original-h4e-inventory=${originalInventory}`]; const noise = path.join(directory, "aa.json"); - const aa = run(["--phase=aa", `--pairs=${await manifest("aa", 181000, { sameRevision: true })}`, `--output=${noise}`]); + const aaManifest = await manifest("aa", 181000, { sameRevision: true }); + const aa = run(["--phase=aa", `--pairs=${aaManifest}`, `--output=${noise}`]); assert.equal(aa.status, 0, aa.stderr); const aaReport = JSON.parse(await readFile(noise, "utf8")); assert.equal(aaReport.h5Assessments.length, 5); + assert.equal(aaReport.measurements.filter(item => item.metric === "h5.independentDrain.elapsedFromEpochMs").length, 5, + "Every non-Stop case needs its own verified-completion A/A noise"); + assert(aaReport.measurements.find(item => item.name === "burst512" && item.metric === "h5.independentDrain.elapsedFromEpochMs").aaNoiseEnvelopeRelative > 0); assert.deepEqual(aaReport.h5Assessments[0].baseline.misses, ["burst512:streaming120"]); + assert.equal(aaReport.aaSuitability.passed, true); assert.equal(aaReport.armChronology.length, 10); + assert.equal(new Set(aaReport.armChronology.map(arm => arm.companyId)).size, 10); + for (const [name, mutateArm, expectation] of [ + ["aa-paired-jitter", (report, { arm, index }) => { if (arm === "candidate" && index === 4) report.cases[0].firstVisibleAfterEmissionMs = 1300; }, "paired"], + ["aa-chronological-drift", (report, { index }) => { report.cases[0].firstVisibleAfterEmissionMs = 1000 + index * 60; }, "chronological"], + ["aa-role-bias", (report, { arm }) => { report.cases[0].firstVisibleAfterEmissionMs = arm === "candidate" ? 1120 : 1000; }, "role"], + ]) { + const output = path.join(directory, `${name}.json`); + const attempt = run(["--phase=aa", `--pairs=${await manifest(name, 180000, { sameRevision: true, mutateArm })}`, `--output=${output}`]); + assert.notEqual(attempt.status, 0, name); assert.match(attempt.stderr, /A\/A suitability failed/); + const unsuitable = JSON.parse(await readFile(output, "utf8")); + assert.equal(unsuitable.aaSuitability.passed, false); assert.equal(unsuitable.armChronology.length, 10); + const metric = unsuitable.measurements.find(item => item.name === "paced" && item.metric === "firstVisibleAfterEmissionMs").aaSuitability; + if (expectation === "paired") assert(metric.pairedDifferencesMs.some((value,index) => value > metric.pairedLimitsMs[index])); + if (expectation === "chronological") { assert.equal(metric.observedNoiseAbsoluteMs, 0); assert(metric.chronologicalMedianDriftMs > metric.chronologicalDriftLimitMs); } + if (expectation === "role") assert(metric.roleMedianDriftMs > metric.roleDriftLimitMs); + } + for (const [name, mutateArm] of [ + ["aa-missing-owner-time", (_report, { arm, index, owner }) => { if (arm === "candidate" && index === 0) delete owner.results[0].startTime; }], + ["aa-overlap", (_report, { arm, index, owner }) => { if (arm === "candidate" && index === 0) owner.results[0].startTime = "2026-11-02T00:00:00.001Z"; }], + ["aa-wrong-order", (_report, { arm, index, owner }) => { if (index === 1) owner.results[0].startTime = new Date(Date.parse("2026-11-02T00:00:00.000Z") + (arm === "baseline" ? 4 : 5) * 1000000).toISOString(); }], + ["aa-reused-company-run", (report) => { const prefix = report.companyId.slice(0,8); Object.assign(report, JSON.parse(JSON.stringify(report).replaceAll(`${prefix}-`, "00000000-"))); }], + ]) { + const attempt = run(["--phase=aa", `--pairs=${await manifest(name, 180000, { sameRevision: true, mutateArm })}`]); + assert.notEqual(attempt.status, 0, name); + } + for (const [name, forge] of [ + ["forged-absolute-noise", noise => { noise.measurements[0].aaNoiseEnvelopeAbsoluteMs = 9999; }], + ["forged-relative-noise", noise => { noise.measurements[0].aaNoiseEnvelopeRelative = 9; }], + ["forged-representation", noise => { noise.measurements[0].representationLimited = !noise.measurements[0].representationLimited; }], + ["forged-suitability", noise => { noise.measurements[0].aaSuitability.pairedLimitsMs[0] = 9999; }], + ["forged-sample", noise => { noise.measurements[0].samples[0].improvementMs++; }], + ["forged-rule", noise => { noise.aaSuitability.rule.contrastRepresentationFloorMs = 1000; }], + ]) { + const forged = structuredClone(aaReport); forge(forged); + const attempt = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression", + `--pairs=${await manifest(name, 180000, { candidateRevision: "B1" })}`, `--noise-report=${await save(`${name}-noise.json`, forged)}`, "--require-performance"]); + assert.notEqual(attempt.status, 0, name); assert.match(attempt.stderr, /forged|rule changed/); + } + for (const repeat of [1,2]) { + const shortNoise = path.join(directory, `short-${repeat}-noise.json`); + const aa = run(["--phase=aa", `--pairs=${await manifest(`short-${repeat}-aa`, 180000, { sameRevision: true, + mutateArm: (report, { arm }) => { report.cases[0].firstVisibleAfterEmissionMs = arm === "candidate" ? repeat : 1; } })}`, `--output=${shortNoise}`]); + assert.equal(aa.status, 0, aa.stderr); + const calibrated = JSON.parse(await readFile(shortNoise, "utf8")); + const control = calibrated.measurements.find(item => item.name === "paced" && item.metric === "firstVisibleAfterEmissionMs"); + assert.equal(control.representationLimited, true); assert.equal(control.percentageGainClaimAllowed, false); + assert.equal(control.medianPairedImprovementRelative, null); assert.equal(control.aaNoiseEnvelopeAbsoluteMs, repeat - 1); + const output = path.join(directory, `short-${repeat}-regression.json`); + const attempt = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression", + `--pairs=${await manifest(`short-${repeat}-regression`, 180000, { candidateRevision: "B1", mutateArm: (report, { arm }) => { + report.cases[0].firstVisibleAfterEmissionMs = repeat === 1 ? arm === "candidate" ? 2 : 1 : arm === "candidate" ? 12 : 10; + } })}`, `--noise-report=${shortNoise}`, `--output=${output}`, "--require-performance"]); + assert.notEqual(attempt.status, 0, "The suitability floor must not pad actual observed absolute noise"); + const compared = JSON.parse(await readFile(output, "utf8")); + const gate = compared.gates.find(item => item.name === "paced" && item.metric === "firstVisibleAfterEmissionMs"); + assert.equal(gate.noiseEnvelopeAbsoluteMs, repeat - 1); assert.equal(gate.representationLimited, true); assert.equal(gate.passed, false); + if (repeat === 2) { + const improving = path.join(directory, "short-calibrated-improvement.json"); + const changed = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression", + `--pairs=${await manifest("short-calibrated-improvement", 180000, { candidateRevision: "B1", mutateArm: (report, { arm }) => { + report.cases[0].firstVisibleAfterEmissionMs = arm === "candidate" ? 1 : 100; + } })}`, `--noise-report=${shortNoise}`, `--output=${improving}`, "--require-performance"]); + assert.equal(changed.status, 0, changed.stderr); + const measured = JSON.parse(await readFile(improving, "utf8")).measurements.find(item => item.name === "paced" && item.metric === "firstVisibleAfterEmissionMs"); + assert.equal(measured.representationLimited, true); assert.equal(measured.percentageGainClaimAllowed, false); + assert.equal(measured.medianPairedImprovementRelative, null, "Short calibration cannot turn into a percentage gain claim"); + } + } + const replayedOwners = JSON.parse(await readFile(await manifest("reused-aa-owner", 180000, { candidateRevision: "B1" }), "utf8")); + const oldOwners = JSON.parse(await readFile(aaManifest, "utf8")); + for (const field of ["baselineFile", "baselinePlaywrightFile", "baselineInventoryFile"]) replayedOwners[0][field] = oldOwners[0][field]; + const replay = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression", + `--pairs=${await save("reused-aa-owner-pairs.json", replayedOwners)}`, `--noise-report=${noise}`, "--require-performance"]); + assert.notEqual(replay.status, 0); assert.match(replay.stderr, /calibration|reus/); const cursor = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression", `--pairs=${await manifest("cursor", 180000, { candidateRevision: "B1" })}`, `--noise-report=${noise}`, "--require-acceptance"]); assert.equal(cursor.status, 0, cursor.stderr); @@ -318,6 +403,69 @@ test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires const goodPairs = await manifest("good", 1000); const good = run(["--candidate-policy=final-b2", `--pairs=${goodPairs}`, `--noise-report=${noise}`, ...finalProofArgs, "--require-acceptance"]); assert.equal(good.status, 0, good.stderr); + const completionRegressionPairs = await manifest("verified-drain-regression", 1000, { + mutateArm: (report, { arm }) => { + if (arm !== "candidate") return; + const item = report.cases[3]; + Object.assign(item.h5.independentDrain, { elapsedFromEpochMs: 200000, verifiedAtMs: item.epochMs + 200000 }); + } }); + const completionRegressionOutput = path.join(directory, "verified-drain-regression.json"); + const completionRegression = run(["--candidate-policy=final-b2", `--pairs=${completionRegressionPairs}`, + `--noise-report=${noise}`, `--output=${completionRegressionOutput}`, "--require-performance"]); + const completionRegressionReport = JSON.parse(await readFile(completionRegressionOutput, "utf8")); + context.diagnostic(JSON.stringify({ regression: "browser and terminal faster but independently verified completion slower", + comparatorSha256: completionRegressionReport.comparatorSha256, exit: completionRegression.status, + performancePassed: completionRegressionReport.performancePassed, baselineVerifiedMs: 180010, candidateVerifiedMs: 200000, + verifiedCompletionGate: completionRegressionReport.gates.find(gate => gate.name === "burst512" + && gate.metric === "h5.independentDrain.elapsedFromEpochMs") ?? null })); + assert.notEqual(completionRegression.status, 0, "Faster browser/terminal must not conceal slower fully verified completion"); + const regressionGate = completionRegressionReport.gates.find(gate => gate.name === "burst512" + && gate.metric === "h5.independentDrain.elapsedFromEpochMs"); + assert.equal(regressionGate.improvementRequired, true); assert.equal(regressionGate.passed, false); + assert(completionRegressionReport.gates.filter(gate => gate.name === "burst512" && + ["emissionToVisible.p95Ms", "intendedToVisible.p95Ms", "drainThroughTerminalMs"].includes(gate.metric)).every(gate => gate.passed)); + const goodReport = JSON.parse(good.stdout); + const rate = goodReport.verifiedCompletionRates.find(item => item.name === "burst512"); + assert.equal(rate.samples[0].measuredDeltas, 51200); + assert.equal(rate.samples[0].baselineDeltasPerSecond, 51200000 / 180010); + assert.equal(rate.samples[0].candidateDeltasPerSecond, 51200000 / 1010); + assert(rate.candidateMedianDeltasPerSecond > rate.baselineMedianDeltasPerSecond); + for (const beyondNoise of [3,4]) { + const output = path.join(directory, `verified-drain-${beyondNoise}-wins.json`); + const attempt = run(["--candidate-policy=final-b2", `--pairs=${await manifest(`verified-drain-${beyondNoise}-wins`, 1000, { + mutateArm: (report, { arm, index }) => { + if (arm !== "candidate") return; + const item = report.cases[3], elapsed = index < beyondNoise ? 90000 : 179500; + Object.assign(item.h5.independentDrain, { elapsedFromEpochMs: elapsed, verifiedAtMs: item.epochMs + elapsed }); + } })}`, `--noise-report=${noise}`, `--output=${output}`, "--require-performance"]); + assert.equal(attempt.status === 0, beyondNoise === 4, attempt.stderr); + const compared = JSON.parse(await readFile(output, "utf8")); + const gate = compared.gates.find(item => item.name === "burst512" && item.metric === "h5.independentDrain.elapsedFromEpochMs"); + assert.equal(gate.noiseExceedingPairedWins, beyondNoise); assert.equal(gate.requiredPairedWins, 4); + assert.equal(gate.passed, beyondNoise === 4); + } + const zeroNoise = path.join(directory, "zero-noise.json"); + const zeroAA = run(["--phase=aa", `--pairs=${await manifest("zero-noise-aa", 180000, { sameRevision: true })}`, `--output=${zeroNoise}`]); + assert.equal(zeroAA.status, 0, zeroAA.stderr); + for (const resolvedPairs of [0,3,4]) { + const output = path.join(directory, `timestamp-gain-${resolvedPairs}.json`); + const attempt = run(["--candidate-policy=final-b2", `--pairs=${await manifest(`timestamp-gain-${resolvedPairs}`, 1000, { + mutateArm: (report, { arm, index }) => { + if (arm !== "candidate") return; + const item = report.cases[3], gain = resolvedPairs === 0 ? 1 : index < resolvedPairs ? 3 : 2; + Object.assign(item.h5.independentDrain, { elapsedFromEpochMs: 180010 - gain, verifiedAtMs: item.epochMs + 180010 - gain }); + } })}`, `--noise-report=${zeroNoise}`, `--output=${output}`, "--require-performance"]); + assert.equal(attempt.status === 0, resolvedPairs === 4, attempt.stderr); + const gate = JSON.parse(await readFile(output, "utf8")).gates.find(item => item.name === "burst512" + && item.metric === "h5.independentDrain.elapsedFromEpochMs"); + assert.equal(gate.noiseExceedingPairedWins, resolvedPairs); assert.equal(gate.requiredAbsoluteImprovementAboveMs, 2); + } + const cursorCompletionRegression = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression", + `--pairs=${await manifest("cursor-verified-drain-regression", 180000, { candidateRevision: "B1", mutateArm: (report, { arm }) => { + if (arm !== "candidate") return; + const item = report.cases[3]; Object.assign(item.h5.independentDrain, { elapsedFromEpochMs: 200000, verifiedAtMs: item.epochMs + 200000 }); + } })}`, `--noise-report=${noise}`, "--require-performance"]); + assert.notEqual(cursorCompletionRegression.status, 0, "Cursor nonregression must include fully verified completion"); const wrongPolicy = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression", `--pairs=${goodPairs}`, `--noise-report=${noise}`, "--require-acceptance"]); assert.notEqual(wrongPolicy.status, 0); assert.match(wrongPolicy.stderr, /Cursor policy requires B0→B1/); @@ -447,6 +595,8 @@ test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires ["stop-recorded-provider-failure", report => { report.cases[5].recordedFailures.rows[0].current_error_code = "provider_transport_failed"; }, /recorded failed run must have recovered/], ["censored", report => { report.cases[3].h5.streaming.observation.censored = true; }, /false/], ["terminal-failed", report => { report.cases[3].terminalStatuses[99].status = "failed"; }, /successful drainage/], + ["missing-independent-drain", report => { delete report.cases[3].h5.independentDrain; }, /independentDrain|verifiedAtMs|Cannot read/], + ["tampered-independent-drain", report => { report.cases[3].h5.independentDrain.elapsedFromEpochMs++; }, /strictly equal/], ["result-missing", report => { report.cases[3].durable[99].resultAcceptedEvents = 0; }, /persisted result/], ["missing-delta", report => { report.cases[3].durable[99].individualDeltas--; }, /individualDeltas|result.events|512/], ["wrong-workload", report => { report.workloadSha256 = "old"; }, /registration changed/], diff --git a/scripts/native-ingestion/README.md b/scripts/native-ingestion/README.md index 3fc66ed3c0f..1a3d87a7300 100644 --- a/scripts/native-ingestion/README.md +++ b/scripts/native-ingestion/README.md @@ -1,6 +1,6 @@ # Native ingestion experiment helpers -For fresh cursor results and the current interleaved source reproduction, see [the new source report](../../doc/benchmarks/2026-11-03-native-event-ingestion-interleaved.md). The frozen runner below preserves the earlier experiment. Current application performance requires the independently frozen [H5c driver and helpers](h5c/README.md), successful qualification and fresh comparisons; its gates remain open. The older H4c runner below remains historical. +For fresh cursor results and the current interleaved source reproduction, see [the new source report](../../doc/benchmarks/2026-11-03-native-event-ingestion-interleaved.md). The frozen runner below preserves the earlier experiment. Current application reproduction uses the separately frozen [H5d driver and helpers](h5d/README.md) and [v3 blocked-write Stop companion](blocked-stop/README.md). Each requires its own immutable checkout and locked install: main `2df9897e` uses measurement `e1d8930c…`, while companion `bc33f09c` inherits H5c measurement `2265aabe…`. This mixed delivery tree and its lock are not an executed driver. Functional qualifications passed; no application A/A or performance gain is established at the recorded 09:30 UTC cutoff. The versioned [H5c helpers](h5c/README.md), H5b and older H4c runner below retain historical evidence. Each current campaign has 50 fresh arms and independent noise gates; source reproduction commands are in its linked README. These helpers reproduce the experiment registered on November 3, 2026. They retain its immutable application revisions, harness revision, workload hashes, diff --git a/scripts/native-ingestion/blocked-stop/README.md b/scripts/native-ingestion/blocked-stop/README.md new file mode 100644 index 00000000000..1090a12904b --- /dev/null +++ b/scripts/native-ingestion/blocked-stop/README.md @@ -0,0 +1,117 @@ +# Frozen blocked-write Stop companion + +These are exact source snapshots of the independently reviewed companion and +orchestrator. `sources.json` records every hash. Packaging itself proves no runtime +qualification, control nonregression or application performance gain. The exact +v3 helper is SHA256 `5748b6c1d271cf9ab270d5473873b85da295ffb03bb55ebc63693369c9b1a29d`; +source freeze receipt SHA256 is `210002a6699059ff9b5f74a49558c9f7f0f5866cd9ed3f786507b3ee12fc7b38`. + +Use independent driver tag `native-ingestion-blocked-stop-v3`, annotated object +`a49c24dc2a690764d1cfcc080077f90485218df7`, commit +`bc33f09cae26404352848169d92a20c89480a5f5`. It inherits unchanged H5c `e431ec63` +and lock SHA256 `54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8`. +The delivery checkout has a different lock and must not replace that frozen driver. +Main H5d is the separate frozen `2df9897e` driver (measurement `e1d8930c…`); +this companion requires its inherited H5c measurement `2265aabe…`. The mixed +delivery checkout cannot execute either frozen protocol in place of its own driver. The [frozen protocol](../../../tests/e2e/fixtures/native-ingestion-blocked-stop.md) +defines the real row-lock, durable intent, physical Stop, transcript and noise gates. + +The helper imports only six checked exports from the unchanged H5c helper, +SHA256 `6b6376b01de6cdc15ca05343a0bdd5c00424f1d6c546b9b4d61cd76716b253c4`: +qualified build/live inventory, preferred UI absence, physical quiet and raw seals. +It never invokes H5c's stage plan or release loader. Copy that exact dependency +from `../h5c/` to its recorded path before using a companion release. + +## Recorded qualification and retained failures + +As of 2026-11-04 09:30 UTC, ONE v3 functional qualification passed: 1,000 prelude +deltas, a 12-delta canonical Stop prefix with zero results, 1,010 ms held across +15 samples, own committed intent/ACK audits, both exact PIDs and both groups +absent, a bound latest server retirement receipt with cleared identity, released +lease, persisted reload and strict quiet. No manual signals were used. This is +not an accepted A/A or performance sample; the separate 50 fresh arms remain. +The immutable protocol document describes the prospective state at source freeze. + +V1 failed before epoch on an unsupported shared-PGID assumption; API200 and port +release did not establish physical quiet, and exact owned cleanup was retained. +V2 failed an unconditional retirement-receipt assertion although retained original +identity after physical exit is supported. Its later transcript/reload/lease gate +was not reached. V3 accepts only the explicit retained or cleared identity contracts; +all independent physical, lease, audited ACK and original-birth gates remain. +Keep both failed trials, old tags/helpers and raw evidence. No failed timing data +is reused or regraded as qualification. + +## Independent setup and source-only checks + +The benchmark VM uses Node **v26.4.0** and pnpm **9.15.4**. Node 24 was used for +local Mac source checks; it is not the recorded VM runtime. Normally install and +build measured apps independently. Preserve actual qualified build receipts, +native inputs and all four inventoried artifact roots. No new B2 candidate or +qualified optimized build is supplied here; the old `ca7cba17` is rejected as final B2. + +Run from this tools checkout on a disposable host. The driver path must be new: + +```sh +TOOLS_CHECKOUT="$(git rev-parse --show-toplevel)" +DRIVER_ROOT=/home/ubuntu/native-ingestion/harness-blocked-stop-v3 +HELPERS_ROOT=/home/ubuntu/native-ingestion/helpers +git -C "$TOOLS_CHECKOUT" show-ref --verify --quiet refs/tags/native-ingestion-blocked-stop-v3 || + git -C "$TOOLS_CHECKOUT" fetch origin tag native-ingestion-blocked-stop-v3 +test "$(git -C "$TOOLS_CHECKOUT" rev-parse 'native-ingestion-blocked-stop-v3^{commit}')" = bc33f09cae26404352848169d92a20c89480a5f5 +git -C "$TOOLS_CHECKOUT" worktree add --detach "$DRIVER_ROOT" native-ingestion-blocked-stop-v3 +pnpm --dir "$DRIVER_ROOT" install --frozen-lockfile --offline --ignore-scripts +mkdir -p "$HELPERS_ROOT" +cp "$TOOLS_CHECKOUT/scripts/native-ingestion/h5c/native-ingestion-run-h5c-browser.mjs" "$HELPERS_ROOT/native-ingestion-run-h5c-browser.mjs" +cp "$TOOLS_CHECKOUT/scripts/native-ingestion/blocked-stop/native-ingestion-run-blocked-stop-v3.mjs" "$HELPERS_ROOT/native-ingestion-run-blocked-stop-v3.mjs" +node "$HELPERS_ROOT/native-ingestion-run-blocked-stop-v3.mjs" +node "$DRIVER_ROOT/tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs" --registration +``` + +The last two commands are dry runs and start no app or arm. Verify both copied +helpers against `sources.json`. The release's `h5HelperFile` is the exact recorded +`/home/ubuntu/native-ingestion/helpers/native-ingestion-run-h5c-browser.mjs` path. +Keep the frozen driver install independent from every app's normal locked build. + +Capture actual runtime versions and executable locations before preparing a +release: `node --version`, `node -p process.execPath`, `pnpm --version`, +`rustc --version`, `cargo --version`, and the actual PostgreSQL and Chromium +executables with `--version`. Use exact absolute executable, argument and output +values in `runtimeChecks`; do not guess provider/browser/database versions. Before +each arm, the helper additionally resolves bare PATH `node` under the exact app +cwd and launch environment and verifies its canonical executable and version. + +## Explicit release and evidence gates + +The dry run prints all required release fields. A release binds authorization, +stage/output, helper hash, exact driver revision and nine hashes, inherited H5c +hash/lock, workload and precision rules, actual runtime checks, qualified app +builds and earlier report/noise file hashes. Source validation is separate from +runtime qualification: + +```sh +node "$HELPERS_ROOT/native-ingestion-run-blocked-stop-v3.mjs" \ + --release="$RELEASE_JSON" --release-sha256="$RELEASE_SHA256" \ + --stage=qualification-b0 --output-root="$EVIDENCE_ROOT" --validate-release +``` + +After the concrete release and quiet window are reviewed, replace +`--validate-release` with `--execute`. First run only `qualification-b0`: one +actual journey, owning Playwright pass, exact attached raw bytes, full functional +oracle, actual build/UI/native inventory, independent cleanup and sealed proof. +Inspect that evidence before separately releasing any calibration. + +Then release `b0-aa`, `b1-aa`, `b0-b1`, `b1-b2`, `b0-b2` in order. Each has five +fresh alternating pairs, for 50 timing arms. Qualification is never reused as a +timing arm. Freeze both suitable A/A noise JSON files and their actual hashes +before further product optimization or comparisons. Unsuitable calibration +retains every raw arm and writes its noise artifact before stopping. No deadline +or precision budget is enlarged after observation. Each owning test keeps its +900-second limit; the 30-minute watchdog only bounds resources. + +A passed arm requires immutable source, compiled/native/UI checks, one unretried +recording-off test, functional lock/intent/audit/process/lease/transcript proof, +exact raw/owner/build seals and physical cleanup. The helper revalidates prior +proofs and compares only against that companion's frozen noise. Preserve failed +or censored runs and stop the stage; it never kills unrelated observed processes. +This deterministic local-provider companion does not establish live Cloud Stop +acceptance or replace the main application's performance measurements. diff --git a/scripts/native-ingestion/blocked-stop/native-ingestion-run-blocked-stop-v3.mjs b/scripts/native-ingestion/blocked-stop/native-ingestion-run-blocked-stop-v3.mjs new file mode 100644 index 00000000000..560f8dc4eeb --- /dev/null +++ b/scripts/native-ingestion/blocked-stop/native-ingestion-run-blocked-stop-v3.mjs @@ -0,0 +1,233 @@ +// DRAFT: no execution is released by this source. Default invocation is a dry run. +import assert from 'node:assert/strict'; +import { readFile, writeFile, mkdir, open, realpath, lstat } from 'node:fs/promises'; +import { spawn, execFileSync } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import net from 'node:net'; +import path from 'node:path'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +const SELF=fileURLToPath(import.meta.url), H5SHA='6b6376b01de6cdc15ca05343a0bdd5c00424f1d6c546b9b4d61cd76716b253c4'; +const COMPANION_REVISION='bc33f09cae26404352848169d92a20c89480a5f5'; +const INHERITED='e431ec63d41fe90c6fd68bd2fc2bb969feeca917', LOCK='54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8'; +const H5MEASUREMENT='2265aabe9c7344f5d4b9c8b08a5c6bbd5571a6e2de9ae7b72a8d1733250e609f'; +export const STAGES=['qualification-b0','b0-aa','b1-aa','b0-b1','b1-b2','b0-b2']; +const CORE=['native-event-ingestion-blocked-stop.journey.ts','playwright-native-ingestion-blocked-stop.config.ts', + 'fixtures/native-ingestion-blocked-stop-authority.ts','fixtures/native-ingestion-blocked-stop-clock.ts','fixtures/native-ingestion-blocked-stop-db.ts','fixtures/native-ingestion-blocked-stop-protocol.mjs']; +const COMPANION=[...CORE,'fixtures/native-ingestion-blocked-stop-protocol.test.mjs','fixtures/native-ingestion-blocked-stop-validate.mjs','fixtures/native-ingestion-blocked-stop.md']; +const H5=['native-event-ingestion.journey.ts','playwright-native-ingestion.config.ts','fixtures/in-feed-codex.mjs','fixtures/native-ingestion-bin/codex', + 'fixtures/composer-stop-codex-loader.mjs','fixtures/native-ingestion-bootstrap.mjs','fixtures/native-ingestion-remote-server.mjs','fixtures/native-ingestion-protocol-qualification.mjs','fixtures/native-ingestion-h5.mjs']; +const CONFIG='tests/e2e/playwright-native-ingestion-blocked-stop.config.ts', SCHEMA='paperclip.native-ingestion.blocked-stop-series.v1'; +const AMBIENT=['DATABASE_URL','PAPERCLIP_RUNNER_STATE_DIR','PAPERCLIP_INGESTION_SERVER_COMMAND','PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_DIR', + 'PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_FS','PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_INSPECTOR','PAPERCLIP_INGESTION_BROWSER_CAPTURE','NODE_OPTIONS', + 'PAPERCLIP_INGESTION_SMOKE','PAPERCLIP_INGESTION_OBSERVATION_MODE']; +const hash=bytes=>createHash('sha256').update(bytes).digest('hex'), json=async file=>JSON.parse(await readFile(file,'utf8')); +const save=(file,value)=>writeFile(file,JSON.stringify(value,null,2)+'\n',{flag:'wx'}); +const option=name=>process.argv.find(value=>value.startsWith(`--${name}=`))?.slice(name.length+3); +const absolute=value=>assert.ok(typeof value==='string'&&path.isAbsolute(value),'An absolute path is required'); +const sha=value=>assert.match(value??'',/^[a-f0-9]{64}$/), revision=value=>assert.match(value??'',/^[a-f0-9]{40}$/); +const git=(root,...args)=>execFileSync('git',args,{cwd:root,encoding:'utf8'}).trim(); +export async function pinned(file,expected){absolute(file);sha(expected);const bytes=await readFile(file);assert.equal(hash(bytes),expected);return bytes;} +export function schedule(stage,registered){ + assert.ok(STAGES.includes(stage));if(stage==='qualification-b0')return [{stage,pair:0,index:0,variant:'b0',role:'qualification'}]; + const rows=registered.filter(row=>row.stage===stage);assert.equal(rows.length,10); + const [a,b]=stage==='b0-aa'?['b0','b0']:stage==='b1-aa'?['b1','b1']:stage.split('-'); + for(let pair=1;pair<=5;pair++)for(let index=0;index<2;index++){ + const row=rows[(pair-1)*2+index],role=(pair%2===1?index===0:index===1)?'baseline':'candidate'; + assert.deepEqual(row,{stage,pair,index,variant:role==='baseline'?a:b,role}); + }return rows; +} +export function assertCompanionRevision(value){assert.equal(value,COMPANION_REVISION,'Use the exact frozen companionv3 revision');} +async function source(context){ + const {r}=context;assertCompanionRevision(r.harnessRevision);await pinned(context.releaseFile,context.releaseSha256);await pinned(SELF,r.orchestratorSha256);await pinned(r.h5HelperFile,H5SHA); + assert.equal(git(r.harnessRoot,'rev-parse','HEAD'),r.harnessRevision);assert.equal(git(r.harnessRoot,'status','--short'),''); + execFileSync('git',['merge-base','--is-ancestor',INHERITED,r.harnessRevision],{cwd:r.harnessRoot}); + assert.deepEqual(git(r.harnessRoot,'diff','--name-only',INHERITED,r.harnessRevision).split('\n').sort(),COMPANION.map(f=>'tests/e2e/'+f).sort()); + assert.equal(hash(await readFile(path.join(r.harnessRoot,'pnpm-lock.yaml'))),LOCK); + assert.equal((await lstat(path.join(r.harnessRoot,'node_modules'))).isSymbolicLink(),false); + assert.deepEqual(Object.keys(r.companionFiles).sort(),[...COMPANION].sort()); + for(const [file,digest]of Object.entries(r.companionFiles))await pinned(path.join(r.harnessRoot,'tests/e2e',file),digest); + assert.equal(hash(Buffer.concat(await Promise.all(H5.map(f=>readFile(path.join(r.harnessRoot,'tests/e2e',f)))))),H5MEASUREMENT); +} +export async function load(releaseFile,expectedSha,stage,out){ + absolute(out);const bytes=await pinned(releaseFile,expectedSha),r=JSON.parse(bytes);revision(r.harnessRevision);assertCompanionRevision(r.harnessRevision); + assert.equal(r.authorized,true);assert.equal(r.stage,stage);assert.equal(r.outputRoot,out);assert.equal(r.h5HelperSha256,H5SHA); + assert.equal(r.resourceWatchdogMillis,1800000);assert.equal(r.inheritedH5cRevision,INHERITED);assert.equal(r.lockSha256,LOCK); + assert.equal(r.runtimeChecks?.node?.expected,'v26.4.0');assert.equal(r.runtimeChecks?.pnpm?.expected,'9.15.4'); + const context={r,out,releaseFile,releaseSha256:hash(bytes)};await source(context); + const h=await import(pathToFileURL(r.h5HelperFile).href),p=await import(pathToFileURL(path.join(r.harnessRoot,'tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.mjs')).href); + assert.equal(p.PROTOCOL,'blocked-stop-history-v1');assert.equal(p.WORKLOAD_SHA256,r.workloadSha256);sha(r.precisionRuleSha256);assert.equal(p.PRECISION_RULE_SHA256,r.precisionRuleSha256); + assert.equal(typeof p.assertNoiseEnvelope,'function');context.h=h;context.p=p;context.rows=schedule(stage,p.registration()); + const needed=STAGES.indexOf(stage)<2?['B0']:STAGES.indexOf(stage)<4?['B0','B1']:['B0','B1','B2']; + for(const variant of needed)await h.validateVariantBuild(variant,r.perVariantBuilds?.[variant]); + return context; +} +async function runtime(r){ + assert.equal(process.version,'v26.4.0');const values={}; + assert.deepEqual(Object.keys(r.runtimeChecks).sort(),['cargo','chromium','node','pnpm','postgres','rustc']); + for(const [name,check]of Object.entries(r.runtimeChecks)){ + absolute(check.executable);assert.ok(Array.isArray(check.args)&&check.args.every(v=>typeof v==='string'));assert.ok(check.expected); + const actual=execFileSync(check.executable,check.args,{encoding:'utf8',timeout:10000}).trim();assert.equal(actual,check.expected);values[name]={...check,actual}; + }assert.equal(await realpath(r.runtimeChecks.node.executable),await realpath(process.execPath));return values; +} +export async function assertPathNode(environment,approvedExecutable,workingDirectory){ + absolute(workingDirectory); + const actual=JSON.parse(execFileSync('node',['-p','JSON.stringify({executable:process.execPath,version:process.version})'],{env:environment,cwd:workingDirectory,encoding:'utf8',timeout:10000})); + assert.equal(actual.version,'v26.4.0','PATH node version differs from approved runtime'); + const canonicalExecutable=await realpath(actual.executable),approvedCanonicalExecutable=await realpath(approvedExecutable); + assert.equal(canonicalExecutable,approvedCanonicalExecutable,'PATH node executable differs from approved runtime'); + assert.equal(approvedCanonicalExecutable,await realpath(process.execPath),'Approved node differs from orchestrator runtime'); + return {...actual,canonicalExecutable,approvedCanonicalExecutable,PATH:environment.PATH,cwd:workingDirectory,canonicalCwd:await realpath(workingDirectory)}; +} +async function buildCheck(context,variant){ + const b=context.r.perVariantBuilds[variant],q=await context.h.validateVariantBuild(variant,b); + assert.equal(git(b.appRoot,'rev-parse','HEAD'),b.revision);assert.equal(git(b.appRoot,'status','--short'),''); + assert.equal((await json(path.join(b.appRoot,'server/dist/build-info.json'))).commit,b.revision); + const target=await realpath(path.join(b.appRoot,'packages/paperclip-runner/runner/target')),binary=path.join(target,'release/paperclip-runnerd'); + assert.equal(target,q.receipt.nativeStageHashes.find(row=>row.variant===variant).targetRealPath); + assert.equal(await realpath(path.join(b.appRoot,'packages/paperclip-runner/runner/target/release/paperclip-runnerd')),binary);await pinned(binary,b.binarySha256); + for(const [file,object]of Object.entries(q.receipt.rustInputObjects))assert.equal(git(b.appRoot,'rev-parse',`${b.revision}:packages/paperclip-runner/${file}`),object); + return {qualified:q,live:await context.h.validateActualInventory(b,q.manifest),ui:await context.h.assertNoPreferredUiShadow(b.appRoot),binary}; +} +async function portProbe(port=0){ + const server=net.createServer();await new Promise((resolve,reject)=>{server.once('error',reject);server.listen(port,'127.0.0.1',resolve);}); + const actual=server.address().port;await new Promise(resolve=>server.close(resolve));return actual; +} +async function launch(context,directory,env,launchEnvironment){ + const record={startedAtMs:Date.now(),executable:process.execPath,args:['node_modules/@playwright/test/cli.js','test','--config',CONFIG],cwd:context.r.harnessRoot,environmentOverrides:env,resourceWatchdogMillis:1800000}; + const stream=(await open(path.join(directory,'playwright.log'),'wx')).createWriteStream(),child=spawn(record.executable,record.args,{cwd:record.cwd,env:launchEnvironment,stdio:['ignore','pipe','pipe']}); + record.childPid=child.pid;child.stdout.pipe(stream,{end:false});child.stderr.pipe(stream,{end:false});let force; + const interrupt=()=>child.kill('SIGINT'),terminate=()=>child.kill('SIGTERM');process.once('SIGINT',interrupt);process.once('SIGTERM',terminate); + const timer=setTimeout(()=>{record.watchdogExpired=true;child.kill('SIGTERM');force=setTimeout(()=>{record.forcedDriverKill=true;child.kill('SIGKILL');},30000);},1800000); + try{Object.assign(record,await new Promise((resolve,reject)=>{child.once('error',reject);child.once('close',(code,signal)=>resolve({code,signal}));}));} + catch(error){record.launchError=String(error);throw error;}finally{ + clearTimeout(timer);clearTimeout(force);process.off('SIGINT',interrupt);process.off('SIGTERM',terminate);await new Promise(resolve=>stream.end(resolve)); + record.finishedAtMs=Date.now();record.logSha256=hash(await readFile(path.join(directory,'playwright.log')));await save(path.join(directory,'command.json'),record); + }return record; +} +export async function owningEvidence(directory){ + const bytes=await readFile(path.join(directory,'playwright.json')),report=JSON.parse(bytes),tests=[]; + function visit(value){if(Array.isArray(value))for(const child of value)visit(child);else if(value&&typeof value==='object'){ + if(Array.isArray(value.results)&&Object.hasOwn(value,'expectedStatus'))tests.push(value);for(const [key,child]of Object.entries(value))if(key!=='results')visit(child); + }}visit(report);assert.deepEqual(report.errors??[],[]);assert.equal(tests.length,1);const test=tests[0];assert.equal(test.expectedStatus,'passed');assert.equal(test.timeout,900000);assert.equal(test.results.length,1); + const result=test.results[0];assert.equal(result.status,'passed');assert.equal(result.retry,0);assert.deepEqual(result.errors??[],[]);assert.ok(!result.error); + assert.ok(Number.isFinite(report.stats?.duration)&&report.stats.duration>=0&&report.stats.duration<=900000,'Owning invocation exceeded original900s'); + assert.ok(Number.isFinite(Date.parse(result.startTime))&&Number.isFinite(result.duration)&&result.duration>=0&&result.duration<=900000,'Missing owning attempt timing'); + const attachments=result.attachments.filter(a=>a.name==='native-ingestion-blocked-stop-evidence');assert.equal(attachments.length,1); + let attached;if(attachments[0].body)attached=Buffer.from(attachments[0].body,'base64');else{ + const target=await realpath(path.resolve(directory,attachments[0].path));assert.ok(target.startsWith((await realpath(directory))+path.sep),'Attachment escapes owning output');attached=await readFile(target); + } + return {bytes:attached,evidence:JSON.parse(attached),reportSha256:hash(bytes),owner:{startTime:result.startTime,durationMs:result.duration,status:result.status,retry:result.retry}}; +} +export async function verifyQuietReceipts(directory){ + for(const phase of ['pre-arm','post-arm']){ + const q=await json(path.join(directory,phase==='pre-arm'?'physical-quiet-pre.json':'physical-quiet-post.json')); + assert.equal(q.schema,'paperclip.native-ingestion.physical-quiet.v1');assert.equal(q.platform,'linux');assert.equal(q.phase,phase);assert.equal(q.passed,true); + assert.equal(q.zombiePolicy,'Strict: matched zombie/dead rows remain blockers until absent; no automatic killing or silent zombie exclusion.'); + assert.ok(q.observations?.length>0);assert.deepEqual(q.observations.at(-1).blockers,[]); + } +} +async function readArm(context,arm,expectedReleaseSha256=context.releaseSha256){ + const {r,p}=context,directory=path.dirname(arm.file),attached=await owningEvidence(directory),raw=await readFile(arm.file);assert.equal(hash(raw),hash(attached.bytes));await verifyQuietReceipts(directory); + const preflight=await json(path.join(directory,'pre-arm-inventory-check.json')),port=await json(path.join(directory,'physical-http-port-check.json')); + assert.equal(preflight.releaseSha256,expectedReleaseSha256);assert.equal(preflight.variant,arm.variant);assert.equal(port.nodeBindAndListenPassed,true); + const e=attached.evidence,b=r.perVariantBuilds[arm.variant],sidecar=await json(arm.inventoryFile);p.assertFunctional(e); + assert.equal(e.variant,arm.variant);assert.equal(e.sourceRevision,b.revision);assert.equal(e.buildInfo.commit,b.revision); + assert.equal(e.serverEntrySha256,(await context.h.validateVariantBuild(arm.variant,b)).manifest.files.find(v=>v.path==='server/dist/index.js').sha256); + assert.deepEqual(e.harnessProvenance,{sourceRevision:r.harnessRevision,sourceStatus:'',driverLockSha256:LOCK,inheritedH5cRevision:INHERITED}); + const expected={};for(const f of [...CORE,...H5])expected[f]=hash(await readFile(path.join(r.harnessRoot,'tests/e2e',f)));assert.deepEqual(e.sourceHashes,expected); + assert.equal(sidecar.variant,arm.variant);assert.equal(sidecar.sourceRevision,b.revision);assert.equal(sidecar.compiledBuildRevision,b.revision);assert.equal(sidecar.playwrightReportSha256,attached.reportSha256); + for(const [key,digest]of [['manifestFile','manifestFileSha256'],['inventorySummaryFile','inventorySummarySha256'],['qualifiedReceiptFile','qualifiedReceiptSha256']]){ + assert.equal(sidecar[digest],b[digest]);assert.equal(sidecar[key],path.join('inventory',path.basename(b[key])));await pinned(path.resolve(directory,sidecar[key]),b[digest]); + }assert.equal(sidecar.aggregateSha256,b.aggregateSha256);assert.equal(sidecar.command.code,0);assert.equal(sidecar.command.signal,null);assert.equal(sidecar.command.watchdogExpired,undefined); + assert.ok(e.startedAtMs>=sidecar.command.startedAtMs&&e.finishedAtMs<=sidecar.command.finishedAtMs,'Raw proof must belong inside its owning command'); + assert.equal(await realpath(sidecar.command.executable),await realpath(r.runtimeChecks.node.executable)); + assert.equal(preflight.pathNode.version,'v26.4.0');assert.equal(preflight.pathNode.PATH,sidecar.command.environmentOverrides.PATH); + assert.equal(preflight.pathNode.cwd,b.appRoot);assert.equal(preflight.pathNode.canonicalCwd,await realpath(b.appRoot)); + assert.equal(preflight.pathNode.canonicalExecutable,await realpath(r.runtimeChecks.node.executable));assert.equal(preflight.pathNode.approvedCanonicalExecutable,preflight.pathNode.canonicalExecutable); + assert.deepEqual(sidecar.command.args,['node_modules/@playwright/test/cli.js','test','--config',CONFIG]);assert.equal(sidecar.command.cwd,r.harnessRoot);assert.equal(sidecar.command.resourceWatchdogMillis,1800000); + assert.equal(sidecar.command.environmentOverrides.PAPERCLIP_INGESTION_APP_ROOT,b.appRoot);assert.equal(sidecar.command.environmentOverrides.PAPERCLIP_INGESTION_VARIANT,arm.variant); + assert.equal(sidecar.command.environmentOverrides.PAPERCLIP_INGESTION_EVIDENCE_DIR,directory);assert.equal(Number(sidecar.command.environmentOverrides.PAPERCLIP_E2E_PORT),port.port); + assert.equal(hash(await readFile(path.join(directory,'playwright.log'))),sidecar.command.logSha256);return e; +} +async function arm(context,row,stageDirectory){ + const {r,h}=context,variant=row.variant.toUpperCase(),label=`${row.stage}-pair${row.pair}-${row.index}-${row.role}-${variant}`,directory=path.join(stageDirectory,label);await mkdir(directory); + await source(context);for(const key of AMBIENT)assert.ok(!process.env[key],`Ambient override rejected:${key}`); + const port=await portProbe(),env={PAPERCLIP_INGESTION_APP_ROOT:r.perVariantBuilds[variant].appRoot,PAPERCLIP_INGESTION_EVIDENCE_DIR:directory, + PAPERCLIP_E2E_PORT:String(port),PAPERCLIP_INGESTION_TRANSPORT:'sandbox-remote-with-local-stop',PAPERCLIP_INGESTION_VARIANT:variant, + PAPERCLIP_INGESTION_DRIVER_DEPENDENCIES:'Independent frozen companiondriver; independent normal qualified appbuild'}; + await h.physicalQuietGuard({receiptFile:path.join(directory,'physical-quiet-pre.json'),phase:'pre-arm',armStage:label}); + const versions=await runtime(r),checked=await buildCheck(context,variant);env.PAPERCLIP_RUNNER_BINARY=checked.binary;env.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH=checked.binary; + env.PATH=process.env.PATH;const launchEnvironment={...process.env,...env},pathNode=await assertPathNode(launchEnvironment,r.runtimeChecks.node.executable,r.perVariantBuilds[variant].appRoot); + const discovery=execFileSync(process.execPath,['node_modules/@playwright/test/cli.js','test','--config',CONFIG,'--list'],{cwd:r.harnessRoot,env:launchEnvironment,encoding:'utf8',timeout:30000}); + assert.match(discovery,/Total: 1 test in 1 file/);assert.match(discovery,/native-event-ingestion-blocked-stop\.journey\.ts/); + await save(path.join(directory,'pre-arm-inventory-check.json'),{releaseSha256:context.releaseSha256,variant,versions,pathNode,discovery,live:checked.live,ui:checked.ui,checkedAtMs:Date.now()}); + let command,failure;try{command=await launch(context,directory,env,launchEnvironment);assert.equal(command.code,0);assert.equal(command.signal,null);assert.equal(command.watchdogExpired,undefined);}catch(error){failure=error;} + const cleanupFailures=[]; + try{await portProbe(port);await save(path.join(directory,'physical-http-port-check.json'),{port,nodeBindAndListenPassed:true,atMs:Date.now()});} + catch(error){cleanupFailures.push({step:'port',error:String(error)});failure??=error;} + try{await h.physicalQuietGuard({receiptFile:path.join(directory,'physical-quiet-post.json'),phase:'post-arm',armStage:label,primaryFailureAlreadyPresent:Boolean(failure)});} + catch(error){cleanupFailures.push({step:'physical-quiet',error:String(error)});failure??=error;} + if(cleanupFailures.length)await save(path.join(directory,'cleanup-failures.json'),cleanupFailures);if(failure)throw failure; + const attached=await owningEvidence(directory),file=path.join(directory,'blocked-stop-evidence.json');await writeFile(file,attached.bytes,{flag:'wx'}); + const inventory=path.join(directory,'inventory');await mkdir(inventory);const b=r.perVariantBuilds[variant],copies={}; + for(const [key,digest]of [['manifestFile','manifestFileSha256'],['inventorySummaryFile','inventorySummarySha256'],['qualifiedReceiptFile','qualifiedReceiptSha256']]){ + const target=path.join(inventory,path.basename(b[key]));await writeFile(target,await pinned(b[key],b[digest]),{flag:'wx'});copies[key]=path.relative(directory,target); + }const inventoryFile=path.join(directory,'arm-inventory.json');await save(inventoryFile,{schema:'paperclip.native-ingestion.arm-inventory.v1',variant,sourceRevision:b.revision,compiledBuildRevision:b.revision, + playwrightReportSha256:attached.reportSha256,...copies,manifestFileSha256:b.manifestFileSha256,qualifiedReceiptSha256:b.qualifiedReceiptSha256,inventorySummarySha256:b.inventorySummarySha256,aggregateSha256:b.aggregateSha256,command}); + const result={...row,variant,label,file,playwrightFile:path.join(directory,'playwright.json'),inventoryFile};await readArm(context,result);result.artifacts=await h.sealArmArtifacts(context.out,result);return result; +} +function identity(context){const r=context.r;return {harnessRevision:r.harnessRevision,companionFiles:r.companionFiles,workloadSha256:r.workloadSha256,precisionRuleSha256:r.precisionRuleSha256,h5HelperSha256:H5SHA,orchestratorSha256:r.orchestratorSha256,runtimeChecks:r.runtimeChecks};} +function pairManifest(arms){return Array.from({length:5},(_,index)=>{ + const entry={pair:index+1};for(const role of ['baseline','candidate']){const arm=arms.find(a=>a.pair===index+1&&a.role===role);assert.ok(arm); + for(const [suffix,key]of [['File','file'],['PlaywrightFile','playwrightFile'],['InventoryFile','inventoryFile'],['Artifacts','artifacts']])entry[role+suffix]=arm[key]; + }return entry; +});} +async function readSeries(context,stage,pin){ + assert.equal(pin.file,path.join(context.out,stage+'.json'));const report=JSON.parse(await pinned(pin.file,pin.sha256));assert.equal(report.schema,SCHEMA);assert.equal(report.stage,stage);assert.equal(report.complete,true);assert.equal(report.passed,true);assert.deepEqual(report.identity,identity(context)); + const ownerRelease=JSON.parse(await pinned(report.releaseFile,report.releaseSha256));assert.equal(ownerRelease.authorized,true);assert.equal(ownerRelease.stage,stage);assert.equal(ownerRelease.outputRoot,context.out); + assert.equal(report.releaseFile,path.join(context.out,stage,'root-release.json'));assert.deepEqual(identity({r:ownerRelease}),report.identity); + const expected=schedule(stage,context.p.registration());assert.equal(report.arms.length,expected.length); + const raw=[];for(const [i,a]of report.arms.entries()){for(const k of ['stage','pair','index','role'])assert.equal(a[k],expected[i][k]);assert.equal(a.variant,expected[i].variant.toUpperCase()); + assert.deepEqual(await context.h.sealArmArtifacts(context.out,a),a.artifacts);raw.push(await readArm(context,a,report.releaseSha256)); + }if(stage!=='qualification-b0'){assert.deepEqual(report.pairManifest,pairManifest(report.arms));await context.h.verifyRawArtifactSeals(context.out,report.pairManifest);} + if(stage.endsWith('-aa')){assert.equal(report.noiseFile,path.join(context.out,stage+'.noise.json')); + assert.deepEqual(JSON.parse(await pinned(report.noiseFile,report.noiseSha256)),report.noise); + assert.deepEqual(report.noise,context.p.noiseEnvelope(logicalPairs(report.arms,raw)));context.p.assertNoiseEnvelope(report.noise); + } + return {report,raw}; +} +function logicalPairs(arms,raw){return Array.from({length:5},(_,i)=>['baseline','candidate'].map(role=>raw[arms.findIndex(a=>a.pair===i+1&&a.role===role)]));} +async function prerequisites(context){ + const index=STAGES.indexOf(context.r.stage),prior={},companies=new Set(),runs=new Set();let completed=0; + for(const stage of STAGES.slice(0,index)){const value=await readSeries(context,stage,context.r.priorStageReports?.[stage]??{});assert.ok(value.report.completedAtMs>completed);completed=value.report.completedAtMs;prior[stage]=value.report; + if(stage!=='qualification-b0'&&!stage.endsWith('-aa')){const noiseStage=stage.startsWith('b1')?'b1-aa':'b0-aa'; + const recomputed=context.p.comparePairs(logicalPairs(value.report.arms,value.raw),prior[noiseStage].noise);assert.deepEqual(value.report.comparison,recomputed);assert.ok(Object.values(recomputed).every(row=>row.pass)); + }for(const e of value.raw){assert.ok(!companies.has(e.companyId)&&!runs.has(e.runId),'An earlier arm was reused');companies.add(e.companyId);runs.add(e.runId);} + }if(index>=3)for(const stage of ['b0-aa','b1-aa']){ + const pin=context.r.noiseReports?.[stage];assert.equal(pin?.file,prior[stage].noiseFile);assert.equal(pin?.sha256,prior[stage].noiseSha256); + context.p.assertNoiseEnvelope(JSON.parse(await pinned(pin.file,pin.sha256))); + } + return {prior,companies,runs,lastCompletedAtMs:completed}; +} +async function execute(context){ + const {r,h,p}=context,prior=await prerequisites(context),directory=path.join(context.out,r.stage);await mkdir(directory);await writeFile(path.join(directory,'root-release.json'),await pinned(context.releaseFile,context.releaseSha256),{flag:'wx'}); + const arms=[],raw=[];try{for(const row of context.rows){const a=await arm(context,row,directory),e=await readArm(context,a); + assert.ok(!prior.companies.has(e.companyId)&&!prior.runs.has(e.runId),'Every arm needs a fresh company and run');prior.companies.add(e.companyId);prior.runs.add(e.runId);arms.push(a);raw.push(e);await save(path.join(directory,`arm${arms.length}-sealed.json`),a); + }await source(context);let noise,comparison;const pairs=r.stage==='qualification-b0'?undefined:logicalPairs(arms,raw),manifest=pairs?pairManifest(arms):undefined; + if(manifest)await h.verifyRawArtifactSeals(context.out,manifest);else assert.deepEqual(await h.sealArmArtifacts(context.out,arms[0]),arms[0].artifacts); + if(r.stage.endsWith('-aa')){noise=p.noiseEnvelope(pairs);}else if(pairs){const baseline=r.stage.startsWith('b1')?'b1-aa':'b0-aa';comparison=p.comparePairs(pairs,prior.prior[baseline].noise);} + let noiseFile,noiseSha256;if(noise){noiseFile=path.join(context.out,r.stage+'.noise.json');await save(noiseFile,noise);noiseSha256=hash(await readFile(noiseFile));} + const passed=noise?noise.suitable&&noise.noiseFrozen:!comparison||Object.values(comparison).every(row=>row.pass),report={schema:SCHEMA,scope:'Separate blocked-write control qualification/nonregression only; not main application performance acceptance',stage:r.stage, + identity:identity(context),releaseFile:path.join(directory,'root-release.json'),releaseSha256:context.releaseSha256,complete:true,passed,arms,pairManifest:manifest,noise,noiseFile,noiseSha256,comparison,completedAtMs:Date.now()}; + await save(path.join(context.out,r.stage+'.json'),report);if(noise)p.assertNoiseEnvelope(noise);assert.equal(passed,true,'Companion control nonregression failed'); + }catch(error){await save(path.join(directory,'failure.json'),{stage:r.stage,complete:false,error:String(error),stack:error.stack,completedArms:arms,atMs:Date.now()});throw error;} +} +async function main(){ + if(!process.argv.includes('--execute')&&!process.argv.includes('--validate-release')){console.log(JSON.stringify({draft:true,executionReleased:false,stages:STAGES,registeredTimingArms:50,qualificationArms:1, + note:'Exact companion source/revision, calibration rules and runtime command outputs are not frozen by this draft. No candidate or qualification result is invented.', + requiredReleaseFields:['authorized','stage','outputRoot','orchestratorSha256','h5HelperFile','h5HelperSha256','harnessRoot','harnessRevision','inheritedH5cRevision','lockSha256','companionFiles(9 exact SHA256)','workloadSha256','precisionRuleSha256','resourceWatchdogMillis:1800000','runtimeChecks(exact executable/args/expected for node26.4.0,pnpm9.15.4,postgres,chromium,rustc,cargo)','perVariantBuilds(actual qualified H5c-compatible descriptors)','priorStageReports(exact passed owner/raw/proof/seal reports by file/SHA256)','noiseReports(exact b0-aa and b1-aa noise JSON file/SHA256 pins before comparisons)' ]},null,2));return;} + const c=await load(option('release'),option('release-sha256'),option('stage'),option('output-root'));await prerequisites(c); + if(!process.argv.includes('--execute')){console.log(JSON.stringify({valid:true,sourceAndBindingsOnly:true,arms:c.rows.length,noRuntimeQualification:true}));return;} + await mkdir(c.out,{recursive:true});await execute(c); +} +if(process.argv[1]&&path.resolve(process.argv[1])===SELF)await main(); diff --git a/scripts/native-ingestion/blocked-stop/sources.json b/scripts/native-ingestion/blocked-stop/sources.json new file mode 100644 index 00000000000..ce54485c21e --- /dev/null +++ b/scripts/native-ingestion/blocked-stop/sources.json @@ -0,0 +1,122 @@ +{ + "schema": "paperclip.native-ingestion.blocked-stop-helper-sources.v3", + "scope": "Exact frozen companion v3 sources and reviewed helper; one functional qualification is not a timing/noise/performance pass.", + "repository": "https://github.com/paperclipai/paperclip.git", + "deliveryLockSha256AtPackaging": "7540075e96f9a93924a1e54775c17346c5760dd5d02902d83fb5228f918996ba", + "deliveryCheckoutWasExecutedDriver": false, + "mixedDeliveryTreeCannotReplaceEitherFrozenDriver": true, + "recordedVmRuntime": { + "node": "v26.4.0", + "pnpm": "9.15.4", + "otherRuntimeOutputs": "Exact absolute executable, argument and output pins from sealed runtime receipts; never guessed." + }, + "rawB2Role": "ca7cba171cd5198ed569d75012edc858041cbc44 is unoptimized B2 provenance, not a final candidate.", + "frozenDriver": { + "commit": "bc33f09cae26404352848169d92a20c89480a5f5", + "tree": "c84d96153d87ef3a31265c3de01568714688200c", + "parent": "6e5349cdbd5e3a197877f1082f84a17e866c03e7", + "tag": "native-ingestion-blocked-stop-v3", + "tagObject": "a49c24dc2a690764d1cfcc080077f90485218df7", + "tagPeeledCommit": "bc33f09cae26404352848169d92a20c89480a5f5", + "companionNineFileConcatOrder": "Lexicographically sorted repository-relative paths in files", + "companionNineFileSha256": "0dea53d3ec7bbf63c45c628e39c65983181d5e19f8bfae872cf78fa85e7831e8", + "frozenH5cNineFileSha256": "2265aabe9c7344f5d4b9c8b08a5c6bbd5571a6e2de9ae7b72a8d1733250e609f", + "driverLockSha256": "54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8", + "protocol": { + "PROTOCOL": "blocked-stop-history-v1", + "WORKLOAD_SHA256": "0fc383bca459682042be313e1c3243746b7428ab3b94c2a6ab3dc81eee252522", + "PRECISION_RULE_SHA256": "a93bb828c91eae6f6fcb3b5f5533465475d57aaa03c79485b51d879af5d14669", + "arms": 50 + } + }, + "sourceFreezeReceiptSha256": "210002a6699059ff9b5f74a49558c9f7f0f5866cd9ed3f786507b3ee12fc7b38", + "sourceFiles": [ + { + "file": "tests/e2e/fixtures/native-ingestion-blocked-stop-authority.ts", + "sha256": "cd94f1f0dcd49a17ea341918954a115a1627357b0b78ea9d94905190a0973514", + "bytes": 10309, + "byteIdenticalToFrozenDriver": true + }, + { + "file": "tests/e2e/fixtures/native-ingestion-blocked-stop-clock.ts", + "sha256": "530bee16cbbec6801d0efe78942917fad63b4e2e5410d273dd4510a62ea87721", + "bytes": 6976, + "byteIdenticalToFrozenDriver": true + }, + { + "file": "tests/e2e/fixtures/native-ingestion-blocked-stop-db.ts", + "sha256": "8d9c637048280feb495f834e0dcd9e35cb4fbc98ded98a8e1d296525f1cbcc96", + "bytes": 2904, + "byteIdenticalToFrozenDriver": true + }, + { + "file": "tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.mjs", + "sha256": "fcb83d72d6ab609fbd6a7e765a756989ece681f27d97675271f8d1f2c9c2ee0c", + "bytes": 28889, + "byteIdenticalToFrozenDriver": true + }, + { + "file": "tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.test.mjs", + "sha256": "91e134430df46488ad19adcdb17591c688e069e747095e4a66361ece5fb33484", + "bytes": 27170, + "byteIdenticalToFrozenDriver": true + }, + { + "file": "tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs", + "sha256": "383e43ce1e210a35ec920d8e3e5d763e7502c61db2057cf8a8d3ba8270b6dbd3", + "bytes": 1502, + "byteIdenticalToFrozenDriver": true + }, + { + "file": "tests/e2e/fixtures/native-ingestion-blocked-stop.md", + "sha256": "e5fe99b4ba353b77e76507bae6f9095a4475f61c9badbfa27c75e6ca8470f77c", + "bytes": 16597, + "byteIdenticalToFrozenDriver": true + }, + { + "file": "tests/e2e/native-event-ingestion-blocked-stop.journey.ts", + "sha256": "8b3f8b3a9704c77a6ddd14d996ebd5cf34a5fe76883c51060edee208e82211a0", + "bytes": 24850, + "byteIdenticalToFrozenDriver": true + }, + { + "file": "tests/e2e/playwright-native-ingestion-blocked-stop.config.ts", + "sha256": "e9766cfd60c2eea5c1e14bedefa95706e5d87c03efebb8229a6536f00f441706", + "bytes": 383, + "byteIdenticalToFrozenDriver": true + } + ], + "helper": { + "file": "native-ingestion-run-blocked-stop-v3.mjs", + "originalTaskFile": "/private/tmp/native-ingestion-run-blocked-stop-v3.mjs", + "bytes": 27973, + "sha256": "5748b6c1d271cf9ab270d5473873b85da295ffb03bb55ebc63693369c9b1a29d", + "packagingChanges": [] + }, + "checkedH5cDependency": { + "sha256": "6b6376b01de6cdc15ca05343a0bdd5c00424f1d6c546b9b4d61cd76716b253c4", + "packagedSource": "scripts/native-ingestion/h5c/native-ingestion-run-h5c-browser.mjs", + "recordedPath": "/home/ubuntu/native-ingestion/helpers/native-ingestion-run-h5c-browser.mjs", + "purpose": "Six checked build/inventory/UI/quiet/raw-seal exports only; original stage plan and loadRelease are not invoked." + }, + "mainCampaignBoundary": "Companion bc33 inherits H5c e431/main9=2265; main H5d2df/main9=e1 is separate. Install each immutable driver independently.", + "priorFailedQualifications": { + "v1": "200396/936d failed unsupported shared PGID assumption before epoch; API200/port did not prove physical quiet; failed raw and exact owned cleanup retained.", + "v2": "6e5349/dc693 failed unconditional retirement receipt although retained identity is supported; transcript/reload/lease gate unreached; failed raw retained, no timing reuse." + }, + "qualificationAsOfUtc": "2026-11-04T09:30:00Z", + "qualificationFacts": { + "onePassed": true, + "preludeDeltas": 1000, + "canonicalStopPrefixDeltas": 12, + "results": 0, + "heldDurationMs": 1010, + "heldSamples": 15, + "bothPidsAndBothGroupsAbsent": true, + "ownIntentAndAckAudits": true, + "retirementState": "cleared-identity-with-authoritative-retirement", + "leaseReleasedReloadAndStrictQuiet": true, + "manualSignals": false, + "noiseOrPerformancePassed": false + } +} diff --git a/scripts/native-ingestion/h5d/README.md b/scripts/native-ingestion/h5d/README.md new file mode 100644 index 00000000000..92f3315166b --- /dev/null +++ b/scripts/native-ingestion/h5d/README.md @@ -0,0 +1,122 @@ +# Frozen H5d application driver + +This package retains the six exact H5d source overlays and six reviewed helpers. +`sources.json` records their hashes. Execute immutable driver **2df9897e1ce2ca90be29fa097d776c3766e217ad**, +tag `native-ingestion-h5d`, annotated object `47ef6d1a5d81c31d4602aad50666f67c58f4cdb4`. +Its measurement hash is `e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4`; +its lock is `54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8`. +The mixed delivery checkout has a different lock and is not an executed driver. +The [blocked Stop companion](../blocked-stop/README.md) uses a separate immutable +checkout inheriting H5c measurement `2265aabe…`, not H5d measurement `e1d8930c…`. + +## Recorded qualification and limits + +As of 2026-11-04 09:30 UTC, H5d SMALL passed: 48 deltas, three results and a +nine-delta Stop prefix with no result; all 40 sealed files and cleanup were checked. +FULL's owning Playwright test passed in 853,339 ms with strict physical quiet. +The frozen assessor retains the original loaded 120-second streaming miss: +41/512 foreground markers near 119.841 seconds. The original 900-second duration +gate passed. All 51,200 loaded deltas and 100 results were independently verified, +including 43 recovered `provider_transport_failed` failures, 402,584 ms through +verified completion, and all leases closed. These are qualification facts, +not accepted A/A noise or evidence of application improvement. + +The existing-history workload is one foreground run with 1,000 emitted and +verified prelude deltas under nine fresh background runs. It does not create ten +history-bearing runs. Retain all source performance results separately from app +measurement. No application A/A campaign had passed at the recorded cutoff. +The lead's dated application report owns subsequent campaign progress. + +The old offline reader `968fbbfc…` failed an unsupported `runIds[0]` assumption. +The new foreground reader `b1e74ffd…` correlates the unique prelude run through +actual durable/provider-clock nonce, marker, activation and authority identities. +Root and peer reviewed its old-red, permutation and nine-negative checks and its +readback of a COPY of the same owning PASS raw report. Original failure, first +seal and raw bytes remain preserved. This repairs a qualification postreader; +it changes no SUT or measurement source and cannot promote a failed owning test. +The old reader is not packaged as the current reader. + +## Independent setup + +Use a disposable Linux host with the recorded Node **v26.4.0**, pnpm **9.15.4**, +PostgreSQL, Chromium and native toolchain. Node 24 is the local Mac check toolchain. +Capture actual executable paths and exact outputs before release: `node --version`, +`node -p process.execPath`, `pnpm --version`, `rustc --version`, `cargo --version`, +and the actual PostgreSQL and Chromium executables with `--version`. Runtime +release pins must use those values. Never infer a version from this package. + +```sh +TOOLS_CHECKOUT="$(git rev-parse --show-toplevel)" +BASE=/home/ubuntu/native-ingestion +DRIVER_ROOT="$BASE/harness-h5d" +HELPERS_ROOT="$BASE/helpers" +git -C "$TOOLS_CHECKOUT" fetch origin tag native-ingestion-h5d +test "$(git -C "$TOOLS_CHECKOUT" rev-parse 'native-ingestion-h5d^{commit}')" = 2df9897e1ce2ca90be29fa097d776c3766e217ad +git -C "$TOOLS_CHECKOUT" worktree add --detach "$DRIVER_ROOT" native-ingestion-h5d +pnpm --dir "$DRIVER_ROOT" install --frozen-lockfile --offline --ignore-scripts +mkdir -p "$HELPERS_ROOT" +cp "$TOOLS_CHECKOUT"/scripts/native-ingestion/h5d/*.mjs "$HELPERS_ROOT/" +cp "$TOOLS_CHECKOUT"/scripts/native-ingestion/h5d/*.py "$HELPERS_ROOT/" +node "$HELPERS_ROOT/native-ingestion-run-h5d-browser.mjs" +``` + +Use a fresh driver path and install its own locked tools. Verify copied bytes +against `sources.json`. Build every measured application independently with its +normal locked install and build. Recorded aligned builds are B0 `bac53ec3…`, B1 +`07061a60…`, and unoptimized B2 `ca7cba17…`; qualified build receipt SHA is +`a1259ba760f3cbbe44817e333f9f620bcf3769fb0a6c16f9ce2f0f5b2ed60d94`. +Validate all four artifact roots, UI shadow absence and native inputs every arm. +`ca7cba17` is not a final B2 candidate; a new candidate requires actual immutable +source, normal build and qualified descriptor. No candidate is invented here. + +The exact setup/readiness wrappers use recorded absolute paths and new output +namespaces: `h5d/setup` and `h5d/setup-full-b0`. Read them before use; they perform +live inventory/quiet checks. After an explicit quiet-window execution release, +the recorded functional commands are: + +```sh +mkdir -p "$BASE/evidence/h5d/setup" +node "$HELPERS_ROOT/native-ingestion-h5d-linux-readiness.mjs" +python3 "$HELPERS_ROOT/native-ingestion-linux-h5d-qualification.py" --tree B0 --name small-b0 +# In a separate released quiet window, with its distinct setup/output namespace: +mkdir -p "$BASE/evidence/h5d/setup-full-b0" +node "$HELPERS_ROOT/native-ingestion-h5d-full-linux-readiness.mjs" +python3 "$HELPERS_ROOT/native-ingestion-linux-h5d-qualification.py" --full --tree B0 --name full-b0 +``` + +The qualification wrapper runs real production-built app/browser/database/provider +paths. Do not execute it during another timing window. Its 30-minute resource +watchdog does not enlarge the original 120/180/900-second gates or H5 observation +600/1200-second limits. Qualification flags and source caps remain unchanged. + +On a COPY of a completed qualification output, run the current offline reader, +then frozen validator. Omit `--full` for SMALL. These read evidence, not the SUT: + +```sh +python3 "$HELPERS_ROOT/native-ingestion-extract-h5d-qualification-foreground.py" "$QUALIFICATION_COPY" --revision=bac53ec3bfeaf6546e7aaeaee516487b30534654 --full +node "$HELPERS_ROOT/native-ingestion-validate-h5d-evidence.mjs" "$QUALIFICATION_COPY" "$DRIVER_ROOT" +``` + +## Fresh measurements + +The runner's default invocation is a dry run. A byte-frozen concrete release +binds source/helper hashes, exact build/native/UI inventories, six actual runtime +outputs, output ownership, qualification and prior noise/series hashes. + +```sh +node "$HELPERS_ROOT/native-ingestion-run-h5d-browser.mjs" --release="$RELEASE_JSON" --stage=browser-aa-b0 --output-root="$EVIDENCE_ROOT" --validate-release +``` + +Only the sole benchmark owner executes the released stage by replacing +`--validate-release` with `--execute`, using the same frozen release bytes. Run +`browser-aa-b0`, `browser-aa-b1`, `browser-b0-b1`, `browser-b1-b2`, `browser-b0-b2` +in order: five fresh alternating pairs each, 50 full arms. Both suitable A/A +artifacts and their exact absolute/relative noise must freeze before comparisons +or further optimization. No qualification arm is reused. Stop and preserve raw +on any owner, cleanup or suitability failure. No replacement sample or looser +budget follows from a failure. A final candidate must pass all original SLAs +and the separate default `browser-finalize-h4e` proof after both comparisons. + +The main and companion 50-arm campaigns have separate workloads and noise. +Diagnostic profiles and counter instrumentation are separate; the unqualified +counter wrapper is intentionally absent from this package. diff --git a/scripts/native-ingestion/h5d/native-ingestion-extract-h5d-qualification-foreground.py b/scripts/native-ingestion/h5d/native-ingestion-extract-h5d-qualification-foreground.py new file mode 100644 index 00000000000..af0b2a44bd9 --- /dev/null +++ b/scripts/native-ingestion/h5d/native-ingestion-extract-h5d-qualification-foreground.py @@ -0,0 +1,74 @@ +import pathlib,json,base64,argparse,hashlib +def assert_history_prelude(case): + pre=case['preludeProof'] + foreground=[v for v in case['durable'] if v['token']==case['nonce'] and v['markerCode']==case['markerCode']] + clocks=[v for v in case['providerClocks'] if v['nonce']==case['nonce']] + activation=[v for v in case['activations'] if v['index']==0 and v['token']==case['nonce']] + authority=[v for v in case['controllerAuthority']['rows'] if v['run_id']==pre['runId']] + assert len(foreground)==len(clocks)==len(activation)==len(authority)==1 + assert pre['runId']==foreground[0]['runId']==clocks[0]['runId'] and pre['runId'] in case['runIds'] + assert authority[0]['issue_id']==activation[0]['issueId'] + assert pre['individualDeltas']==1000 and pre['uniqueSourceIdentities']==1000 + assert pre['verifiedAtMs']=1000 + +p=argparse.ArgumentParser();p.add_argument('directory');p.add_argument('--revision',required=True);p.add_argument('--full',action='store_true');a=p.parse_args();d=pathlib.Path(a.directory);r=json.loads((d/'playwright.json').read_text());attachments=[] +def visit(value): + if isinstance(value,list): + for child in value:visit(child) + elif isinstance(value,dict): + attachments.extend(item for item in value.get('attachments',[])if item.get('name')=='native-ingestion-evidence') + for key,child in value.items(): + if key!='attachments':visit(child) +visit(r);assert len(attachments)==1,len(attachments);item=attachments[0] +raw=base64.b64decode(item['body'])if item.get('body')else pathlib.Path(item['path']).read_bytes();report=json.loads(raw);(d/'native-ingestion-evidence.json').write_bytes(raw) +assert report.get('complete')is True and not report.get('primaryFailure'),report.get('primaryFailure') +assert report['sourceRevision']==report['compiledBuildRevision']==a.revision +assert report['protocolVersion']=='v5c-independent-agents-complete-settlement-original-sla' +assert report['observationMode']=='h5c-observe-original-sla-independent-agents-settlement' and report['observationComplete'] is True +assert report['measurementSha256']=='e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4' +assert report['transport']=='sandbox-remote-with-local-stop' +assert report['admissionProtocol']=='two-at-a-time-provider-and-clock-readiness-before-common-epoch' +assert report['setupActivationConcurrency']==2 +assert report['harnessProvenance']['sourceRevision']=='2df9897e1ce2ca90be29fa097d776c3766e217ad' +assert report['qualificationOnly']is(not a.full) +assert report['serverArtifact']=='compiled' and report.get('diagnosticOnly')is False +assert report['cleanup']=={'errors':[],'complete':True} +expected=['paced','burst16','burst128','burst512','existing-history','operator-stop']if a.full else['qualification','operator-stop'];assert [c['name']for c in report['cases']]==expected +ids=report['remote']['agentIds'];setup=report['remote']['agentSetup'] +assert len(ids)==(100 if a.full else 3) and len(set(ids))==len(ids) +assert setup['complete'] is True and setup['concurrency']==8 and setup['count']==len(ids) +assert len(setup['attempts'])==len(ids) and all(x['status']=='fulfilled'for x in setup['attempts']) +assert all(ids[x['index']]==x['agentId']for x in setup['attempts']) +rows=[] +for case in report['cases']: + stop=case['name']=='operator-stop';assert case['persistedAfterReload']is True + assert case['setupActivationConcurrency']==2 and case['participants']==case['runs'] + assert sorted(x['index']for x in case['activations'])==list(range(case['runs'])) + assert all(x['status']=='fulfilled'for x in case['activations']) + waves=case['activationWaves'];assert len(waves)==(case['runs']//2) + assert all(1<=len(w['indices'])<=2 and w['statuses']==['fulfilled']*len(w['indices'])for w in waves) + if not a.full and not stop:assert(case['runs'],case['events'],case['intervalMs'])==(3,16,25) + assert len(case['durable'])==case['runs']==len(case['runIds']) + if case['name']=='existing-history': + assert_history_prelude(case) + assert case['matchedVisibleClocks']==len(case['visible']) and len(case['visible'])>=(3 if stop else case['events']) + authority=case['controllerAuthority'];assert authority['kind']==('host-file-journal'if stop else'database-controller-checkpoint') + assert authority['observedReadOnly']=='on'and authority['observedIsolation']=='repeatable read' + assert len(authority['rows'])==case['runs']and authority['connectionClosedAtMs']int(db['process_identity_seq']) + else: + assert len(set(x['agent_id']for x in case['controllerAuthority']['rows']))==case['runs'] + assert all(x['agent_id']in ids for x in case['controllerAuthority']['rows']) + assert len(case['remoteLeases'])==case['runs'] + assert all(lease['releasedAt']and lease['cleanupStatus']=='success'for lease in case['remoteLeases']) + rows.append({'name':case['name'],'runs':case['runs'],'eventsPerRun':case['events'],'visibleMarkers':len(case['visible']),'individualDeltas':sum(v['individualDeltas']for v in case['durable']),'sourceClocksMatched':sum(v['matchedSourceClocks']for v in case['durable']),'runIds':case['runIds'],'transport':case['transport'],'setupWaves':len(case['activationWaves']),'setupMs':case['allReadyObservedAtMs']-case['setupReadinessStartedAtMs'],'drainThroughTerminalMs':case['drainThroughTerminalMs']}) +summary={'complete':True,'qualificationOnly':True,'performanceAcceptanceEligible':False,'sourceRevision':a.revision,'harnessRevision':report['harnessProvenance']['sourceRevision'],'reportSha256':hashlib.sha256(raw).hexdigest(),'fullFixture':a.full,'cases':rows,'cleanup':report['cleanup']};(d/'independent-qualification-check.json').write_text(json.dumps(summary,indent=2)+'\n');print(json.dumps(summary)) diff --git a/scripts/native-ingestion/h5d/native-ingestion-h5d-full-linux-readiness.mjs b/scripts/native-ingestion/h5d/native-ingestion-h5d-full-linux-readiness.mjs new file mode 100644 index 00000000000..1a3bea15b8a --- /dev/null +++ b/scripts/native-ingestion/h5d/native-ingestion-h5d-full-linux-readiness.mjs @@ -0,0 +1,28 @@ +import assert from 'node:assert/strict'; +import {readFile,writeFile,realpath} from 'node:fs/promises'; +import {execFileSync} from 'node:child_process'; +import {createHash} from 'node:crypto'; +import path from 'node:path'; +const base='/home/ubuntu/native-ingestion',helper=base+'/helpers/native-ingestion-run-h5d-browser.mjs',out=base+'/evidence/h5d/setup-full-b0'; +const sha=b=>createHash('sha256').update(b).digest('hex'); +assert.equal(sha(await readFile(helper)),'4afaa8ffcd764b498bada29c8916715b73d34462fd49152e44ab8437ffb330eb'); +const h=await import(helper); +const quiet=await h.physicalQuietGuard({receiptFile:out+'/linux-quiet-readiness.json',phase:'pre-arm',armStage:'H5c-SETUP-ONLY'}); +const receiptFile=base+'/aligned-4a83-builds-qualified.json',summaryFile=base+'/evidence/aligned-4a83/environment/artifact-inventory-summary.json'; +const receipt=JSON.parse(await readFile(receiptFile,'utf8')),summary=JSON.parse(await readFile(summaryFile,'utf8')); +const results=[]; +for(const [variant,directory] of [['B0','aligned-baseline'],['B1','aligned-cursors']]) { + const row=receipt.results.find(v=>v.variant===variant),i=summary.inventories[variant],native=receipt.nativeStageHashes.find(v=>v.variant===variant); + const descriptor={appRoot:base+'/'+directory,revision:row.revision,qualifiedReceiptFile:receiptFile,qualifiedReceiptSha256:sha(await readFile(receiptFile)),inventorySummaryFile:summaryFile,inventorySummarySha256:sha(await readFile(summaryFile)),manifestFile:path.join(path.dirname(summaryFile),i.manifest),manifestFileSha256:i.manifestSha256,aggregateSha256:i.aggregateSha256,binarySha256:native.binarySha256,rustSourceTree:receipt.rustSourceTree}; + const q=await h.validateVariantBuild(variant,descriptor),live=await h.validateActualInventory(descriptor,q.manifest),ui=await h.assertNoPreferredUiShadow(descriptor.appRoot); + assert.equal(execFileSync('git',['rev-parse','HEAD'],{cwd:descriptor.appRoot,encoding:'utf8'}).trim(),descriptor.revision); + assert.equal(execFileSync('git',['status','--short'],{cwd:descriptor.appRoot,encoding:'utf8'}).trim(),''); + assert.equal(JSON.parse(await readFile(path.join(descriptor.appRoot,'server/dist/build-info.json'),'utf8')).commit,descriptor.revision); + const target=path.join(descriptor.appRoot,'packages/paperclip-runner/runner/target'),binary=path.join(target,'release/paperclip-runnerd'); + assert.equal(await realpath(target),native.targetRealPath);assert.equal(sha(await readFile(binary)),native.binarySha256); + results.push({variant,descriptor,live,ui,nativeTargetRealPath:await realpath(target)}); +} +const pathNode=await h.assertPathNode(process.env,process.execPath,base+'/aligned-baseline'); +const data={setupOnly:true,noSutBrowserProviderOrEvents:true,performanceAcceptanceEligible:false,atMs:Date.now(),helperSha256:sha(await readFile(helper)),harnessRevision:'2df9897e1ce2ca90be29fa097d776c3766e217ad',quiet,path:pathNode,results}; +await writeFile(out+'/linux-readiness.json',JSON.stringify(data,null,2)+'\n',{flag:'wx'}); +console.log(JSON.stringify({complete:true,setupOnly:true,noSutBrowserProviderOrEvents:true,variants:results.map(v=>({variant:v.variant,fileCount:v.live.fileCount,aggregate:v.live.aggregateSha256,uiShadowAbsent:v.ui.preferredIndexAbsent})),quiet:quiet.passed,nodeVersion:process.version})); diff --git a/scripts/native-ingestion/h5d/native-ingestion-h5d-linux-readiness.mjs b/scripts/native-ingestion/h5d/native-ingestion-h5d-linux-readiness.mjs new file mode 100644 index 00000000000..05d0e331d40 --- /dev/null +++ b/scripts/native-ingestion/h5d/native-ingestion-h5d-linux-readiness.mjs @@ -0,0 +1,28 @@ +import assert from 'node:assert/strict'; +import {readFile,writeFile,realpath} from 'node:fs/promises'; +import {execFileSync} from 'node:child_process'; +import {createHash} from 'node:crypto'; +import path from 'node:path'; +const base='/home/ubuntu/native-ingestion',helper=base+'/helpers/native-ingestion-run-h5d-browser.mjs',out=base+'/evidence/h5d/setup'; +const sha=b=>createHash('sha256').update(b).digest('hex'); +assert.equal(sha(await readFile(helper)),'4afaa8ffcd764b498bada29c8916715b73d34462fd49152e44ab8437ffb330eb'); +const h=await import(helper); +const quiet=await h.physicalQuietGuard({receiptFile:out+'/linux-quiet-readiness.json',phase:'pre-arm',armStage:'H5c-SETUP-ONLY'}); +const receiptFile=base+'/aligned-4a83-builds-qualified.json',summaryFile=base+'/evidence/aligned-4a83/environment/artifact-inventory-summary.json'; +const receipt=JSON.parse(await readFile(receiptFile,'utf8')),summary=JSON.parse(await readFile(summaryFile,'utf8')); +const results=[]; +for(const [variant,directory] of [['B0','aligned-baseline'],['B1','aligned-cursors']]) { + const row=receipt.results.find(v=>v.variant===variant),i=summary.inventories[variant],native=receipt.nativeStageHashes.find(v=>v.variant===variant); + const descriptor={appRoot:base+'/'+directory,revision:row.revision,qualifiedReceiptFile:receiptFile,qualifiedReceiptSha256:sha(await readFile(receiptFile)),inventorySummaryFile:summaryFile,inventorySummarySha256:sha(await readFile(summaryFile)),manifestFile:path.join(path.dirname(summaryFile),i.manifest),manifestFileSha256:i.manifestSha256,aggregateSha256:i.aggregateSha256,binarySha256:native.binarySha256,rustSourceTree:receipt.rustSourceTree}; + const q=await h.validateVariantBuild(variant,descriptor),live=await h.validateActualInventory(descriptor,q.manifest),ui=await h.assertNoPreferredUiShadow(descriptor.appRoot); + assert.equal(execFileSync('git',['rev-parse','HEAD'],{cwd:descriptor.appRoot,encoding:'utf8'}).trim(),descriptor.revision); + assert.equal(execFileSync('git',['status','--short'],{cwd:descriptor.appRoot,encoding:'utf8'}).trim(),''); + assert.equal(JSON.parse(await readFile(path.join(descriptor.appRoot,'server/dist/build-info.json'),'utf8')).commit,descriptor.revision); + const target=path.join(descriptor.appRoot,'packages/paperclip-runner/runner/target'),binary=path.join(target,'release/paperclip-runnerd'); + assert.equal(await realpath(target),native.targetRealPath);assert.equal(sha(await readFile(binary)),native.binarySha256); + results.push({variant,descriptor,live,ui,nativeTargetRealPath:await realpath(target)}); +} +const pathNode=await h.assertPathNode(process.env,process.execPath,base+'/aligned-baseline'); +const data={setupOnly:true,noSutBrowserProviderOrEvents:true,performanceAcceptanceEligible:false,atMs:Date.now(),helperSha256:sha(await readFile(helper)),harnessRevision:'2df9897e1ce2ca90be29fa097d776c3766e217ad',quiet,path:pathNode,results}; +await writeFile(out+'/linux-readiness.json',JSON.stringify(data,null,2)+'\n',{flag:'wx'}); +console.log(JSON.stringify({complete:true,setupOnly:true,noSutBrowserProviderOrEvents:true,variants:results.map(v=>({variant:v.variant,fileCount:v.live.fileCount,aggregate:v.live.aggregateSha256,uiShadowAbsent:v.ui.preferredIndexAbsent})),quiet:quiet.passed,nodeVersion:process.version})); diff --git a/scripts/native-ingestion/h5d/native-ingestion-linux-h5d-qualification.py b/scripts/native-ingestion/h5d/native-ingestion-linux-h5d-qualification.py new file mode 100644 index 00000000000..f52d3b3177f --- /dev/null +++ b/scripts/native-ingestion/h5d/native-ingestion-linux-h5d-qualification.py @@ -0,0 +1,66 @@ +import pathlib,subprocess,json,os,socket,datetime,hashlib,traceback,argparse +p=argparse.ArgumentParser();p.add_argument('--full',action='store_true');p.add_argument('--tree',choices=['B0','B1','B2'],default='B2');p.add_argument('--name',required=True);a=p.parse_args() +base=pathlib.Path('/home/ubuntu/native-ingestion');repo=base/'paperclip';h=base/'harness-h5d';out=base/'evidence/h5d/qualification'/a.name;out.mkdir(parents=True,exist_ok=False) +pins={'B0':('aligned-baseline','bac53ec3bfeaf6546e7aaeaee516487b30534654'),'B1':('aligned-cursors','07061a601cdc20603c8c7a46f9ddbad7ac5f5939'),'B2':('aligned-batching','ca7cba171cd5198ed569d75012edc858041cbc44')};app=base/pins[a.tree][0] +ref='2df9897e1ce2ca90be29fa097d776c3766e217ad';protocol='e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4';comparator='09daef50fad5eef4dbda4e63a191749d0d167d3ae9d38bfd22df8721e2aac4a3';lock='54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8' +r={'startedAt':datetime.datetime.now(datetime.timezone.utc).isoformat(),'qualificationOnly':True,'performanceAcceptanceEligible':False,'fullFixture':a.full,'sourceRevision':pins[a.tree][1],'harnessRevision':ref,'steps':[]} +def now():return datetime.datetime.now(datetime.timezone.utc).isoformat() +def save():(out/'owner-receipt.json').write_text(json.dumps(r,indent=2)+'\n') +def cmd(name,args,cwd,env=None,timeout=None): + row={'name':name,'argv':list(map(str,args)),'cwd':str(cwd),'startedAt':now()};r['steps'].append(row);save() + with(out/(name+'.log')).open('xb')as log:result=subprocess.run(args,cwd=cwd,env=env,stdout=log,stderr=subprocess.STDOUT,timeout=timeout) + row.update({'exit':result.returncode,'finishedAt':now()});save();assert result.returncode==0,(name,result.returncode) +try: + assert(base/'aligned-4a83-builds.exit').read_text().strip()=='0' + qualified=json.loads((base/'aligned-4a83-builds-qualified.json').read_text()) + assert {row['variant']:row['revision'] for row in qualified['results']}=={v:pin for v,(directory,pin) in pins.items()} + rows=[] + for q in pathlib.Path('/proc').iterdir(): + if not q.name.isdigit():continue + try: + exe=os.readlink(q/'exe') + if pathlib.Path(exe).name in['node','postgres','paperclip-runner','rustc','cargo','chrome','chromium','chrome-headless-shell']:rows.append({'pid':int(q.name),'executable':exe}) + except OSError:pass + assert not rows,rows + for directory,pin in pins.values(): + root=base/directory;assert subprocess.check_output(['git','rev-parse','HEAD'],cwd=root,text=True).strip()==pin + assert not subprocess.check_output(['git','status','--short'],cwd=root,text=True).strip() + assert json.loads((root/'server/dist/build-info.json').read_text())['commit']==pin + assert h.exists(), 'Stage the exact independent driver before qualification' + assert not(h/'node_modules').is_symlink();assert subprocess.check_output(['git','rev-parse','HEAD'],cwd=h,text=True).strip()==ref + assert not subprocess.check_output(['git','status','--short'],cwd=h,text=True).strip() + files=['native-event-ingestion.journey.ts','playwright-native-ingestion.config.ts','fixtures/in-feed-codex.mjs','fixtures/native-ingestion-bin/codex','fixtures/composer-stop-codex-loader.mjs','fixtures/native-ingestion-bootstrap.mjs','fixtures/native-ingestion-remote-server.mjs','fixtures/native-ingestion-protocol-qualification.mjs','fixtures/native-ingestion-h5.mjs'] + assert hashlib.sha256(b''.join((h/'tests/e2e'/f).read_bytes()for f in files)).hexdigest()==protocol + assert hashlib.sha256((h/'scripts/compare-native-ingestion-browser.ts').read_bytes()).hexdigest()==comparator + assert hashlib.sha256((h/'pnpm-lock.yaml').read_bytes()).hexdigest()==lock + assert hashlib.sha256((h/'tests/e2e/fixtures/native-ingestion-h5.md').read_bytes()).hexdigest()=='c647cfb68b360882a3bf42771b35e37c52e0aba7bca01511f4ebfb9482bf45bc' + r['resourceWatchdogMillis']=1800000;r['resourceWatchdogScope']='Resource-only including startup/test/teardown; original SLA and H5 observation unchanged';r['protocolSha256']=protocol;r['comparatorSha256']=comparator;r['driverLockSha256']=lock + with socket.socket()as s:s.bind(('127.0.0.1',0));port=s.getsockname()[1] + env=dict(os.environ) + for key in ['DATABASE_URL','PAPERCLIP_RUNNER_STATE_DIR','PAPERCLIP_INGESTION_SERVER_COMMAND','PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_DIR','PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_FS','PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_INSPECTOR','PAPERCLIP_INGESTION_BROWSER_CAPTURE','PAPERCLIP_INGESTION_SMOKE','NODE_OPTIONS','PAPERCLIP_INGESTION_OBSERVATION_MODE']:env.pop(key,None) + binary=str(app/'packages/paperclip-runner/runner/target/release/paperclip-runnerd') + overrides={'PAPERCLIP_INGESTION_OBSERVATION_MODE':'h5c-observe-original-sla-independent-agents-settlement','PAPERCLIP_INGESTION_TRANSPORT':'sandbox-remote-with-local-stop','PAPERCLIP_INGESTION_APP_ROOT':str(app),'PAPERCLIP_INGESTION_EVIDENCE_DIR':str(out),'PAPERCLIP_E2E_PORT':str(port),'PAPERCLIP_RUNNER_BINARY':binary,'PAPERCLIP_RUNNER_REMOTE_BINARY_PATH':binary,'PAPERCLIP_INGESTION_VARIANT':a.name+'-FUNCTIONAL-QUALIFICATION-ONLY','PAPERCLIP_INGESTION_DRIVER_DEPENDENCIES':'Independent H5c frozen offline ignore-scripts driver install; application owns independent normal locked install/build'} + if not a.full:overrides['PAPERCLIP_INGESTION_SMOKE']='1' + overrides['PATH']=env['PATH'];env.update(overrides);r['environmentOverrides']=overrides;save();subprocess.run(['sync'],check=True) + node=pathlib.Path('/opt/native-ingestion/node-v26.4.0-linux-x64/bin/node') + helper=base/'helpers/native-ingestion-run-h5d-browser.mjs' + assert hashlib.sha256(helper.read_bytes()).hexdigest()=='4afaa8ffcd764b498bada29c8916715b73d34462fd49152e44ab8437ffb330eb' + path_guard="const {assertPathNode}=await import("+json.dumps(str(helper))+");console.log(JSON.stringify(await assertPathNode(process.env,process.execPath,process.argv[1])));" + path_argv=[str(node),'--input-type=module','-e',path_guard,str(app)] + path_result=subprocess.run(path_argv,cwd=app,env=env,check=True,capture_output=True,text=True) + r['nodePathPreflight']={'executable':str(node),'argv':path_argv[1:],'cwd':str(app),'pathNode':json.loads(path_result.stdout)} + (out/'path-node-preflight.json').write_text(json.dumps(r['nodePathPreflight'],indent=2)+'\n');save() + cmd('playwright',['timeout','--signal=INT','--kill-after=40s','30m',str(node),'node_modules/@playwright/test/cli.js','test','--config','tests/e2e/playwright-native-ingestion.config.ts'],h,env,1845) + r['playwrightPassed']=True;r['finishedAt']=now();save() +except BaseException as error:r.update({'passed':False,'error':str(error),'traceback':traceback.format_exc(),'finishedAt':now()});save();(out/'owner.exit').write_text('1\n');raise + +finally: + helper=base/'helpers/native-ingestion-run-h5d-browser.mjs' + guard="import net from 'node:net'; import {physicalQuietGuard} from '"+str(helper)+"'; const s=net.createServer(); await new Promise((ok,bad)=>{s.once('error',bad);s.listen(Number(process.argv[2]),'127.0.0.1',ok)}); await new Promise(ok=>s.close(ok)); await physicalQuietGuard({receiptFile:process.argv[1],phase:'post-arm',armStage:'H5d-QUALIFICATION-ONLY'});" + try: + result=subprocess.run(['node','--input-type=module','-e',guard,str(out/'physical-quiet-after.json'),str(r['environmentOverrides']['PAPERCLIP_E2E_PORT'])],check=True,capture_output=True,text=True) + (out/'port-after.json').write_text(json.dumps({'complete':True,'port':r['environmentOverrides']['PAPERCLIP_E2E_PORT'],'checkedAt':now(),'nodeBindAndListenPassed':True})+'\n') + if r.get('playwrightPassed'): + r['passed']=True;r['cleanupComplete']=True;save();(out/'owner.exit').write_text('0\n') + except BaseException as cleanup_error: + r['externalQuietFailure']=str(cleanup_error);r['passed']=False;r['cleanupComplete']=False;save();(out/'owner.exit').write_text('1\n');raise diff --git a/scripts/native-ingestion/h5d/native-ingestion-run-h5d-browser.mjs b/scripts/native-ingestion/h5d/native-ingestion-run-h5d-browser.mjs new file mode 100644 index 00000000000..7d8f4c631d8 --- /dev/null +++ b/scripts/native-ingestion/h5d/native-ingestion-run-h5d-browser.mjs @@ -0,0 +1,705 @@ +import assert from 'node:assert/strict'; +import { spawn, execFileSync } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { readFile, writeFile, mkdir, open, access, readdir, realpath, stat, lstat } from 'node:fs/promises'; +import { createReadStream } from 'node:fs'; +import net from 'node:net'; +import path from 'node:path'; +import { fileURLToPath, pathToFileURL } from 'node:url'; +import { gunzipSync } from 'node:zlib'; + +// Prospective H5 application orchestrator. No launch without --execute and a +// byte-frozen root release. Original ce33 helper SHA is provenance only: +// 2244e238642caafd488a7c86fb463376c8459ee0f06f8054555407a174698fa1. +export const MODE = 'h5c-observe-original-sla-independent-agents-settlement'; +export const PROTOCOL = 'v5c-independent-agents-complete-settlement-original-sla'; +export const WORKLOAD_SHA256 = '3eea2f24c0c0c38134492a0a17e1b9a398335c085dd248f03b14c14bb5a19835'; +const H5_REVISION = '2df9897e1ce2ca90be29fa097d776c3766e217ad'; +const OLD_NATIVE_TARGET = '/home/ubuntu/native-ingestion/aligned-baseline/packages/paperclip-runner/runner/target'; +const H5_PROTOCOL_SHA256 = 'e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4'; +const H5_COMPARATOR_SHA256 = '09daef50fad5eef4dbda4e63a191749d0d167d3ae9d38bfd22df8721e2aac4a3'; +const H5_INVENTORY_HELPER_SHA256 = '76b897dd6b7a90e784e228d78d79633a6d7011fe3ea01737f3bef9c6113298e9'; +const H5_PROTOCOL_DOCUMENT_SHA256 = 'c647cfb68b360882a3bf42771b35e37c52e0aba7bca01511f4ebfb9482bf45bc'; +const INVENTORY_ROOTS = ['server/dist', 'ui/dist', 'packages/db/src', 'packages/shared/src']; +export const RESOURCE_WATCHDOG_MILLIS = 30 * 60_000; +export const STAGES = Object.freeze(['browser-aa-b0', 'browser-aa-b1', 'browser-b0-b1', 'browser-b1-b2', 'browser-b0-b2']); +export const FINALIZE_STAGE = 'browser-finalize-h4e'; +const protocolFiles = ['native-event-ingestion.journey.ts', 'playwright-native-ingestion.config.ts', 'fixtures/in-feed-codex.mjs', + 'fixtures/native-ingestion-bin/codex', 'fixtures/composer-stop-codex-loader.mjs', 'fixtures/native-ingestion-bootstrap.mjs', + 'fixtures/native-ingestion-remote-server.mjs', 'fixtures/native-ingestion-protocol-qualification.mjs', 'fixtures/native-ingestion-h5.mjs']; +const BASELINES = Object.freeze({ B0: 'bac53ec3bfeaf6546e7aaeaee516487b30534654', B1: '07061a601cdc20603c8c7a46f9ddbad7ac5f5939' }); +const OLD_B2 = 'ca7cba171cd5198ed569d75012edc858041cbc44'; +const OLD_RECEIPT = 'a1259ba760f3cbbe44817e333f9f620bcf3769fb0a6c16f9ce2f0f5b2ed60d94'; +const OLD_SUMMARY = 'd3766fc4206d569d10fd02224433f1e9a2caa6992a7c62216e80af368aa4e44d'; +const OLD_BINARY = 'abb65bdc073991e428bd37f9809603ee7564543cca0ed0305dc231137c93070f'; +const OLD_RUST = 'bbe29bba574ce8d5cfddf2f7461ae0eb5983df07'; +const OLD_INVENTORIES = Object.freeze({ + B0: { manifestFileSha256: 'a1c797a75dd3174f414fc5529acf56783e1a3dfe7b22f1e460a87b60f6bff01d', aggregateSha256: 'a50720be2ce53f9222187d63609c5936f94b6a00c60df2ca112261abd1bf921e' }, + B1: { manifestFileSha256: 'df48e5c511a2f31e31180e05f18e2322ca4f384ea06f6f96a57de24d39849ef8', aggregateSha256: 'bca5b6fad62e70631c135a9ff377efeb587aba17ef0842c8fd82804ed99a81ab' }, +}); +const LOCK_SHA256 = '54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8'; +const GATES_SHA256 = 'ded8bcbb8f05732348f6520b9c5e41b7c35bc098997eb263a9dcb174eede75fc'; +const selfFile = fileURLToPath(import.meta.url); +const hash = bytes => createHash('sha256').update(bytes).digest('hex'); +const getJson = async file => JSON.parse(await readFile(file, 'utf8')); +const saveJson = async (file, value, fresh = false) => writeFile(file, JSON.stringify(value, null, 2) + '\n', fresh ? { flag: 'wx' } : undefined); +const digest = (value, name) => assert.ok(typeof value === 'string' && /^[a-f0-9]{64}$/.test(value), `Missing exact ${name} SHA-256`); +const revision = (value, name) => assert.ok(typeof value === 'string' && /^[a-f0-9]{40}$/.test(value), `Missing immutable full ${name} revision`); +const absolute = (value, name) => assert.ok(typeof value === 'string' && path.isAbsolute(value), `Missing absolute ${name} path`); +const option = name => process.argv.find(value => value.startsWith(`--${name}=`))?.slice(name.length + 3); + +export function stagePlan(stage) { + assert.ok(STAGES.includes(stage) || stage === FINALIZE_STAGE, 'Select one registered H5 stage'); + if (stage === FINALIZE_STAGE) return { stage, requiredVariants: ['B0', 'B1', 'B2'], priorStages: [...STAGES], fullArms: 0, finalizationOnly: true }; + const [baseline, candidate] = stage === STAGES[0] ? ['B0', 'B0'] : stage === STAGES[1] ? ['B1', 'B1'] + : stage === STAGES[2] ? ['B0', 'B1'] : stage === STAGES[3] ? ['B1', 'B2'] : ['B0', 'B2']; + const aa = baseline === candidate; + return { stage, baseline, candidate, aa, expectation: stage === STAGES[2] ? 'nonregression' : 'improvement', + candidatePolicy: aa ? null : stage === STAGES[2] ? 'cursor-nonregression' : 'final-b2', + requiredVariants: [...new Set([baseline, candidate])], priorStages: STAGES.slice(0, STAGES.indexOf(stage)), + pairs: 5, fullArms: 10, noiseStage: aa ? null : `browser-aa-${baseline.toLowerCase()}` }; +} + +async function hashedJson(file, expected, name, compressed = false) { + absolute(file, name); digest(expected, name); + const bytes = await readFile(file); assert.equal(hash(bytes), expected, `${name} bytes changed`); + return { bytes, value: JSON.parse((compressed ? gunzipSync(bytes, { maxOutputLength: 128 * 1024 * 1024 }) : bytes).toString('utf8')) }; +} + +export async function validateVariantBuild(variant, descriptor) { + assert.ok(descriptor && typeof descriptor === 'object', `Required ${variant} qualified build is missing`); + absolute(descriptor.appRoot, `${variant} appRoot`); revision(descriptor.revision, variant); + digest(descriptor.binarySha256, `${variant} binary`); revision(descriptor.rustSourceTree, `${variant} Rust source tree`); + const receipt = (await hashedJson(descriptor.qualifiedReceiptFile, descriptor.qualifiedReceiptSha256, `${variant} qualified receipt`)).value; + const summary = (await hashedJson(descriptor.inventorySummaryFile, descriptor.inventorySummarySha256, `${variant} inventory summary`)).value; + const manifest = (await hashedJson(descriptor.manifestFile, descriptor.manifestFileSha256, `${variant} inventory manifest`, true)).value; + digest(descriptor.aggregateSha256, `${variant} aggregate inventory`); + const rows = receipt.results?.filter(row => row.variant === variant); + assert.equal(rows?.length, 1, `${variant} qualified receipt must select this variant exactly once`); + assert.equal(rows[0].revision, descriptor.revision, `${variant} qualified receipt revision mismatch`); + assert.equal(rows[0].compiledBuildRevision, descriptor.revision, `${variant} qualified compiled revision mismatch`); + assert.equal(rows[0].sourceClean, true, `${variant} qualified source was not clean`); + const native = receipt.nativeStageHashes?.filter(row => row.variant === variant); + assert.equal(native?.length, 1, `${variant} qualified native stage is missing`); + assert.equal(native[0].binarySha256, descriptor.binarySha256, `${variant} native stage binary mismatch`); + assert.equal(receipt.binarySha256, descriptor.binarySha256, `${variant} receipt binary mismatch`); + assert.equal(receipt.rustSourceTree, descriptor.rustSourceTree, `${variant} Rust source identity mismatch`); + assert.equal(receipt.rustInputObjects?.runner, descriptor.rustSourceTree, `${variant} qualified Rust input objects are missing`); + absolute(native[0].targetRealPath, `${variant} qualified native target`); + for (const [file, object] of Object.entries(receipt.rustInputObjects)) { + assert.ok(file && !path.posix.isAbsolute(file) && path.posix.normalize(file) === file && !file.includes('\\') && !file.split('/').includes('..'), 'Qualified Rust input path is invalid'); + revision(object, `${variant} Rust input ${file}`); + } + assert.equal(receipt.artifactInventorySummary, descriptor.qualifiedInventorySummaryReference ?? descriptor.inventorySummaryFile, `${variant} receipt points to another inventory summary`); + assert.equal(manifest.variant, variant); assert.equal(manifest.revision, descriptor.revision); + assert.equal(manifest.compiledBuildRevision, descriptor.revision); assert.equal(manifest.aggregateSha256, descriptor.aggregateSha256); + assert.deepEqual(manifest.roots, INVENTORY_ROOTS); + assert.ok(Array.isArray(manifest.files) && manifest.files.length === manifest.fileCount && manifest.fileCount > 0); + const row = summary.inventories?.[variant]; assert.ok(row, `${variant} inventory summary entry missing`); + assert.equal(row.manifest, path.basename(descriptor.manifestFile)); assert.equal(row.manifestSha256, descriptor.manifestFileSha256); + assert.equal(row.aggregateSha256, descriptor.aggregateSha256); assert.equal(row.fileCount, manifest.fileCount); + if (Object.hasOwn(BASELINES, variant)) { + assert.equal(descriptor.revision, BASELINES[variant], 'Original baseline reference must remain unchanged'); + assert.equal(descriptor.qualifiedReceiptSha256, OLD_RECEIPT, 'B0/B1 must use the original, unmodified aligned4a83 qualified receipt'); + assert.equal(descriptor.inventorySummarySha256, OLD_SUMMARY); assert.equal(descriptor.binarySha256, OLD_BINARY); + assert.equal(descriptor.rustSourceTree, OLD_RUST); + for (const [key, value] of Object.entries(OLD_INVENTORIES[variant])) assert.equal(descriptor[key], value, `Original ${variant} inventory changed`); + } else { + assert.equal(variant, 'B2'); + assert.ok(![...Object.values(BASELINES), OLD_B2].includes(descriptor.revision), 'Final B2 needs its new actual immutable product revision'); + assert.notEqual(descriptor.qualifiedReceiptSha256, OLD_RECEIPT, 'Final B2 needs its own new qualified receipt'); + assert.notEqual(descriptor.binarySha256, OLD_BINARY, 'Final Rust B2 needs its newly qualified binary'); + assert.notEqual(descriptor.rustSourceTree, OLD_RUST, 'Final Rust B2 needs its newly qualified Rust inputs'); + assert.notEqual(native[0].targetRealPath, OLD_NATIVE_TARGET, 'Final Rust B2 must not reuse the original B0/B1 target'); + } + return { receipt, summary, manifest }; +} + +// Match the existing inventory generator and H5 helper exactly: Unicode code +// point order, Python ensure_ascii JSON with sorted bytes/path/sha256 keys. +function pythonOrder(left, right) { + const a = [...left], b = [...right]; + for (let index = 0; index < Math.min(a.length, b.length); index++) { + const difference = a[index].codePointAt(0) - b[index].codePointAt(0); + if (difference) return difference; + } + return a.length - b.length; +} +const pythonRow = row => JSON.stringify({ bytes: row.bytes, path: row.path, sha256: row.sha256 }) + .replace(/[\u007f-\uffff]/g, character => `\\u${character.charCodeAt(0).toString(16).padStart(4, '0')}`); +function inventoryPath(value) { + assert.ok(typeof value === 'string' && value.length && !value.includes('\\') && !value.includes('\0') + && !path.posix.isAbsolute(value) && path.posix.normalize(value) === value && !value.split('/').includes('..') + && INVENTORY_ROOTS.some(root => value.startsWith(`${root}/`)), 'Inventory filename escapes registered roots'); + return value; +} +export async function validateActualInventory(build, manifest) { + const startedAtMs = Date.now(), started = performance.now(), canonicalRoot = await realpath(build.appRoot); + assert.deepEqual(manifest.roots, INVENTORY_ROOTS); + const expected = new Map(), canonicalFiles = new Set(); + for (const row of manifest.files) { + const name = inventoryPath(row.path); + assert.equal(expected.has(name), false, 'Duplicate inventory filename'); expected.set(name, row); + digest(row.sha256, 'inventory file'); assert.ok(Number.isSafeInteger(row.bytes) && row.bytes >= 0); + } + assert.equal(expected.size, manifest.fileCount, 'Inventory file count is incomplete'); + const actualNames = [], canonicalDirectories = new Set(); + const walk = async relative => { + const directory = path.join(build.appRoot, relative), canonical = await realpath(directory); + assert.ok(canonical.startsWith(`${canonicalRoot}${path.sep}`), 'Inventory directory resolves outside APP_ROOT'); + assert.equal(canonicalDirectories.has(canonical), false, 'Inventory directory aliases another registered directory'); canonicalDirectories.add(canonical); + for (const entry of await readdir(directory, { withFileTypes: true })) { + const child = `${relative}/${entry.name}`; + if (entry.isDirectory()) await walk(child); + else { + const file = path.join(build.appRoot, child), metadata = await lstat(file); + if (metadata.isSymbolicLink()) assert.equal((await stat(file)).isDirectory(), false, 'Inventory directory symlink is not a frozen file'); + assert.ok(metadata.isFile() || metadata.isSymbolicLink(), 'Inventory includes a nonregular entry'); + inventoryPath(child); actualNames.push(child); + } + } + }; + for (const root of INVENTORY_ROOTS) await walk(root); + actualNames.sort(pythonOrder); + assert.deepEqual(actualNames, [...expected.keys()].sort(pythonOrder), 'Actual APP_ROOT file set differs from frozen inventory'); + const actualRows = new Array(actualNames.length); let next = 0; + await Promise.all(Array.from({ length: Math.min(8, actualNames.length) }, async () => { + for (;;) { + const index = next++; if (index >= actualNames.length) return; + const name = actualNames[index], file = path.join(build.appRoot, name), canonical = await realpath(file); + assert.ok(canonical.startsWith(`${canonicalRoot}${path.sep}`), 'Inventory file resolves outside APP_ROOT'); + assert.equal(canonicalFiles.has(canonical), false, 'Inventory filenames alias the same canonical file'); canonicalFiles.add(canonical); + const before = await stat(file); assert.ok(before.isFile(), 'Inventory evidence must be a regular file'); + const content = createHash('sha256'); for await (const bytes of createReadStream(file)) content.update(bytes); + const after = await stat(file); + assert.equal(after.size, before.size, 'Artifact changed during inventory hashing'); assert.equal(after.mtimeMs, before.mtimeMs, 'Artifact changed during inventory hashing'); + const row = { bytes: after.size, path: name, sha256: content.digest('hex') }, frozen = expected.get(name); + assert.equal(row.bytes, frozen.bytes, `Actual artifact size changed: ${name}`); assert.equal(row.sha256, frozen.sha256, `Actual artifact bytes changed: ${name}`); + actualRows[index] = row; + } + })); + const aggregateSha256 = hash(Buffer.from(`${actualRows.map(pythonRow).join('\n')}\n`, 'utf8')); + assert.equal(aggregateSha256, manifest.aggregateSha256, 'Actual APP_ROOT aggregate differs from frozen inventory'); + assert.equal(aggregateSha256, build.aggregateSha256, 'Actual APP_ROOT aggregate differs from qualified build'); + return { appRoot: build.appRoot, canonicalAppRoot: canonicalRoot, fileCount: actualRows.length, aggregateSha256, + startedAtMs, completedAtMs: Date.now(), elapsedMs: performance.now() - started, + scope: 'Live registered file set, canonical containment/uniqueness, size and SHA checked before this arm; not an assertion of future immutability' }; +} + +function harnessFields(release) { + return Object.fromEntries(['harnessRevision', 'protocolSha256', 'comparatorSha256', 'inventoryHelperSha256', 'protocolDocumentSha256', 'lockSha256', 'gatesSha256', 'workloadSha256', 'observationMode', 'protocolVersion'].map(key => [key, release[key]])); +} + +export async function loadRelease(releaseFile, stage, outputRoot) { + absolute(releaseFile, 'root release'); absolute(outputRoot, 'output root'); + const releaseBytes = await readFile(releaseFile), release = JSON.parse(releaseBytes); + const plan = stagePlan(stage); + assert.equal(release.authorized, true, 'Root release is not authorized'); assert.equal(release.stage, stage); + assert.equal(release.outputRoot, outputRoot, 'Root release freezes a different exact output path'); + assert.equal(release.orchestratorSha256, hash(await readFile(selfFile)), 'Orchestrator changed after release'); + absolute(release.harnessRoot, 'H5 harness'); revision(release.harnessRevision, 'H5 harness'); + assert.equal(release.harnessRevision, H5_REVISION, 'Use the actual independently frozen H5 full revision'); + assert.equal(release.observationMode, MODE); assert.equal(release.protocolVersion, PROTOCOL); + assert.equal(release.workloadSha256, WORKLOAD_SHA256); + assert.equal(release.resourceWatchdogMillis, RESOURCE_WATCHDOG_MILLIS, 'Resource watchdog registration changed; this is not an SLA'); + assert.equal(release.lockSha256, LOCK_SHA256); assert.equal(release.gatesSha256, GATES_SHA256); + for (const [key, expected] of [['protocolSha256', H5_PROTOCOL_SHA256], ['comparatorSha256', H5_COMPARATOR_SHA256], ['inventoryHelperSha256', H5_INVENTORY_HELPER_SHA256], ['protocolDocumentSha256', H5_PROTOCOL_DOCUMENT_SHA256]]) + assert.equal(release[key], expected, `Reviewed H5 source pin changed: ${key}`); + for (const variant of plan.requiredVariants) await validateVariantBuild(variant, release.perVariantBuilds?.[variant]); + const context = { releaseFile, releaseBytesSha256: hash(releaseBytes), release, plan, out: outputRoot, commands: [], commandIndex: 0 }; + await checkPriorStages(context); + if (plan.noiseStage) await checkNoise(context); + if (plan.finalizationOnly) await checkFinalizationInputs(context); + return context; +} + +async function checkReleaseBytes(context) { + assert.equal(hash(await readFile(context.releaseFile)), context.releaseBytesSha256, 'Exact root release bytes changed'); + assert.equal(hash(await readFile(selfFile)), context.release.orchestratorSha256, 'Orchestrator changed after release'); + if (context.rootReleaseCopy) assert.equal(hash(await readFile(context.rootReleaseCopy)), context.releaseBytesSha256, 'Preserved root release bytes changed'); +} + +function checkComparisonIdentity(report, context) { + assert.equal(report.complete, true); assert.equal(report.observationMode, MODE); assert.equal(report.protocolVersion, PROTOCOL); + assert.equal(report.workloadSha256, WORKLOAD_SHA256); assert.equal(report.measurementSha256, context.release.protocolSha256); + assert.equal(report.comparatorSha256, context.release.comparatorSha256); assert.equal(report.gatesSha256, GATES_SHA256); + assert.equal(report.pairCount, 5); +} + +export function assertAaNoiseSuitability(noise) { + assert.equal(noise.aaSuitability?.passed, true, 'Unsuitable or missing browser A/A cannot freeze noise'); + assert.deepEqual(noise.aaSuitability.rule, { pairCount: 5, pairedFraction: 0.20, medianDriftFraction: 0.10, + contrastRepresentationFloorMs: 2, representationLimitedMedianMs: 20 }, 'Browser A/A suitability rule changed'); + const keys = []; + for (const name of ['paced', 'burst16', 'burst128', 'burst512', 'existing-history', 'operator-stop']) { + keys.push(...['firstVisibleAfterEmissionMs', 'emissionToVisible.medianMs', 'drainThroughTerminalMs'].map(metric => `${name}:${metric}`)); + if (name === 'paced') keys.push(`${name}:interChunkGaps.maxMs`); + if (!['burst16', 'operator-stop'].includes(name)) keys.push(`${name}:emissionToVisible.p95Ms`, `${name}:intendedToVisible.p95Ms`); + if (name === 'operator-stop') keys.push(`${name}:stopThroughTerminalMs`, `${name}:stopThroughAcknowledgementMs`); + else keys.push(`${name}:h5.independentDrain.elapsedFromEpochMs`); + } + const rows = noise.measurements; + assert.ok(Array.isArray(rows), 'Browser A/A measurements are missing'); + // The frozen comparator adds both Stop p95 metrics only for a >=100-marker + // observed Stop prefix. Its owning raw validation remains authoritative. + if (rows.some(row => row.name === 'operator-stop' && ['emissionToVisible.p95Ms', 'intendedToVisible.p95Ms'].includes(row.metric))) + keys.push('operator-stop:emissionToVisible.p95Ms', 'operator-stop:intendedToVisible.p95Ms'); + assert.deepEqual(rows.map(row => `${row.name}:${row.metric}`).sort(), keys.sort(), 'Browser A/A metric set changed or verified completion metric is missing'); + assert.equal(noise.aaSuitability.metricCount, keys.length, 'Browser A/A metric count changed'); + for (const row of rows) { + assert.equal(row.aaSuitability?.passed, true, 'Every browser A/A measurement must pass suitability'); + assert.equal(row.aaSuitability.pairCount, 5); assert.equal(row.aaSuitability.sampleCount, 10); + assert.equal(row.samples?.length, 5, 'Every browser A/A metric needs five pairs'); + for (const key of ['aaNoiseEnvelopeRelative', 'aaNoiseEnvelopeAbsoluteMs']) + assert.ok(Number.isFinite(row[key]) && row[key] >= 0, 'Browser A/A actual noise must be finite and nonnegative'); + assert.equal(row.aaNoiseEnvelopeRelative, row.aaSuitability.observedNoiseRelative, 'Browser A/A relative noise changed'); + assert.equal(row.aaNoiseEnvelopeAbsoluteMs, row.aaSuitability.observedNoiseAbsoluteMs, 'Browser A/A absolute noise changed'); + } +} + +async function checkPriorStages(context) { + for (const stage of context.plan.priorStages) { + const pin = context.release.priorStageReports?.[stage]; assert.ok(pin, `Stage order requires frozen ${stage} first`); + assert.equal(pin.file, path.join(context.out, `${stage}.json`), 'Prior stage report path changed'); + const report = (await hashedJson(pin.file, pin.sha256, stage)).value; + checkComparisonIdentity(report, context); + const priorPlan = stagePlan(stage); + assert.equal(report.phase, priorPlan.aa ? 'aa' : 'compare'); + if (priorPlan.aa) assertAaNoiseSuitability(report); + assert.equal(report.baselineRevision, BASELINES[priorPlan.baseline]); + const candidateRevision = BASELINES[priorPlan.candidate] ?? context.release.perVariantBuilds.B2?.revision; + assert.equal(report.candidateRevision, candidateRevision, 'Prior stage used a different measured candidate revision'); + for (const [arm, variant] of [['baseline', priorPlan.baseline], ['candidate', priorPlan.candidate]]) { + const expected = context.release.perVariantBuilds?.[variant] ?? (BASELINES[variant] ? { ...OLD_INVENTORIES[variant], qualifiedReceiptSha256: OLD_RECEIPT, inventorySummarySha256: OLD_SUMMARY } : undefined); + assert.ok(expected, 'Prior candidate qualified build is unavailable'); + for (const key of ['manifestFileSha256', 'aggregateSha256', 'qualifiedReceiptSha256', 'inventorySummarySha256']) + assert.equal(report.artifactInventories?.[arm]?.[key], expected[key], 'Prior stage changed a measured qualified inventory'); + } + if (!priorPlan.aa) assert.equal(report.performancePassed, true, `Prior ${stage} performance failed`); + } +} + +export async function checkNoise(context) { + const { release, plan, out } = context; + const expected = path.join(out, `${plan.noiseStage}.json`); + assert.equal(release.noiseReportFile, expected, 'Use the same exact frozen A/A noise path'); + const noise = (await hashedJson(expected, release.noiseReportSha256, 'Reviewed A/A noise report')).value; + checkComparisonIdentity(noise, context); assert.equal(noise.phase, 'aa'); + assert.equal(noise.baselineRevision, release.perVariantBuilds[plan.baseline].revision); + assert.equal(noise.candidateRevision, noise.baselineRevision); + assert.equal(noise.h5Assessments?.length, 5, 'Reviewed A/A must contain all five complete assessments'); + assertAaNoiseSuitability(noise); + const build = release.perVariantBuilds[plan.baseline]; + for (const arm of ['baseline', 'candidate']) for (const key of ['manifestFileSha256', 'aggregateSha256', 'qualifiedReceiptSha256', 'inventorySummarySha256']) + assert.equal(noise.artifactInventories?.[arm]?.[key], build[key], 'Frozen A/A baseline inventory differs from required build'); + return noise; +} + +const QUIET_SETTLE_MILLIS = 30_000, QUIET_POLL_MILLIS = 1_000, QUIET_PS_TIMEOUT_MILLIS = 2_000; +const QUIET_SCHEMA = 'paperclip.native-ingestion.physical-quiet.v1'; +const QUIET_SCOPE = 'Dedicated Linux process-name guard for specified Node/PostgreSQL/Runner/browser/Rust-build names only; not a claim that every possible host activity is absent. Qualified H5 Stop proof separately covers provider/sleep control groups.'; +const QUIET_ZOMBIE_POLICY = 'Strict: matched zombie/dead rows remain blockers until absent; no automatic killing or silent zombie exclusion.'; +function processFamily(comm) { + if (/^(node|node-MainThread)$/.test(comm)) return 'node'; + if (/^(postgres|postmaster|pg_ctl|initdb)$/.test(comm)) return 'postgresql'; + if (/^(paperclip-runne|paperclip-runner.*|Runner)$/.test(comm)) return 'runner'; + if (/^(chrome|headless|chromium)/i.test(comm)) return 'browser'; + if (/^(rustc|cargo)$/.test(comm)) return 'rust-build'; + return null; +} +export function classifyLinuxProcesses(text, orchestratorPid = process.pid) { + assert.ok(typeof text === 'string' && text.trim(), 'Linux ps returned no process rows'); + assert.ok(Number.isSafeInteger(orchestratorPid) && orchestratorPid > 0, 'Invalid orchestrator PID'); + const seen = new Set(), matched = []; let rowCount = 0; + for (const line of text.trim().split(/\r?\n/)) { + const fields = line.match(/^\s*(\d+)\s+(\d+)\s+(.+?)\s+([DIRSTtWXZPK])\s*$/); + assert.ok(fields, 'Malformed Linux ps PID/PPID/comm/state row'); + const row = { pid: Number(fields[1]), ppid: Number(fields[2]), comm: fields[3].trim(), state: fields[4] }; + assert.ok(Number.isSafeInteger(row.pid) && row.pid > 0 && Number.isSafeInteger(row.ppid) && row.ppid >= 0 && row.comm.length > 0 && row.comm.length <= 256 && !row.comm.includes('\0'), 'Malformed Linux ps process fields'); + assert.equal(seen.has(row.pid), false, 'Duplicate Linux ps PID'); seen.add(row.pid); rowCount++; + const family = processFamily(row.comm); + if (family) matched.push({ ...row, family, allowed: row.pid === orchestratorPid && !['Z', 'X'].includes(row.state) }); + } + assert.ok(seen.has(orchestratorPid), 'Linux ps omitted the current orchestrator PID'); + return { rowCount, matched, blockers: matched.filter(row => !row.allowed) }; +} +function linuxProcessRows(timeoutMillis) { + return execFileSync('/bin/ps', ['-eo', 'pid=,ppid=,comm=,state='], { + encoding: 'utf8', timeout: timeoutMillis, maxBuffer: 2 * 1024 * 1024, env: { LC_ALL: 'C' }, + }); +} +export async function physicalQuietGuard({ receiptFile, phase, armStage, primaryFailureAlreadyPresent = false, + platform = process.platform, orchestratorPid = process.pid, readPs = linuxProcessRows, + now = () => performance.now(), pause = milliseconds => new Promise(resolve => setTimeout(resolve, milliseconds)) }) { + absolute(receiptFile, 'physical quiet receipt'); + const started = now(), receipt = { schema: QUIET_SCHEMA, scope: QUIET_SCOPE, phase, armStage, platform, orchestratorPid, + fields: ['pid', 'ppid', 'comm', 'state'], zombiePolicy: QUIET_ZOMBIE_POLICY, + resourceSettleLimitMillis: QUIET_SETTLE_MILLIS, pollIntervalMillis: QUIET_POLL_MILLIS, psTimeoutMillis: QUIET_PS_TIMEOUT_MILLIS, + primaryFailureAlreadyPresent, startedAtMs: Date.now(), passed: false, observations: [] }; + let failure; + try { + assert.equal(platform, 'linux', 'Accepted H5 physical quiet guard requires Linux'); + assert.ok(['pre-arm', 'post-arm'].includes(phase), 'Unknown physical quiet phase'); + for (;;) { + const remaining = QUIET_SETTLE_MILLIS - (now() - started); + assert.ok(remaining > 0, 'Physical quiet settle deadline expired; stop for cleanup'); + const observed = classifyLinuxProcesses(readPs(Math.max(1, Math.min(QUIET_PS_TIMEOUT_MILLIS, Math.floor(remaining)))), orchestratorPid); + receipt.observations.push({ atMs: Date.now(), elapsedMillis: now() - started, ...observed }); + assert.ok(now() - started < QUIET_SETTLE_MILLIS, 'Physical quiet settle deadline expired; stop for cleanup'); + if (observed.blockers.length === 0) { receipt.passed = true; break; } + const wait = Math.min(QUIET_POLL_MILLIS, QUIET_SETTLE_MILLIS - (now() - started)); + assert.ok(wait > 0, 'Physical quiet settle deadline expired; stop for cleanup'); + await pause(wait); + } + } catch (error) { + failure = error; + receipt.failure = { name: error instanceof Error ? error.name : 'UnknownError', message: error instanceof Error ? error.message : String(error) }; + } + receipt.completedAtMs = Date.now(); receipt.elapsedMillis = now() - started; + const bytes = Buffer.from(JSON.stringify(receipt, null, 2) + '\n'); + await writeFile(receiptFile, bytes, { flag: 'wx' }); + const result = { file: receiptFile, bytes: bytes.length, sha256: hash(bytes), passed: receipt.passed }; + if (failure) { + const error = new Error(`Physical quiet guard failed (${phase}); receipt=${receiptFile}; sha256=${result.sha256}`, { cause: failure }); + Object.assign(error, { receiptFile, receiptSha256: result.sha256 }); throw error; + } + return result; +} +export async function assertNoPreferredUiShadow(appRoot, statPath = lstat) { + const preferredIndex = path.join(appRoot, 'server/ui-dist/index.html'), preferredDirectory = path.dirname(preferredIndex); + try { await statPath(preferredIndex); assert.fail('Preferred server/ui-dist/index.html exists (including a symlink); ui/dist inventory cannot attest the selected UI'); } + catch (error) { if (error.code !== 'ENOENT') throw error; } + try { assert.equal((await statPath(preferredDirectory)).isSymbolicLink(), false, 'Preferred server/ui-dist directory symlink prevents the absence guard'); } + catch (error) { if (error.code !== 'ENOENT') throw error; } + return { checkedAtMs: Date.now(), preferredIndex, preferredIndexAbsent: true, preferredDirectorySymlink: false, + selectedUiRoot: path.join(appRoot, 'ui/dist'), + assumption: 'Preferred server/ui-dist index absent at preflight; inventoried ui/dist is the compiled fallback. No routing/build changes or future immutability claim.' }; +} + +async function checkFrozen(context, selectedVariant) { + await checkReleaseBytes(context); + for (const key of ['DATABASE_URL', 'PAPERCLIP_RUNNER_STATE_DIR', 'PAPERCLIP_INGESTION_SERVER_COMMAND', 'PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_DIR', + 'PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_FS', 'PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_INSPECTOR', 'PAPERCLIP_INGESTION_BROWSER_CAPTURE', 'NODE_OPTIONS']) + assert.ok(!process.env[key], `Acceptance rejects ambient ${key}`); + const { release, plan } = context, actualInventories = {}; + for (const variant of plan.requiredVariants) { + const build = release.perVariantBuilds[variant], qualified = await validateVariantBuild(variant, build); + assert.equal(execFileSync('git', ['rev-parse', 'HEAD'], { cwd: build.appRoot, encoding: 'utf8' }).trim(), build.revision, `${variant} SUT revision changed`); + assert.equal(execFileSync('git', ['status', '--short'], { cwd: build.appRoot, encoding: 'utf8' }).trim(), '', `${variant} SUT is dirty`); + assert.equal((await getJson(path.join(build.appRoot, 'server/dist/build-info.json'))).commit, build.revision); + const binaryFile = path.join(build.appRoot, 'packages/paperclip-runner/runner/target/release/paperclip-runnerd'); + const native = qualified.receipt.nativeStageHashes.find(row => row.variant === variant); + const actualTarget = await realpath(path.join(build.appRoot, 'packages/paperclip-runner/runner/target')); + assert.equal(actualTarget, native.targetRealPath, `${variant} actual native target differs from qualified receipt`); + assert.equal(await realpath(binaryFile), path.join(actualTarget, 'release/paperclip-runnerd'), `${variant} native binary resolves outside its exact qualified target`); + assert.equal(hash(await readFile(binaryFile)), build.binarySha256, `${variant} actual native binary changed`); + for (const [file, object] of Object.entries(qualified.receipt.rustInputObjects)) + assert.equal(execFileSync('git', ['rev-parse', `${build.revision}:packages/paperclip-runner/${file}`], { cwd: build.appRoot, encoding: 'utf8' }).trim(), object, `${variant} qualified Rust input changed: ${file}`); + assert.equal(execFileSync('git', ['rev-parse', `${build.revision}:packages/paperclip-runner/runner`], { cwd: build.appRoot, encoding: 'utf8' }).trim(), build.rustSourceTree); + if (!selectedVariant || selectedVariant === variant) actualInventories[variant] = { + ...await validateActualInventory(build, qualified.manifest), uiServing: await assertNoPreferredUiShadow(build.appRoot), + }; + } + assert.equal(execFileSync('git', ['rev-parse', 'HEAD'], { cwd: release.harnessRoot, encoding: 'utf8' }).trim(), release.harnessRevision); + assert.equal(execFileSync('git', ['status', '--short'], { cwd: release.harnessRoot, encoding: 'utf8' }).trim(), ''); + assert.equal(hash(Buffer.concat(await Promise.all(protocolFiles.map(file => readFile(path.join(release.harnessRoot, 'tests/e2e', file)))))), release.protocolSha256); + for (const [file, expected] of [['scripts/compare-native-ingestion-browser.ts', release.comparatorSha256], ['scripts/native-ingestion-h5-inventory.mjs', release.inventoryHelperSha256], ['tests/e2e/fixtures/native-ingestion-h5.md', release.protocolDocumentSha256], ['scripts/native-ingestion-performance-gates.mjs', release.gatesSha256], ['pnpm-lock.yaml', release.lockSha256]]) + assert.equal(hash(await readFile(path.join(release.harnessRoot, file))), expected, `Frozen H5 support changed: ${file}`); + await checkPriorStages(context); + if (plan.noiseStage) await checkNoise(context); + return { checkedAtMs: Date.now(), releaseFileSha256: context.releaseBytesSha256, actualInventories }; +} + +async function execute(context, step) { + const index = ++context.commandIndex, startedAt = new Date().toISOString(), log = path.join(context.out, `${step.stage}.log`); + const command = { index, stage: step.stage, cwd: step.cwd, executable: process.execPath, args: step.args, environmentOverrides: step.env ?? {}, startedAt, log, + resourceWatchdogMillis: RESOURCE_WATCHDOG_MILLIS, resourceWatchdogScope: 'Resource-only cap including startup/test/teardown; original SLA and H5 observation limits remain driver-owned' }; + context.commands.push(command); + const save = () => saveJson(path.join(context.out, `${context.plan.stage}-commands.json`), context.commands); + await save(); console.log(`START ${step.stage} ${startedAt}; log=${log}`); + const stream = (await open(log, 'wx')).createWriteStream(); + const child = spawn(process.execPath, step.args, { cwd: step.cwd, env: step.launchEnvironment ?? { ...process.env, ...step.env }, stdio: ['ignore', 'pipe', 'pipe'] }); + command.childPid = child.pid; await save(); + const interrupt = () => child.kill('SIGINT'), terminate = () => child.kill('SIGTERM'); + process.once('SIGINT', interrupt); process.once('SIGTERM', terminate); + let forcedStop; + const watchdog = setTimeout(() => { + command.watchdogExpired = true; child.kill('SIGTERM'); + forcedStop = setTimeout(() => { command.forcedDriverKill = true; child.kill('SIGKILL'); }, 30_000); + }, RESOURCE_WATCHDOG_MILLIS); + child.stdout.pipe(stream, { end: false }); child.stderr.pipe(stream, { end: false }); + let exit; + try { exit = await new Promise((resolve, reject) => { child.once('error', reject); child.once('close', (code, signal) => resolve({ code, signal })); }); } + finally { clearTimeout(watchdog); clearTimeout(forcedStop); process.off('SIGINT', interrupt); process.off('SIGTERM', terminate); await new Promise(resolve => stream.end(resolve)); } + Object.assign(command, { finishedAt: new Date().toISOString(), ...exit }); await save(); + console.log(`END ${step.stage} code=${exit.code} signal=${exit.signal ?? 'none'}`); + assert.equal(command.watchdogExpired, undefined, 'Resource watchdog expired; stop for physical cleanup, never accept or advance'); + assert.equal(exit.code, 0, `Stage ${step.stage} failed; retain raw files and perform physical cleanup before continuing`); +} + +async function freePort() { + const server = net.createServer(); + await new Promise((resolve, reject) => { server.once('error', reject); server.listen(0, '127.0.0.1', resolve); }); + const port = server.address().port; await new Promise(resolve => server.close(resolve)); return port; +} +async function assertPortReleased(port) { + const server = net.createServer(); + await new Promise((resolve, reject) => { server.once('error', reject); server.listen(port, '127.0.0.1', resolve); }); + await new Promise(resolve => server.close(resolve)); +} +async function extractEvidence(directory) { + const playwrightFile = path.join(directory, 'playwright.json'), report = await getJson(playwrightFile), attachments = []; + const visit = node => { if (Array.isArray(node)) for (const child of node) visit(child); else if (node && typeof node === 'object') { + if (Array.isArray(node.attachments)) attachments.push(...node.attachments.filter(value => value.name === 'native-ingestion-evidence')); + for (const [key, child] of Object.entries(node)) if (key !== 'attachments') visit(child); + } }; + visit(report); assert.equal(attachments.length, 1, 'Exactly one native-ingestion evidence attachment is required'); + const attachment = attachments[0]; + const evidence = attachment.body ? JSON.parse(Buffer.from(attachment.body, 'base64').toString()) : await getJson(path.resolve(directory, attachment.path)); + const file = path.join(directory, 'native-ingestion-evidence.json'); await saveJson(file, evidence, true); + return { file, evidence, playwrightFile }; +} + +async function armSidecar(directory, variant, build) { + const destination = path.join(directory, 'inventory'); await mkdir(destination); + const copies = {}; + for (const [key, sha] of [['manifestFile', 'manifestFileSha256'], ['inventorySummaryFile', 'inventorySummarySha256'], ['qualifiedReceiptFile', 'qualifiedReceiptSha256']]) { + const bytes = await readFile(build[key]); assert.equal(hash(bytes), build[sha], 'Per-arm provenance changed while copying'); + const file = path.join(destination, path.basename(build[key])); await writeFile(file, bytes, { flag: 'wx' }); copies[key] = path.relative(directory, file); + } + const sidecar = { schema: 'paperclip.native-ingestion.arm-inventory.v1', variant, sourceRevision: build.revision, compiledBuildRevision: build.revision, + playwrightReportSha256: hash(await readFile(path.join(directory, 'playwright.json'))), ...copies, + manifestFileSha256: build.manifestFileSha256, aggregateSha256: build.aggregateSha256, + qualifiedReceiptSha256: build.qualifiedReceiptSha256, inventorySummarySha256: build.inventorySummarySha256 }; + const file = path.join(directory, 'arm-inventory.json'); await saveJson(file, sidecar, true); return file; +} + +async function armArtifactFiles(result) { + const sidecar = await getJson(result.inventoryFile), base = path.dirname(result.inventoryFile); + return [result.file, result.playwrightFile, result.inventoryFile, + ...['manifestFile', 'inventorySummaryFile', 'qualifiedReceiptFile'].map(key => path.resolve(base, sidecar[key])), + path.join(base, 'pre-arm-inventory-check.json'), path.join(base, 'physical-http-port-check.json'), + path.join(base, 'physical-quiet-pre.json'), path.join(base, 'physical-quiet-post.json')]; +} +export async function sealArmArtifacts(out, result) { + const canonicalRoot = await realpath(out), files = await armArtifactFiles(result), seen = new Set(), seals = []; + for (const file of files) { + absolute(file, 'raw arm artifact'); const canonical = await realpath(file); + assert.ok(canonical.startsWith(`${canonicalRoot}${path.sep}`), 'Raw arm artifact resolves outside owned output'); + assert.equal(seen.has(canonical), false, 'Duplicate raw arm artifact'); seen.add(canonical); + const bytes = await readFile(file); seals.push({ file, bytes: bytes.length, sha256: hash(bytes) }); + } + return seals; +} +export async function verifyRawArtifactSeals(out, manifest) { + const canonicalRoot = await realpath(out); + for (const entry of manifest) for (const arm of ['baseline', 'candidate']) { + const seals = entry[`${arm}Artifacts`]; assert.ok(Array.isArray(seals) && seals.length === 10, 'Raw arm byte seals are incomplete'); + const seen = new Set(); + for (const seal of seals) { + absolute(seal.file, 'sealed arm artifact'); digest(seal.sha256, 'sealed arm artifact'); + assert.equal(seen.has(seal.file), false, 'Duplicate sealed arm artifact'); seen.add(seal.file); + assert.ok((await realpath(seal.file)).startsWith(`${canonicalRoot}${path.sep}`), 'Sealed arm artifact resolves outside output'); + const bytes = await readFile(seal.file); + assert.equal(bytes.length, seal.bytes, 'Raw arm artifact size changed after timing'); + assert.equal(hash(bytes), seal.sha256, 'Raw arm artifact bytes changed after timing'); + } + const expected = await armArtifactFiles({ file: entry[`${arm}File`], playwrightFile: entry[`${arm}PlaywrightFile`], inventoryFile: entry[`${arm}InventoryFile`] }); + assert.deepEqual([...seen].sort(), expected.sort(), 'Raw seals do not cover this exact evidence/PW/sidecar/inventory set'); + for (const [filename, phase] of [['physical-quiet-pre.json', 'pre-arm'], ['physical-quiet-post.json', 'post-arm']]) { + const receipt = await getJson(path.join(path.dirname(entry[`${arm}InventoryFile`]), filename)); + assert.equal(receipt.schema, QUIET_SCHEMA); assert.equal(receipt.passed, true); assert.equal(receipt.platform, 'linux'); + assert.equal(receipt.phase, phase); assert.equal(receipt.zombiePolicy, QUIET_ZOMBIE_POLICY); + } + } +} + +function physicalGuardReferences(manifest) { + return manifest.flatMap((entry, pair) => ['baseline', 'candidate'].map(arm => ({ pair: pair + 1, arm, + receipts: entry[`${arm}Artifacts`].filter(seal => /^physical-quiet-(pre|post)\.json$/.test(path.basename(seal.file))) }))); +} + +export async function assertPathNode(environment, approvedExecutable, workingDirectory, + { run = execFileSync, resolve = realpath, currentExecutable = process.execPath } = {}) { + absolute(workingDirectory, 'PATH Node working directory'); absolute(approvedExecutable, 'Approved Node executable'); + assert.ok(typeof environment.PATH === 'string' && environment.PATH.length > 0, 'Actual launch PATH is missing'); + const actual = JSON.parse(run('node', ['-p', 'JSON.stringify({executable:process.execPath,version:process.version,cwd:process.cwd()})'], + { env: environment, cwd: workingDirectory, encoding: 'utf8', timeout: 10000 })); + assert.equal(actual.version, 'v26.4.0', 'PATH node version differs from approved VM runtime'); + absolute(actual.executable, 'PATH-selected Node executable'); absolute(actual.cwd, 'PATH-selected Node cwd'); + const canonicalExecutable = await resolve(actual.executable), approvedCanonicalExecutable = await resolve(approvedExecutable); + assert.equal(canonicalExecutable, approvedCanonicalExecutable, 'PATH node executable differs from approved runtime'); + assert.equal(approvedCanonicalExecutable, await resolve(currentExecutable), 'Approved Node differs from orchestrator runtime'); + const canonicalCwd = await resolve(actual.cwd), approvedCanonicalCwd = await resolve(workingDirectory); + assert.equal(canonicalCwd, approvedCanonicalCwd, 'PATH node did not run in the actual appRoot cwd'); + return { ...actual, canonicalExecutable, approvedCanonicalExecutable, canonicalCwd, approvedCanonicalCwd, + PATH: environment.PATH, requestedCwd: workingDirectory }; +} + +async function browserRun(context, armStage, variant) { + const preflight = await checkFrozen(context, variant); // ALL receipts; selected live artifact inventory before EVERY arm. + const { release } = context, build = release.perVariantBuilds[variant], directory = path.join(context.out, armStage); + await mkdir(directory, { recursive: false }); + await physicalQuietGuard({ receiptFile: path.join(directory, 'physical-quiet-pre.json'), phase: 'pre-arm', armStage }); + const port = await freePort(); + const environmentOverrides = { PAPERCLIP_INGESTION_OBSERVATION_MODE: MODE, PAPERCLIP_RUNNER_BINARY: path.join(build.appRoot, 'packages/paperclip-runner/runner/target/release/paperclip-runnerd'), + PAPERCLIP_RUNNER_REMOTE_BINARY_PATH: path.join(build.appRoot, 'packages/paperclip-runner/runner/target/release/paperclip-runnerd'), + PAPERCLIP_INGESTION_TRANSPORT: 'sandbox-remote-with-local-stop', PAPERCLIP_E2E_PORT: String(port), PAPERCLIP_INGESTION_SMOKE: '0', + PAPERCLIP_INGESTION_VARIANT: `${armStage}-${variant}`, PAPERCLIP_INGESTION_APP_ROOT: build.appRoot, PAPERCLIP_INGESTION_EVIDENCE_DIR: directory, + PAPERCLIP_INGESTION_DRIVER_DEPENDENCIES: 'Independent frozen H5 driver install; each measured variant owns its separately qualified application/native build', PATH: process.env.PATH }; + const launchEnvironment = { ...process.env, ...environmentOverrides }; + const pathNode = await assertPathNode(launchEnvironment, process.execPath, build.appRoot); + await saveJson(path.join(directory, 'pre-arm-inventory-check.json'), { ...preflight, pathNode }, true); + let primaryError, inventoryFile, extracted; + try { + await execute(context, { stage: armStage, cwd: release.harnessRoot, + args: ['node_modules/@playwright/test/cli.js', 'test', '--config', 'tests/e2e/playwright-native-ingestion.config.ts'], + env: environmentOverrides, launchEnvironment }); + } catch (error) { primaryError = error; } + try { + inventoryFile = await armSidecar(directory, variant, build); extracted = await extractEvidence(directory); + if (!primaryError) { + const { evidence } = extracted; + assert.equal(evidence.complete, true); assert.equal(evidence.observationComplete, true); assert.equal(evidence.observationMode, MODE); + assert.equal(evidence.workloadSha256, WORKLOAD_SHA256); assert.equal(evidence.protocolVersion, PROTOCOL); + assert.equal(evidence.transport, 'sandbox-remote-with-local-stop'); assert.equal(evidence.cleanup?.complete, true); assert.deepEqual(evidence.cleanup.errors, []); + assert.equal(evidence.measurementSha256, release.protocolSha256); assert.equal(evidence.harnessProvenance?.sourceRevision, release.harnessRevision); assert.equal(evidence.harnessProvenance?.sourceStatus, ''); + assert.equal(evidence.sourceRevision, build.revision); assert.equal(evidence.compiledBuildRevision, build.revision); + assert.equal(evidence.serverArtifact, 'compiled'); assert.equal(evidence.qualificationOnly, false); assert.equal(evidence.diagnosticOnly, false); assert.equal(evidence.performanceAcceptanceEligible, true); + assert.equal(evidence.browserRecording?.trace, 'off'); assert.equal(evidence.browserRecording?.video, 'off'); assert.equal(evidence.browserRecording?.screencast, false); + const loaded = evidence.cases.find(item => item.name === 'burst512'); assert.equal(loaded.runs, 100); assert.equal(loaded.participants, 100); assert.equal(loaded.durable.length, 100); + const { validateArmInventory } = await import(pathToFileURL(path.join(release.harnessRoot, 'scripts/native-ingestion-h5-inventory.mjs')).href); + await validateArmInventory(inventoryFile, { evidence, owningPlaywright: { reportSha256: hash(await readFile(extracted.playwrightFile)) } }); + } + } catch (error) { primaryError ??= error; } + try { await assertPortReleased(port); await saveJson(path.join(directory, 'physical-http-port-check.json'), { released: true, port, scope: 'Owned HTTP listener only; PG/Runner/provider physical cleanup is separately required before the next release' }, true); } + catch (error) { primaryError ??= error; } + try { await physicalQuietGuard({ receiptFile: path.join(directory, 'physical-quiet-post.json'), phase: 'post-arm', armStage, + primaryFailureAlreadyPresent: Boolean(primaryError) }); } + catch (error) { + if (primaryError) console.error(`Secondary physical quiet failure; primary fixture failure is retained: ${error.message}`); + primaryError ??= error; + } // Failed guard receipt and SHA remain independent evidence; the fixture failure stays primary. + if (primaryError) throw primaryError; + const result = { file: extracted.file, playwrightFile: extracted.playwrightFile, inventoryFile }; + return { ...result, artifacts: await sealArmArtifacts(context.out, result) }; +} + +function comparatorArgs(context, plan, pairsFile, output, originalProof) { + const args = ['scripts/compare-native-ingestion-browser.ts', `--observation-mode=${MODE}`, `--phase=${plan.aa ? 'aa' : 'compare'}`, + `--expect=${plan.expectation}`, `--pairs=${pairsFile}`, `--output=${output}`]; + if (!plan.aa) args.push(`--candidate-policy=${plan.candidatePolicy}`, `--noise-report=${path.join(context.out, `${plan.noiseStage}.json`)}`, + ...(originalProof ? [`--original-h4e-proof=${originalProof.proofFile}`, `--original-h4e-inventory=${originalProof.inventoryFile}`, '--require-acceptance'] : ['--require-performance'])); + return args; +} + +async function browserSeries(context) { + const { plan, release } = context, manifest = [], file = path.join(context.out, `${plan.stage}-pairs.json`), comparisonFile = path.join(context.out, `${plan.stage}.json`); + await access(comparisonFile).then(() => assert.fail('Prior comparison output already exists'), error => { if (error.code !== 'ENOENT') throw error; }); + await saveJson(file, manifest, true); + for (let pair = 0; pair < 5; pair++) { + const entry = {}, arms = pair % 2 ? [['candidate', plan.candidate], ['baseline', plan.baseline]] : [['baseline', plan.baseline], ['candidate', plan.candidate]]; + for (const [arm, variant] of arms) { + const result = await browserRun(context, `${plan.stage}-pair${pair + 1}-${arm}`, variant); + entry[`${arm}File`] = result.file; entry[`${arm}PlaywrightFile`] = result.playwrightFile; entry[`${arm}InventoryFile`] = result.inventoryFile; entry[`${arm}Artifacts`] = result.artifacts; + } + manifest.push(entry); await saveJson(file, manifest); + } + await checkFrozen(context); // Same exact release/noise path and bytes pre-comparator. + await verifyRawArtifactSeals(context.out, manifest); + const manifestSha256 = hash(await readFile(file)); + await execute(context, { stage: `${plan.stage}-comparison`, cwd: release.harnessRoot, args: comparatorArgs(context, plan, file, comparisonFile) }); + assert.equal(hash(await readFile(file)), manifestSha256, 'Raw paired manifest changed during comparison'); + await verifyRawArtifactSeals(context.out, manifest); + const result = await getJson(comparisonFile); checkComparisonIdentity(result, context); + if (!plan.aa) assert.equal(result.performancePassed, true); + await saveJson(path.join(context.out, `${plan.stage}.completion.json`), { stage: plan.stage, complete: true, fullArms: 10, completedAtMs: Date.now(), + releaseFile: context.rootReleaseCopy, originalReleaseFile: context.releaseFile, releaseFileSha256: context.releaseBytesSha256, harnessPins: harnessFields(release), + perVariantBuilds: Object.fromEntries(plan.requiredVariants.map(variant => [variant, release.perVariantBuilds[variant]])), + manifestFile: file, manifestSha256, comparisonFile, comparisonSha256: hash(await readFile(comparisonFile)), + physicalGuards: physicalGuardReferences(manifest), + ...(plan.noiseStage ? { noiseReportFile: release.noiseReportFile, noiseReportSha256: release.noiseReportSha256 } : {}), + performancePassed: result.performancePassed ?? null, acceptancePassed: result.acceptancePassed ?? null, + finalOriginalH4eConfirmationPending: plan.candidatePolicy === 'final-b2' }, true); +} + +async function checkFinalizationInputs(context) { + const proof = context.release.originalH4eProof; + assert.ok(proof, 'One later unchanged H4e proof is required for both final comparisons'); + for (const [key, sha] of [['proofFile', 'proofSha256'], ['inventoryFile', 'inventorySha256']]) await hashedJson(proof[key], proof[sha], `Original H4e ${key}`); + const proofReport = await getJson(proof.proofFile), proofAttempts = []; + const visit = value => { + if (Array.isArray(value)) for (const child of value) visit(child); + else if (value && typeof value === 'object') { + if (Array.isArray(value.results) && Object.hasOwn(value, 'expectedStatus')) proofAttempts.push(...value.results); + for (const [key, child] of Object.entries(value)) if (key !== 'results') visit(child); + } + }; + visit(proofReport); assert.equal(proofAttempts.length, 1, 'One later unchanged original H4e attempt must confirm both stages'); + const proofStartedAtMs = Date.parse(proofAttempts[0].startTime); + assert.ok(Number.isFinite(proofStartedAtMs), 'Original H4e proof has no actual attempt start time'); + for (const stage of ['browser-b1-b2', 'browser-b0-b2']) { + const pin = context.release.completedSeries?.[stage]; assert.ok(pin, `Missing frozen completed ${stage}`); + assert.equal(pin.file, path.join(context.out, `${stage}.completion.json`)); + const completion = (await hashedJson(pin.file, pin.sha256, `${stage} completion`)).value; + assert.equal(completion.stage, stage); assert.equal(completion.complete, true); assert.equal(completion.fullArms, 10); assert.equal(completion.performancePassed, true); + assert.ok(Number.isFinite(completion.completedAtMs) && completion.completedAtMs < proofStartedAtMs, 'Original H4e proof must run after BOTH timing stages complete'); + assert.deepEqual(completion.harnessPins, harnessFields(context.release)); + const plan = stagePlan(stage); + for (const variant of plan.requiredVariants) assert.deepEqual(completion.perVariantBuilds[variant], context.release.perVariantBuilds[variant], 'Finalization changed a measured qualified build'); + assert.equal(completion.manifestFile, path.join(context.out, `${stage}-pairs.json`)); + assert.equal(completion.comparisonFile, path.join(context.out, `${stage}.json`)); + assert.equal(hash(await readFile(completion.manifestFile)), completion.manifestSha256, 'Raw timing manifest changed before finalization'); + const manifest = await getJson(completion.manifestFile); assert.equal(manifest.length, 5); + await verifyRawArtifactSeals(context.out, manifest); + assert.deepEqual(completion.physicalGuards, physicalGuardReferences(manifest), 'Completion physical-guard receipts differ from sealed arm bytes'); + assert.equal(hash(await readFile(completion.comparisonFile)), completion.comparisonSha256, 'Preliminary performance assessment changed'); + assert.equal(completion.noiseReportFile, path.join(context.out, `${plan.noiseStage}.json`), 'Finalization rebound the frozen noise path'); + assert.equal(hash(await readFile(completion.noiseReportFile)), completion.noiseReportSha256, 'Frozen noise bytes changed before finalization'); + const originalRelease = await hashedJson(completion.releaseFile, completion.releaseFileSha256, `${stage} original release`); + assert.equal(originalRelease.value.noiseReportFile, completion.noiseReportFile); assert.equal(originalRelease.value.noiseReportSha256, completion.noiseReportSha256); + } +} + +async function finalizeSeries(context) { + await checkFrozen(context); await checkFinalizationInputs(context); + for (const stage of ['browser-b1-b2', 'browser-b0-b2']) { + await checkFrozen(context); await checkFinalizationInputs(context); + const pin = context.release.completedSeries[stage], completion = (await hashedJson(pin.file, pin.sha256, `${stage} completion`)).value; + const output = path.join(context.out, `${stage}.final.json`); + await access(output).then(() => assert.fail('Final assessment output already exists'), error => { if (error.code !== 'ENOENT') throw error; }); + await execute(context, { stage: `${stage}-original-h4e-finalization`, cwd: context.release.harnessRoot, + args: comparatorArgs(context, stagePlan(stage), completion.manifestFile, output, context.release.originalH4eProof) }); + await checkFinalizationInputs(context); + assert.equal((await getJson(output)).acceptancePassed, true); + } +} + +async function main() { + const stage = option('stage'); + if (!process.argv.includes('--execute') && !process.argv.includes('--validate-release')) { + console.log(JSON.stringify({ draft: true, observationMode: MODE, protocolVersion: PROTOCOL, workloadSha256: WORKLOAD_SHA256, + stages: STAGES.map(stagePlan), fullArms: 50, finalization: stagePlan(FINALIZE_STAGE), + baselines: BASELINES, originalReceiptSha256: OLD_RECEIPT, resourceWatchdogMillis: RESOURCE_WATCHDOG_MILLIS, + frozenH5: { revision: H5_REVISION, protocolSha256: H5_PROTOCOL_SHA256, comparatorSha256: H5_COMPARATOR_SHA256, inventoryHelperSha256: H5_INVENTORY_HELPER_SHA256, protocolDocumentSha256: H5_PROTOCOL_DOCUMENT_SHA256 }, + pendingPins: 'Future B2 has no descriptor until its actual new source and separately qualified native/application build are available. No H5 VM execution is released by this file.', + note: 'Plan only; no SUT/build/VM/network or arm starts. Root and independent peer review plus qualification release required.' }, null, 2)); return; + } + const out = option('output-root') ?? '/home/ubuntu/native-ingestion/evidence/h5/browser'; + const context = await loadRelease(option('release'), stage, out); + if (process.argv.includes('--validate-release') && !process.argv.includes('--execute')) { + console.log(JSON.stringify({ valid: true, stage, requiredVariants: context.plan.requiredVariants, + validationScope: 'Static immutable release/receipt/manifest/noise/stage-order bindings only; no live SUT/driver/build attestation and no arm execution', releaseFileSha256: context.releaseBytesSha256 })); return; + } + await checkFrozen(context); await mkdir(out, { recursive: true }); + // Keep the exact release bytes as a distinct artifact, so later stages never + // require an owner to retain an overwriteable release-input filename. + context.rootReleaseCopy = path.join(out, `${stage}-root-release.json`); + const releaseBytes = await readFile(context.releaseFile); + assert.equal(hash(releaseBytes), context.releaseBytesSha256, 'Root release changed before preservation'); + await writeFile(context.rootReleaseCopy, releaseBytes, { flag: 'wx' }); + await saveJson(path.join(out, `${stage}-release-record.json`), { ...context.release, releaseFile: context.releaseFile, + releaseFileSha256: context.releaseBytesSha256, recordedAt: new Date().toISOString() }, true); + if (context.plan.finalizationOnly) await finalizeSeries(context); else await browserSeries(context); +} +if (process.argv[1] && path.resolve(process.argv[1]) === selfFile) await main(); diff --git a/scripts/native-ingestion/h5d/native-ingestion-validate-h5d-evidence.mjs b/scripts/native-ingestion/h5d/native-ingestion-validate-h5d-evidence.mjs new file mode 100644 index 00000000000..f5af7e38a61 --- /dev/null +++ b/scripts/native-ingestion/h5d/native-ingestion-validate-h5d-evidence.mjs @@ -0,0 +1,77 @@ +import assert from 'node:assert/strict'; +import {readFile,writeFile} from 'node:fs/promises'; +import {createHash} from 'node:crypto'; +import {execFileSync} from 'node:child_process'; +import path from 'node:path'; +import {pathToFileURL} from 'node:url'; +const directory=process.argv[2],harness=process.argv[3]??'/home/ubuntu/native-ingestion/harness-h5d'; +assert.ok(path.isAbsolute(directory)&&path.isAbsolute(harness)); +const hash=bytes=>createHash('sha256').update(bytes).digest('hex'); +const helper=path.join(harness,'tests/e2e/fixtures/native-ingestion-h5.mjs'); +assert.equal(hash(await readFile(helper)),'7d0629849fe776faa51a2ca1d987429be1a529df38efa8d92466bff6a8dabff2'); +assert.equal(execFileSync('git',['rev-parse','HEAD'],{cwd:harness,encoding:'utf8'}).trim(),'2df9897e1ce2ca90be29fa097d776c3766e217ad'); +const {assertParticipantSetup,assertParticipantTopology,assertSettlementObservation,assertRecordedFailures,assessH5Arm,H5_REGISTRATION,H5_WORKLOADS,H5_LIMITS}=await import(pathToFileURL(helper).href); +const evidenceBytes=await readFile(path.join(directory,'native-ingestion-evidence.json')),evidence=JSON.parse(evidenceBytes); +const pwBytes=await readFile(path.join(directory,'playwright.json')),pw=JSON.parse(pwBytes),tests=[]; +assert.deepEqual(pw.errors??[],[]); +function visit(value){if(Array.isArray(value))for(const child of value)visit(child);else if(value&&typeof value==='object'){ + if(Array.isArray(value.results)&&Object.hasOwn(value,'expectedStatus'))tests.push(value); + for(const [key,child]of Object.entries(value))if(key!=='results')visit(child); +}} +visit(pw);assert.equal(tests.length,1);const test=tests[0];assert.equal(test.expectedStatus,'passed');assert.equal(test.results.length,1); +const result=test.results[0];assert.equal(result.status,'passed');assert.equal(result.retry,0);assert.deepEqual(result.errors,[]);assert.ok(!result.error); +const attachments=result.attachments.filter(value=>value.name==='native-ingestion-evidence');assert.equal(attachments.length,1); +const attached=attachments[0].body?Buffer.from(attachments[0].body,'base64'):await readFile(path.resolve(directory,attachments[0].path)); +assert.equal(hash(attached),hash(evidenceBytes),'Owning Playwright attempt must attach these exact evidence bytes'); +const owner={reportFile:path.join(directory,'playwright.json'),reportSha256:hash(pwBytes),status:result.status,retry:result.retry,errors:result.errors,startedAt:result.startTime,durationMs:result.duration,timeoutMs:test.timeout}; +assert.equal(evidence.complete,true);assert.equal(evidence.observationComplete,true);assert.equal(evidence.protocolVersion,'v5c-independent-agents-complete-settlement-original-sla'); +assert.equal(evidence.measurementSha256,'e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4'); +assert.equal(evidence.harnessProvenance.sourceRevision,'2df9897e1ce2ca90be29fa097d776c3766e217ad');assert.equal(evidence.harnessProvenance.sourceStatus,''); +assert.equal(evidence.diagnosticOnly,false);assert.equal(evidence.browserRecording.trace,'off');assert.equal(evidence.browserRecording.video,'off');assert.equal(evidence.browserRecording.screencast,false); +const outer=JSON.parse(await readFile(path.join(directory,'owner-receipt.json'),'utf8')); +assert.equal(typeof outer.fullFixture,'boolean');assert.equal(typeof evidence.qualificationOnly,'boolean'); +assert.equal(outer.fullFixture,!evidence.qualificationOnly,'Owning command FULL/SMOKE mode must match its evidence'); +assert.equal(outer.sourceRevision,evidence.sourceRevision);assert.equal(outer.sourceRevision,evidence.compiledBuildRevision);assert.equal(evidence.serverArtifact,'compiled'); +assert.equal(outer.harnessRevision,evidence.harnessProvenance.sourceRevision); +const pathPreflight=JSON.parse(await readFile(path.join(directory,'path-node-preflight.json'),'utf8')); +assert.deepEqual(pathPreflight,outer.nodePathPreflight); +assert.equal(pathPreflight.executable,'/opt/native-ingestion/node-v26.4.0-linux-x64/bin/node'); +assert.equal(pathPreflight.cwd,outer.environmentOverrides.PAPERCLIP_INGESTION_APP_ROOT); +assert.equal(pathPreflight.pathNode.version,'v26.4.0'); +for(const key of ['executable','cwd','canonicalCwd','approvedCanonicalCwd','canonicalExecutable','approvedCanonicalExecutable']) + assert.ok(typeof pathPreflight.pathNode[key]==='string'&&path.isAbsolute(pathPreflight.pathNode[key]),'Missing absolute PATH Node proof field'); +assert.ok(typeof pathPreflight.pathNode.PATH==='string'&&pathPreflight.pathNode.PATH.length>0,'Missing recorded launch PATH'); +assert.equal(pathPreflight.pathNode.requestedCwd,pathPreflight.cwd); +assert.equal(pathPreflight.pathNode.canonicalCwd,pathPreflight.pathNode.approvedCanonicalCwd); +assert.equal(pathPreflight.pathNode.canonicalExecutable,pathPreflight.pathNode.approvedCanonicalExecutable); +assert.equal(pathPreflight.pathNode.PATH,outer.environmentOverrides.PATH); +assert.equal(owner.timeoutMs,H5_LIMITS.fullInvocationMs); +const expectedCases=outer.fullFixture?H5_WORKLOADS:[ + {name:'qualification',runs:3,events:16,intervalMs:25,prelude:0,stop:false}, + {name:'operator-stop',runs:1,events:512,intervalMs:100,prelude:0,stop:true}, +]; +const matrix=item=>Object.fromEntries(['name','runs','events','intervalMs','prelude','stop'].map(key=>[key,item[key]])); +assert.deepEqual(evidence.cases.map(matrix),expectedCases.map(matrix),'Qualification requires the complete exact case matrix'); +assertParticipantSetup(evidence);const cases=[]; +for(const item of evidence.cases){ + assertParticipantTopology(evidence,item); + const observed=assertSettlementObservation(item.h5.terminalObservation,item); + const recorded=assertRecordedFailures(item.recordedFailures,item,evidence.companyId); + cases.push({name:item.name,runs:item.runs,observedSettlement:observed,recordedFailures:recorded, + observedTransitions:item.h5.terminalObservation.transitions,recordedFailureProof:item.recordedFailures, + recordedRetryRate:recorded.uniqueEverFailedRuns/item.runs}); +} +let assessment; +if(!evidence.qualificationOnly){assessment=assessH5Arm(evidence,owner);assert.ok(assessment.misses.every(value=>H5_REGISTRATION.knownBaselineMisses.includes(value)));} +assert.equal((await readFile(path.join(directory,'owner.exit'),'utf8')).trim(),'0'); +assert.equal(outer.passed,true);assert.equal(outer.cleanupComplete,true); +const quiet=JSON.parse(await readFile(path.join(directory,'physical-quiet-after.json'),'utf8'));assert.equal(quiet.passed,true);assert.equal(quiet.platform,'linux'); +const port=JSON.parse(await readFile(path.join(directory,'port-after.json'),'utf8'));assert.equal(port.nodeBindAndListenPassed,true); +const startup=await readFile(path.join(directory,'playwright.log'),'utf8');assert.match(startup,/Heartbeat\s+enabled\s*\(/,'Actual global scheduler must be enabled'); +const summary={complete:true,qualificationOnly:true,performanceAcceptanceEligible:false, + scope:'Read-only post-run qualification validation with exact frozen H5c helpers; a full observation may retain original-SLA misses and is not an accepted A/A sample', + evidenceSha256:hash(evidenceBytes),owningPlaywright:owner,helperSha256:hash(await readFile(helper)), + actualGlobalSchedulerConfiguredEnabled:true,assessment,cases}; +await writeFile(path.join(directory,'independent-settlement-validation.json'),JSON.stringify(summary,null,2)+'\n',{flag:'wx'}); +console.log(JSON.stringify({complete:true,qualificationOnly:true,performanceAcceptanceEligible:false,assessment, + cases:cases.map(value=>({name:value.name,runs:value.runs,recordedRetryRate:value.recordedRetryRate,recordedErrorClasses:value.recordedFailures.recordedErrorClasses}))})); diff --git a/scripts/native-ingestion/h5d/sources.json b/scripts/native-ingestion/h5d/sources.json new file mode 100644 index 00000000000..a33c60357d1 --- /dev/null +++ b/scripts/native-ingestion/h5d/sources.json @@ -0,0 +1,165 @@ +{ + "schema": "paperclip.native-ingestion.h5d-helper-sources.v1", + "scope": "Exact frozen sources and reviewed helper snapshots; packaging makes no performance claim.", + "repository": "https://github.com/paperclipai/paperclip.git", + "deliveryLockSha256AtPackaging": "7540075e96f9a93924a1e54775c17346c5760dd5d02902d83fb5228f918996ba", + "deliveryCheckoutWasExecutedDriver": false, + "mixedDeliveryTreeCannotReplaceEitherFrozenDriver": true, + "recordedVmRuntime": { + "node": "v26.4.0", + "pnpm": "9.15.4", + "otherRuntimeOutputs": "Exact absolute executable, argument and output pins from sealed runtime receipts; never guessed." + }, + "rawB2Role": "ca7cba171cd5198ed569d75012edc858041cbc44 is unoptimized B2 provenance, not a final candidate.", + "frozenDriver": { + "commit": "2df9897e1ce2ca90be29fa097d776c3766e217ad", + "tree": "d0604f6fb899715f7ed623494fab24d64445e12b", + "baseRevision": "e431ec63d41fe90c6fd68bd2fc2bb969feeca917", + "annotatedTag": "native-ingestion-h5d", + "tagObject": "47ef6d1a5d81c31d4602aad50666f67c58f4cdb4", + "measurementSha256": "e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4", + "driverLockSha256": "54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8", + "comparatorSha256": "09daef50fad5eef4dbda4e63a191749d0d167d3ae9d38bfd22df8721e2aac4a3", + "preservedWorkloadSha256": "3eea2f24c0c0c38134492a0a17e1b9a398335c085dd248f03b14c14bb5a19835" + }, + "sourceFreezeReceiptSha256": "d1b7a8d5a18bada9bf12aefb2ee98dbf3e5daef8e8a9adadab65c991004d091f", + "sourceOverlay": [ + { + "file": "scripts/compare-native-ingestion-browser.ts", + "sha256": "09daef50fad5eef4dbda4e63a191749d0d167d3ae9d38bfd22df8721e2aac4a3", + "byteIdenticalToFrozenDriver": true + }, + { + "file": "scripts/native-ingestion-performance-gates.test.mjs", + "sha256": "0724e6ed30ad0ad79851dbdd475e1f9e94c2028572b7b70b0c91af620227532e", + "byteIdenticalToFrozenDriver": true + }, + { + "file": "scripts/native-ingestion-h5.test.mjs", + "sha256": "47fe63eff16017db43f146e3e88991801ee2f97efc5e757754b2076e843a4fd5", + "byteIdenticalToFrozenDriver": true + }, + { + "file": "tests/e2e/fixtures/native-ingestion-h5.md", + "sha256": "c647cfb68b360882a3bf42771b35e37c52e0aba7bca01511f4ebfb9482bf45bc", + "byteIdenticalToFrozenDriver": true + }, + { + "file": "tests/e2e/fixtures/native-ingestion-h5.mjs", + "sha256": "7d0629849fe776faa51a2ca1d987429be1a529df38efa8d92466bff6a8dabff2", + "byteIdenticalToFrozenDriver": true + }, + { + "file": "tests/e2e/native-event-ingestion.journey.ts", + "sha256": "de0e8613e85aa9dc76cbd87b0f8418f25807266c60fa79fdf87ceec5b819711a", + "byteIdenticalToFrozenDriver": true + } + ], + "helpers": [ + { + "file": "native-ingestion-run-h5d-browser.mjs", + "originalTaskFile": "/private/tmp/native-ingestion-run-h5d-browser.mjs", + "bytes": 58316, + "sha256": "4afaa8ffcd764b498bada29c8916715b73d34462fd49152e44ab8437ffb330eb", + "packagingChanges": [] + }, + { + "file": "native-ingestion-linux-h5d-qualification.py", + "originalTaskFile": "/private/tmp/native-ingestion-linux-h5d-qualification.py", + "bytes": 7779, + "sha256": "9df879140eda2953e1794bf9d68e1e09457fd33d8ca397731b8bba255c8f8c6d", + "packagingChanges": [] + }, + { + "file": "native-ingestion-h5d-linux-readiness.mjs", + "originalTaskFile": "/private/tmp/native-ingestion-h5d-linux-readiness.mjs", + "bytes": 3278, + "sha256": "9a5b4ae1a543d937f4811e41fc814b8528df2adb8602f931dc7b4166f0c8e9c7", + "packagingChanges": [] + }, + { + "file": "native-ingestion-h5d-full-linux-readiness.mjs", + "originalTaskFile": "/private/tmp/native-ingestion-h5d-full-linux-readiness.mjs", + "bytes": 3286, + "sha256": "7ffae7a9aa278837be3e40e2e4ca2d77c3c76cfd881b492b76dd861fc80e7040", + "packagingChanges": [] + }, + { + "file": "native-ingestion-validate-h5d-evidence.mjs", + "originalTaskFile": "/private/tmp/native-ingestion-validate-h5d-evidence.mjs", + "bytes": 7590, + "sha256": "a733ea5af01fa25c2fba1af83110bbaec1a2de5a003b4c4e137b8654dec93c11", + "packagingChanges": [] + }, + { + "file": "native-ingestion-extract-h5d-qualification-foreground.py", + "originalTaskFile": "/private/tmp/native-ingestion-extract-h5d-qualification-foreground.py", + "bytes": 6339, + "sha256": "b1e74ffd6696d28ce0b5a060f98310c4f1963e40d860486cc1962898e8cfdcc1", + "packagingChanges": [] + } + ], + "sourceQualifiedAppRevisions": { + "B0": "bac53ec3bfeaf6546e7aaeaee516487b30534654", + "B1": "07061a601cdc20603c8c7a46f9ddbad7ac5f5939", + "B2": "ca7cba171cd5198ed569d75012edc858041cbc44" + }, + "qualifiedBuildReceiptSha256": "a1259ba760f3cbbe44817e333f9f620bcf3769fb0a6c16f9ce2f0f5b2ed60d94", + "postreaderRepair": { + "reviewedAtUtc": "2026-11-04T09:17:11.678514+00:00", + "oldReaderSha256": "968fbbfce1942e96495f299b69c77ac904db72facf112664f54be6fc9b4e0cc1", + "newReaderSha256": "b1e74ffd6696d28ce0b5a060f98310c4f1963e40d860486cc1962898e8cfdcc1", + "diffSha256": "60314f2893acd3b34bda9f3864f323a225fd6cdcda4c389ab73f3cd1e4aef628", + "rawReportSha256": "d12ad9128d6a30bc335a69761d6004b2b3618fc275a07a84ef5434cfdb00b1e1", + "owningPlaywrightPassed": true, + "oldInvalidArrayPositionAssumptionRed": true, + "actualPreludeRunIndex": 9, + "checks": [ + "actual immutable foreground run correlation", + "run-array permutation stays valid", + "reject wrong-prelude-member", + "reject wrong-provider-clock", + "reject wrong-foreground-token", + "reject wrong-authority-issue", + "reject duplicate-foreground-proof", + "reject missing-prelude-event", + "reject nonunique-prelude", + "reject prelude-not-before-epoch", + "reject short-retained-history" + ], + "correctedExtractorExit": 0, + "rawOwnerSourceAndMeasurementUnchanged": true, + "originalSealUntouched": true, + "scope": "Offline postreader defect repair only; qualification remains ineligible for noise/performance. No owning failed trial regraded; no runtime/test rerun." + }, + "qualificationAsOfUtc": "2026-11-04T09:30:00Z", + "qualificationFacts": { + "small": { + "passed": true, + "deltas": 48, + "results": 3, + "stopDeltas": 9, + "stopResults": 0, + "sealedFiles": 40 + }, + "full": { + "ownerPassed": true, + "ownerDurationMs": 853339, + "original900BudgetPassed": true, + "originalSlaMisses": [ + "burst512:streaming120" + ], + "loadedDeltasNearOriginalDeadline": 41, + "foregroundExpected": 512, + "loadedDurableDeltas": 51200, + "loadedResults": 100, + "recoveredProviderTransportFailures": 43, + "verifiedCompletionDurationMs": 402584, + "allLeasesClosed": true, + "cleanupQuietPassed": true + }, + "performanceNoiseOrGainPassed": false, + "historyTopology": "One 1000-delta foreground history under nine fresh background runs; not ten history-bearing runs." + }, + "excluded": "Unqualified counter wrapper and resource instrumentation are not packaged." +} diff --git a/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts b/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts index fe0d043aefa..7e1e9e4b737 100644 --- a/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts +++ b/server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts @@ -265,16 +265,13 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => { const [delivered] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id)); expect(delivered.runId).toBeTruthy(); // The receipt can commit before execution enters liveRunExecutions. - // Observe actual adapter admission before awaiting its drain. - await vi.waitFor(async () => { - const current = await heartbeat.getRun(delivered.runId!); - expect(countExecuteCallsForRun(delivered.runId!), JSON.stringify({ - status: current?.status, errorCode: current?.errorCode, - })).toBe(1); - }, { timeout: 5_000 }); - await heartbeat.waitForRunExecutionDrain(delivered.runId!); + // Join the tracked dispatch even when live registration has not started. + await heartbeat.drainActiveRunExecutions(); expect(countExecuteCallsForRun(delivered.runId!)).toBe(1); - expect((await heartbeat.getRun(delivered.runId!))?.contextSnapshot).toMatchObject({ wakeCommentIds: [comment.id] }); + expect(await heartbeat.getRun(delivered.runId!)).toMatchObject({ + status: "succeeded", + contextSnapshot: { wakeCommentIds: [comment.id] }, + }); await heartbeat.resumeQueuedRuns(); expect(countExecuteCallsForRun(delivered.runId!)).toBe(1); expect((await db.select().from(issueComments).where(eq(issueComments.id, comment.id)))[0]?.body).toBe(comment.body); diff --git a/tests/e2e/fixtures/native-ingestion-blocked-stop-authority.ts b/tests/e2e/fixtures/native-ingestion-blocked-stop-authority.ts new file mode 100644 index 00000000000..7df78a02111 --- /dev/null +++ b/tests/e2e/fixtures/native-ingestion-blocked-stop-authority.ts @@ -0,0 +1,138 @@ +// Bounded verbatim copy of the H5c authority/process proof, e431ec63; frozen measured source stays unchanged. +import { createHash } from "node:crypto"; +import { createRequire } from "node:module"; +import { lstat, readFile, readdir, realpath } from "node:fs/promises"; +import { hostname } from "node:os"; +import path from "node:path"; +import { expect } from "@playwright/test"; +export async function runtimeAuthority(input: { + appRoot: string; companyId: string; environmentId: string; issueIds: string[]; transport: "local" | "sandbox-remote"; + afterStop?: boolean; recordedFailureRunIds?: string[]; +}) { + const configuration = process.env.PAPERCLIP_E2E_SERVER_CONFIG; + if (!configuration) throw new Error("Missing owned server configuration"); + const config = JSON.parse(await readFile(configuration, "utf8")); + expect(config.database.mode).toBe("embedded-postgres"); + const dataDirectory = await realpath(config.database.embeddedPostgresDataDir); + const ownedHome = await realpath(path.resolve(path.dirname(configuration), "../..")); + expect(dataDirectory.startsWith(`${ownedHome}${path.sep}`)).toBe(true); + const owner = (await readFile(path.join(dataDirectory, "postmaster.pid"), "utf8")).split(/\r?\n/); + expect(await realpath(owner[1])).toBe(dataDirectory); + const port = Number(owner[3]); + expect(Number.isSafeInteger(port) && port > 0 && port <= 65535).toBe(true); + const require = createRequire(path.join(input.appRoot, "packages/db/package.json")); + const postgres = require("postgres"); + const sql = postgres({ host: "127.0.0.1", port, database: "paperclip", username: "paperclip", password: "paperclip", + max: 1, connect_timeout: 10, connection: { application_name: "native_ingestion_remote_authority_proof" } }); + let proof; + try { + proof = await sql.begin("isolation level repeatable read read only", async (tx: any) => { + const settings = await tx`select current_setting('transaction_read_only') as read_only, + current_setting('transaction_isolation') as isolation`; + expect(settings[0].read_only).toBe("on"); + expect(settings[0].isolation).toBe("repeatable read"); + if (input.recordedFailureRunIds) { + const queryStartedAtMs = Date.now(); + const rows = await tx`select r.id as run_id, r.company_id, r.agent_id, r.runtime_mode, r.status, r.native_phase, + r.error_code as current_error_code, r.error is not null as current_has_error, + jsonb_typeof(r.result_json->'recoveredExecutionFailure') as recovered_type, + r.result_json->'recoveredExecutionFailure'->>'schema' as recovered_schema, + r.result_json->'recoveredExecutionFailure'->>'errorCode' as recovered_error_code, + r.result_json->'recoveredExecutionFailure'->>'observedAt' as recovered_observed_at, + f.attempt as finalization_attempt + from heartbeat_runs r left join native_run_finalizations f on f.run_id = r.id and f.company_id = r.company_id + where r.company_id = ${input.companyId} and r.id = any(${input.recordedFailureRunIds}::uuid[])`; + return { kind: "recorded-run-failure-codes", queryStartedAtMs, checkedAtMs: Date.now(), + transaction: "READ ONLY REPEATABLE READ", observedReadOnly: settings[0].read_only, + observedIsolation: settings[0].isolation, rows }; + } + const rows = await tx`select r.id as run_id, r.company_id, r.agent_id, r.native_issue_id as issue_id, r.runtime_mode, + r.native_session_id, r.runner_instance_id, r.status, r.process_pid, r.process_group_id, r.process_started_at, + r.runner_profile_json->'nativeExecutionInput'->'binding'->>'executionWorkspaceId' as admitted_execution_workspace_id, + r.result_json->'nativeCancellation'->>'dispatchState' as cancellation_dispatch_state, + r.result_json->'nativeCancellation' as native_cancellation, + i.seq as process_identity_seq, i.payload as process_identity, i.source_event_id as process_identity_source_event_id, + s.seq as process_stopped_seq, s.payload as process_stopped, s.source_event_id as process_stopped_source_event_id, + p.seq as process_lifecycle_seq, p.event_type as process_lifecycle_event_type, p.source_event_id as process_lifecycle_source_event_id, + l.id as lease_id, l.environment_id, l.provider, l.provider_lease_id, + l.status as lease_status, l.cleanup_status as lease_cleanup_status, l.released_at as lease_released_at, + c.part as checkpoint_part, octet_length(c.material::text) as checkpoint_header_bytes + from heartbeat_runs r + join environment_leases l on l.heartbeat_run_id = r.id and l.company_id = r.company_id + left join native_runner_checkpoints c on c.run_id = r.id and c.company_id = r.company_id + and c.issue_id = r.native_issue_id and c.part = 'header' + left join lateral (select seq, payload, source_event_id from heartbeat_run_events e where e.run_id = r.id + and e.company_id = r.company_id and e.event_type = 'native.process_identity_recorded' + and e.source_event_id is null order by seq desc limit 1) i on true + left join lateral (select seq, payload, source_event_id from heartbeat_run_events e where e.run_id = r.id + and e.company_id = r.company_id and e.event_type = 'native.local_process_stopped' + and e.source_event_id is null order by seq desc limit 1) s on true + left join lateral (select seq, event_type, source_event_id from heartbeat_run_events e where e.run_id = r.id + and e.company_id = r.company_id and e.event_type in ('native.process_start_requested', + 'native.process_identity_recorded', 'native.local_process_stopped') + and e.source_event_id is null order by seq desc limit 1) p on true + where r.company_id = ${input.companyId} and r.native_issue_id = any(${input.issueIds}::uuid[])`; + expect(rows).toHaveLength(input.issueIds.length); + expect(new Set(rows.map((row: any) => row.issue_id)).size).toBe(input.issueIds.length); + for (const row of rows) { + expect(row.company_id).toBe(input.companyId); + expect(row.runtime_mode).toBe("native"); + expect(row.environment_id).toBe(input.environmentId); + if (input.transport === "sandbox-remote") { + expect(row.provider).toBe("fake-plugin"); + expect(row.provider_lease_id).toMatch(/^fake-plugin:\/\//); + expect(row.checkpoint_part).toBe("header"); + expect(Number(row.checkpoint_header_bytes)).toBeGreaterThan(0); + } else { + expect(row.provider).toBe("local"); expect(row.provider_lease_id).toBeNull(); + expect(row.checkpoint_part, "Local raw transport keeps its existing host-file journal authority").toBeNull(); + expect(row.process_identity?.targetKind).toBe("local"); expect(row.process_identity?.hostname).toBe(hostname()); + expect(row.process_identity_source_event_id).toBeNull(); + expect(BigInt(String(row.process_identity_seq))).toBeGreaterThan(0n); + expect(row.admitted_execution_workspace_id).toMatch(/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i); + if (row.process_started_at) expect(row.process_identity.processStartedAt).toBe(new Date(row.process_started_at).toISOString()); + } + } + return { kind: input.transport === "local" ? "host-file-journal" : "database-controller-checkpoint", + checkedBeforeMeasuredEpoch: !input.afterStop, checkedAtMs: Date.now(), transaction: "READ ONLY REPEATABLE READ", + observedReadOnly: settings[0].read_only, observedIsolation: settings[0].isolation, rows }; + }); + } finally { await sql.end({ timeout: 5 }); } + const connectionClosedAtMs = Date.now(); + let localJournal; + if (input.transport === "local" && !input.afterStop && !input.recordedFailureRunIds) { + expect(input.issueIds).toHaveLength(1); + const base = await realpath(path.join(path.dirname(configuration), "runtime/paperclip-runner/durable-sessions")); + expect(base.startsWith(`${ownedHome}${path.sep}`)).toBe(true); + const directories = (await readdir(base, { withFileTypes: true })).filter(entry => entry.isDirectory() && /^[a-f0-9]{64}$/.test(entry.name)); + expect(directories.length, "Bound the owned fixture journal metadata search").toBeLessThanOrEqual(256); + for (const directory of directories) { + const file = path.join(base, directory.name, "control-plane/control-plane-state.json"); + const metadata = await lstat(file).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return null; throw error; + }); + if (!metadata) continue; + expect(metadata.isFile() && !metadata.isSymbolicLink() && metadata.size <= 16 * 1024 * 1024).toBe(true); + expect((await realpath(file)).startsWith(`${base}${path.sep}`)).toBe(true); + const bytes = await readFile(file), state = JSON.parse(bytes.toString("utf8")); + if (state.identity?.runId !== proof!.rows[0].run_id) continue; + expect(localJournal, "Exactly one owned local journal must match the actual run").toBeUndefined(); + expect(state.schema).toBe("paperclip.runner.durable.control-plane-state.v1"); + expect(state.checkpointEventPolicy).not.toBe("control-v1"); + expect(state.identity.runnerInstanceId).toBe(proof!.rows[0].runner_instance_id); + expect(state.identity.normalizedSessionId).toBe(proof!.rows[0].native_session_id); + expect(state.identity.environmentLeaseId).toBe(proof!.rows[0].admitted_execution_workspace_id); + localJournal = { file, sha256: createHash("sha256").update(bytes).digest("hex"), bytes: bytes.length, + schema: state.schema, identity: Object.fromEntries(["runnerInstanceId", "environmentLeaseId", "runId", "normalizedSessionId", "turnId", "itemId"] + .map(key => [key, state.identity[key]])), checkpointEventPolicy: state.checkpointEventPolicy, checkedAtMs: Date.now() }; + } + expect(localJournal, "Missing actual owned local raw journal").toBeTruthy(); + } + return { ...proof, connectionClosedAtMs, localJournal }; +} + +export const processAbsent = (pid: number) => { + expect(Number.isSafeInteger(pid) && Math.abs(pid) > 1).toBe(true); + try { process.kill(pid, 0); return false; } + catch (error) { if ((error as NodeJS.ErrnoException).code === "ESRCH") return true; throw error; } +}; diff --git a/tests/e2e/fixtures/native-ingestion-blocked-stop-clock.ts b/tests/e2e/fixtures/native-ingestion-blocked-stop-clock.ts new file mode 100644 index 00000000000..918ecd8a5fa --- /dev/null +++ b/tests/e2e/fixtures/native-ingestion-blocked-stop-clock.ts @@ -0,0 +1,127 @@ +// Gate and two-frame observer copied from frozen H5c e431ec63, narrowed to one owned participant. +import { createServer, type ServerResponse } from "node:http"; +import { expect, type Page } from "@playwright/test"; +export type SeenMarker = { markerCode: string; index: number; visibleMs: number }; +export type Emission = { nonce: string; markerCode: string; index: number; intendedMs: number; emittedMs: number; receivedAtMs: number }; +type ClockStream = { nonce: string; markerCode: string; opened: boolean; hello: boolean; ended: boolean; aborted: boolean; + partialBytes: number; records: Map; error?: string; expectedLauncher: "host-local" | "staged-remote"; + launcher?: string; providerPid?: number; providerReadyAtMs?: number; clockHelloAtMs?: number; readinessObservedAtMs?: number }; +type ClockCase = { events: number; intervalMs: number; epochMs?: number; streams: Map; errors: string[] }; +export async function openClock(page: Page) { + const pending = new Map>(); + const clocks = new Map(); + let markerCounter = 0; + const nextMarkerCode = () => { + expect(markerCounter).toBeLessThan(36 ** 2); + return (markerCounter++).toString(36).padStart(2, "0"); + }; + const gate = createServer((req, res) => { + const url = new URL(req.url!, "http://127.0.0.1"); + const route = url.pathname.match(/^\/native-ingestion-gate\/([a-f0-9]{32})(\/clock)?$/); + const key = route?.[1], participant = url.searchParams.get("participant"); + const state = key ? clocks.get(key) : undefined; + const stream = participant ? state?.streams.get(participant) : undefined; + if (!key || !participant || !state || !stream) { res.writeHead(404).end(); return; } + if (!route?.[2] && req.method === "GET") { + if (pending.get(key)!.has(participant)) { res.writeHead(409).end(); return; } + stream.providerReadyAtMs = Date.now(); + pending.get(key)!.set(participant, res); + res.on("close", () => pending.get(key)?.delete(participant)); + return; + } + if (!route?.[2] || req.method !== "POST" || stream.opened) { res.writeHead(409).end(); return; } + stream.opened = true; + let buffered = ""; + req.setEncoding("utf8"); + const fail = (message: string) => { + stream.error = message; state.errors.push(`${participant}: ${message}`); + if (!res.writableEnded) res.writeHead(400).end(); + }; + const line = (text: string) => { + const value = JSON.parse(text); + if (value.nonce !== participant || value.markerCode !== stream.markerCode) throw new Error("Clock identity mismatch"); + if (!stream.hello) { + if (value.type !== "hello" || value.events !== state.events || value.intervalMs !== state.intervalMs + || value.launcher !== stream.expectedLauncher || !Number.isSafeInteger(value.providerPid) || value.providerPid <= 1 + || Object.keys(value).sort().join(",") !== "events,intervalMs,launcher,markerCode,nonce,providerPid,type") throw new Error("Clock hello mismatch"); + stream.launcher = value.launcher; stream.providerPid = value.providerPid; + stream.hello = true; stream.clockHelloAtMs = Date.now(); return; + } + if (value.type !== "emission" || !Number.isSafeInteger(value.index) || value.index < 0 || value.index >= state.events + || !Number.isSafeInteger(value.intendedMs) || !Number.isSafeInteger(value.emittedMs) || value.emittedMs <= 0 + || value.intendedMs !== state.epochMs! + value.index * state.intervalMs) throw new Error("Invalid original emission clock"); + if (stream.records.has(value.index)) throw new Error("Duplicate or conflicting original emission clock"); + stream.records.set(value.index, { nonce: participant, markerCode: stream.markerCode, index: value.index, + intendedMs: value.intendedMs, emittedMs: value.emittedMs, receivedAtMs: Date.now() }); + }; + req.on("data", (chunk: string) => { + if (stream.error) return; + try { + buffered += chunk; + for (;;) { + const end = buffered.indexOf("\n"); if (end < 0) break; + const text = buffered.slice(0, end); buffered = buffered.slice(end + 1); + if (text.length > 1024) throw new Error("Clock record exceeds bounded metadata size"); + if (text) line(text); + } + if (buffered.length > 1024) throw new Error("Clock record exceeds bounded metadata size"); + } catch (error) { fail(String(error)); } + }); + req.on("end", () => { + stream.ended = true; stream.partialBytes = Buffer.byteLength(buffered); + if (buffered.length) fail("Clock stream ended with a partial record"); + if (!stream.error && !res.writableEnded) res.writeHead(200).end(); + }); + req.on("aborted", () => { stream.aborted = true; stream.partialBytes = Buffer.byteLength(buffered); }); + req.on("error", error => { if (!stream.aborted) fail(String(error)); }); + }); + await new Promise(resolve => gate.listen(0, "127.0.0.1", resolve)); + const address = gate.address(); + if (!address || typeof address === "string") throw new Error("Local ingestion gate failed"); + const base = `http://127.0.0.1:${address.port}/native-ingestion-gate/`; + await page.addInitScript(() => { + const seen = new Set(), visible: SeenMarker[] = []; + Object.assign(window, { nativeIngestionVisible: visible }); + let scheduled = false, dirty = false; + const snapshot = () => { + const text = document.body?.innerText ?? ""; + const pattern = /\[([a-z0-9]{2})([a-z0-9]{1,3})\]/g; + const entries: Omit[] = []; + for (const match of text.matchAll(pattern)) { + const index = Number.parseInt(match[2], 36); + const key = `${match[1]}:${index}`; + if (seen.has(key)) continue; + entries.push({ markerCode: match[1], index }); + } + return entries; + }; + const schedule = () => { + dirty = true; + if (scheduled) return; + scheduled = true; + requestAnimationFrame(() => { + dirty = false; + const entries = snapshot(); + requestAnimationFrame(() => { + // Only stamp the previous frame's snapshot. A token inserted + // between frames has not had this snapshot's paint opportunity. + const visibleMs = Date.now(); + for (const entry of entries) { + const key = `${entry.markerCode}:${entry.index}`; + if (seen.has(key)) continue; + seen.add(key); visible.push({ ...entry, visibleMs }); + } + scheduled = false; + if (dirty) schedule(); + }); + }); + }; + const install = () => { + new MutationObserver(schedule).observe(document.body, { subtree: true, childList: true, characterData: true }); + schedule(); + }; + if (document.body) install(); else addEventListener("DOMContentLoaded", install, { once: true }); + }); + + return { pending, clocks, nextMarkerCode, base, gate }; +} diff --git a/tests/e2e/fixtures/native-ingestion-blocked-stop-db.ts b/tests/e2e/fixtures/native-ingestion-blocked-stop-db.ts new file mode 100644 index 00000000000..acb6a275c3b --- /dev/null +++ b/tests/e2e/fixtures/native-ingestion-blocked-stop-db.ts @@ -0,0 +1,43 @@ +import { createRequire } from "node:module"; +import { readFile, realpath } from "node:fs/promises"; +import path from "node:path"; +import { expect } from "@playwright/test"; + +export async function ownedDatabase(appRoot: string, name: string) { + const configuration = process.env.PAPERCLIP_E2E_SERVER_CONFIG; + if (!configuration) throw new Error("Missing disposable application configuration"); + const config = JSON.parse(await readFile(configuration, "utf8")); + expect(config.database.mode).toBe("embedded-postgres"); + const home = await realpath(path.resolve(path.dirname(configuration), "../..")); + const directory = await realpath(config.database.embeddedPostgresDataDir); + expect(directory.startsWith(`${home}${path.sep}`)).toBe(true); + const owner = (await readFile(path.join(directory, "postmaster.pid"), "utf8")).split(/\r?\n/); + expect(await realpath(owner[1])).toBe(directory); + const port = Number(owner[3]); + expect(Number.isSafeInteger(port) && port > 0 && port <= 65535).toBe(true); + const postgres = createRequire(path.join(appRoot, "packages/db/package.json"))("postgres"); + return postgres({ host: "127.0.0.1", port, database: "paperclip", username: "paperclip", password: "paperclip", + max: 1, connect_timeout: 10, connection: { application_name: name } }); +} +export async function cancellationSnapshot(sql: any, companyId: string, runId: string) { + const startedAtMs = Date.now(); + const rows = await sql`select id as run_id, company_id, native_issue_id as issue_id, status, + result_json->'nativeCancellation' as intent, + result_json->'executionCancellation' as shared_execution_cancellation, + result_json->'cancellation' as run_cancellation, + (select jsonb_build_object('id', d.id, 'companyId', d.company_id, 'runId', d.run_id, 'issueId', d.issue_id, + 'reasonCode', d.reason_code) from status_decisions d where d.company_id = r.company_id and d.run_id = r.id + and d.issue_id = r.native_issue_id and d.id::text = r.result_json->'nativeCancellation'->>'decisionId') as shared_decision, + (select coalesce(jsonb_agg(jsonb_build_object('id', a.id, 'action', a.action, 'details', a.details)), '[]'::jsonb) + from activity_log a where a.company_id = r.company_id and a.run_id = r.id + and a.action in ('native.cancellation_intent_recorded', 'native.cancellation_dispatch_acknowledged')) as audits + from heartbeat_runs r where r.company_id = ${companyId} and r.id = ${runId}`; + expect(rows).toHaveLength(1); + return { ...rows[0], queryStartedAtMs: startedAtMs, observedAtMs: Date.now() }; +} +export async function blockers(sql: any, holderPid: number) { + return sql`select pid, application_name, state, wait_event_type, wait_event, query, query_start, + pg_blocking_pids(pid) as blocking_pids from pg_stat_activity + where datname = current_database() and pid <> pg_backend_pid() + and ${holderPid} = any(pg_blocking_pids(pid))`; +} diff --git a/tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.mjs b/tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.mjs new file mode 100644 index 00000000000..14e6c6e3eb9 --- /dev/null +++ b/tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.mjs @@ -0,0 +1,394 @@ +import assert from 'node:assert/strict'; +import { createHash } from 'node:crypto'; +export const PROTOCOL = 'blocked-stop-history-v1'; +export const WORKLOAD = Object.freeze({ prelude: 1000, events: 256, intervalMs: 100, visibleBeforeLock: 3, + heldAfterActualRequestMs: 1000, pollIntervalMs: 50, readinessMs: 300000, streamMs: 120000, + terminalMs: 180000, acknowledgementAfterSettlementMs: 30000, leaseAfterSettlementMs: 30000, invocationMs: 900000 }); +export const WORKLOAD_SHA256 = createHash('sha256').update(JSON.stringify(WORKLOAD)).digest('hex'); +export const METRICS = ['actionToAckMs', 'releaseAckToAbsenceMs', 'releaseAckToAckMs']; +export function isAppendLockQuery(query) { + // Shared B0/B1/B2 projection prefix, unlike cancellation's full-row SELECT. + const sql = String(query).replace(/\s+/g, ' ').trim(); + return /^select "company_id", "agent_id"(?:,| from)/i.test(sql) + && / from "heartbeat_runs" where "heartbeat_runs"\."id" = \$\d+.* for update/i.test(sql); +} +export function assertHeldSample(sample, holderPid, expectedProcesses) { + assert.equal(sample.httpPending, true, 'Stop HTTP must remain pending while its durable barrier is locked'); + assert.equal(sample.intent, null, 'No committed intent before releasing the run lock'); + assert.deepEqual(sample.auditIds, [], 'No committed cancellation audit before releasing the run lock'); + assert.equal(sample.processesAlive, true, 'No physical provider Stop before the durable intent barrier'); + assert(expectedProcesses?.runner && expectedProcesses.provider, 'Missing original owned process births'); + assert.deepEqual(sample.processes?.runner, expectedProcesses.runner); + assert.deepEqual(sample.processes?.provider, expectedProcesses.provider); + assert.equal(sample.processes?.group, expectedProcesses.group); + assert.equal(sample.processes?.providerGroup, expectedProcesses.providerGroup); + assert.equal(sample.processes?.runnerGroupAlive, true); + assert.equal(sample.processes?.providerGroupAlive, true); + assert.equal(sample.clockOpen, true, 'The unchanged provider fixture closes its clock on turn/interrupt'); + assert(sample.blockers.some(row => row.blocking_pids.includes(holderPid)), 'Missing independent PostgreSQL blocker'); +} +// Public cancellation supports active/scheduled runs and native retryable failures. +// A receipt is not physical Stop proof; the independent process/lease oracle remains required. +export function cleanupCancellationEligible(run) { + return ['queued', 'running', 'scheduled_retry'].includes(run.status) + || (run.status === 'failed' && run.runtimeMode === 'native' && run.nativePhase === 'retryable_failure'); +} +export async function cleanupOwnedRunBeforeClockClose({ companyId, runId, readRun, cancelRun, closeClock, record }) { + const snapshot = run => { + assert.equal(run.id, runId); assert.equal(run.companyId, companyId); + return { id: run.id, companyId: run.companyId, status: run.status, runtimeMode: run.runtimeMode, + nativePhase: run.nativePhase ?? null, errorCode: run.errorCode ?? null, + nativeCancellation: run.resultJson?.nativeCancellation ?? null }; + }; + try { + if (readRun) { + const before = await readRun(); record.before = snapshot(before); + record.cancel = { attempted: cleanupCancellationEligible(before) }; + if (record.cancel.attempted) { + const response = await cancelRun(); + record.cancel.httpStatus = response.status; record.cancel.response = snapshot(response.run); + } + record.after = snapshot(await readRun()); + } + } catch (error) { record.error = String(error); throw error; } + finally { + record.clockCloseStartedAtMs = Date.now(); + try { await closeClock(); record.clockClosedAtMs = Date.now(); } + catch (error) { record.clockCloseError = String(error); throw error; } + } +} +export function assertIntentAudit(row) { + const intent = row.intent; + assert.equal(intent?.schema, 'paperclip.native-cancellation.v1'); + assert.equal(intent.companyId, row.company_id); assert.equal(intent.runId, row.run_id); + assert.equal(intent.issueId, row.issue_id); + assert.equal(intent.scope, 'run'); assert.equal(intent.reasonCode, 'cancellation_run_only'); + if (intent.decisionId != null) { + assert.equal(row.shared_decision?.id, intent.decisionId); + assert.equal(row.shared_decision.companyId, row.company_id); assert.equal(row.shared_decision.runId, row.run_id); + assert.equal(row.shared_decision.issueId, row.issue_id); + } + assert(row.audits.some(a => a.id === intent.intentAuditId && a.action === 'native.cancellation_intent_recorded' + && a.details?.intentId === intent.intentId), 'Intent must match its committed own audit'); + if (intent.dispatchState === 'acknowledged') { + assert.equal(intent.dispatched, true); + assert(row.audits.some(a => a.id === intent.acknowledgementAuditId + && a.action === 'native.cancellation_dispatch_acknowledged' && a.details?.intentId === intent.intentId + && a.details.intentAuditId === intent.intentAuditId && a.details.dispatched === true + && (a.details.decisionId ?? null) === (intent.decisionId ?? null)), 'ACK must match its own committed audit and decision'); + } +} +export function metrics(evidence) { + assert.equal(evidence.protocol, PROTOCOL); assert.equal(evidence.workloadSha256, WORKLOAD_SHA256); + assert.equal(evidence.complete, true); assert.equal(evidence.cleanup?.complete, true); + const t = evidence.timing; + for (const k of ['actionAtMs', 'requestAtMs', 'releaseRequestedAtMs', 'releaseCommitAckAtMs', 'absenceObservedAtMs', 'ackObservedAtMs']) + assert(Number.isSafeInteger(t[k]) && t[k] > 0, `Missing ${k}`); + assert(t.actionAtMs <= t.requestAtMs); assert(t.requestAtMs + WORKLOAD.heldAfterActualRequestMs <= t.releaseRequestedAtMs); + assert(t.releaseRequestedAtMs <= t.releaseCommitAckAtMs); assert(t.releaseCommitAckAtMs <= t.absenceObservedAtMs); + assert(t.releaseCommitAckAtMs <= t.ackObservedAtMs); + return { actionToAckMs: t.ackObservedAtMs - t.actionAtMs, + releaseAckToAbsenceMs: t.absenceObservedAtMs - t.releaseCommitAckAtMs, + releaseAckToAckMs: t.ackObservedAtMs - t.releaseCommitAckAtMs, + releaseRequestToAbsenceMs: t.absenceObservedAtMs - t.releaseRequestedAtMs, + releaseRequestToAckMs: t.ackObservedAtMs - t.releaseRequestedAtMs }; +} +export function registration() { + const stages = [['b0-aa','b0','b0'], ['b1-aa','b1','b1'], ['b0-b1','b0','b1'], ['b1-b2','b1','b2'], ['b0-b2','b0','b2']]; + return stages.flatMap(([stage,a,b]) => Array.from({ length: 5 }, (_, pair) => (pair % 2 ? [b,a] : [a,b]) + .map((variant, index) => ({ stage, pair: pair + 1, index, variant, role: pair % 2 ? (index ? 'baseline' : 'candidate') : (index ? 'candidate' : 'baseline') })))).flat(); +} +const median = values => { + assert(values.length > 0); + const sorted = [...values].sort((a,b) => a-b), middle = Math.floor(sorted.length / 2); + return sorted.length % 2 ? sorted[middle] : (sorted[middle - 1] + sorted[middle]) / 2; +}; +const spread = values => ({ minimumMs: Math.min(...values), maximumMs: Math.max(...values), + spreadMs: Math.max(...values) - Math.min(...values) }); +const hashReport = evidence => createHash('sha256').update(JSON.stringify(evidence)).digest('hex'); +export const PRECISION_RULE = Object.freeze({ schema: 'blocked-stop-precision-v1', + pollIntervalMs: WORKLOAD.pollIntervalMs, floorMs: 2 * WORKLOAD.pollIntervalMs, pooledMedianFraction: 0.10, + pairedRule: 'maximum absolute difference <= budget', driftRule: 'absolute first-five vs last-five chronological median drift <= budget', + bracketRule: 'each observer bracket <= corresponding metric budget', unsuitableRule: 'retain all ten arms; no noise freeze or replacement', + scope: 'Measurement repeatability; not a product Stop deadline' }); +export const PRECISION_RULE_SHA256 = createHash('sha256').update(JSON.stringify(PRECISION_RULE)).digest('hex'); +export function observerBrackets(evidence) { + const t = evidence.timing, origin = t.releaseCommitAckAtMs; + const samples = evidence.observerSamples; + assert(Array.isArray(samples), 'Missing raw observer query/check samples'); + const result = {}; + for (const [kind, timestamp, negativeTimestamp] of [ + ['ack-query', 'ackObservedAtMs', 'lastAckNotObservedAtMs'], + ['process-check', 'absenceObservedAtMs', 'lastProcessesPresentAtMs']]) { + const rows = samples.filter(row => row.kind === kind); + assert(rows.length > 0, `Missing ${kind} observer samples`); + let priorEnd = origin; + for (const [index,row] of rows.entries()) { + assert(Number.isSafeInteger(row.startedAtMs) && Number.isSafeInteger(row.completedAtMs)); + assert(row.startedAtMs >= priorEnd && row.completedAtMs >= row.startedAtMs); + assert.equal(row.durationMs, row.completedAtMs - row.startedAtMs); + assert.equal(row.value, index === rows.length - 1, 'Retain every negative sample through the first successful observation'); + priorEnd = row.completedAtMs; + } + const success = rows.at(-1), negative = rows.at(-2); + assert.equal(t[timestamp], success.completedAtMs); + if (negative) assert.equal(t[negativeTimestamp], negative.completedAtMs); + else assert(t[negativeTimestamp] === undefined, 'A negative timestamp needs its raw query/check sample'); + // A negative query can read its snapshot at any point during execution; + // use its start as the conservative lower boundary and retain both ends. + const lowerAtMs = negative?.startedAtMs ?? origin; + result[kind] = { lowerAtMs, upperAtMs: success.completedAtMs, + widthMs: success.completedAtMs - lowerAtMs, lowerKind: negative ? 'last-negative-query-or-check-start' : 'release-COMMIT-ACK', + successfulStartedAtMs: success.startedAtMs, successfulDurationMs: success.durationMs, + ...(negative ? { negativeStartedAtMs: negative.startedAtMs, negativeCompletedAtMs: negative.completedAtMs, + negativeDurationMs: negative.durationMs } : {}), sampleCount: rows.length }; + } + return { actionToAckMs: result['ack-query'], releaseAckToAckMs: result['ack-query'], + releaseAckToAbsenceMs: result['process-check'] }; +} +function validatedPairs(pairs) { + assert.equal(pairs.length, 5, 'Require five fresh pairs'); + const validated = pairs.map(pair => { + assert.equal(pair.length, 2); + return pair.map(evidence => ({ evidence, values: assertFunctional(evidence), brackets: observerBrackets(evidence) })); + }); + const chronological = validated.flat().sort((a,b) => a.evidence.startedAtMs - b.evidence.startedAtMs); + assert.equal(new Set(chronological.map(arm => arm.evidence.runId)).size, 10, 'Ten fresh owned runs are required'); + assert.equal(new Set(chronological.map(arm => arm.evidence.companyId)).size, 10, 'Ten fresh disposable invocations are required'); + for (const [index,arm] of chronological.entries()) { + assert(Number.isSafeInteger(arm.evidence.startedAtMs) && Number.isSafeInteger(arm.evidence.finishedAtMs)); + assert(arm.evidence.finishedAtMs >= arm.evidence.startedAtMs); + if (index) assert(chronological[index-1].evidence.finishedAtMs < arm.evidence.startedAtMs, + 'Arms must not overlap'); + } + validated.forEach((pair,index) => { + assert.equal(pair[0].evidence.startedAtMs, chronological[2*index + index%2].evidence.startedAtMs, + 'Pairs must alternate baseline-first then candidate-first in chronological order'); + assert.equal(pair[1].evidence.startedAtMs, chronological[2*index + (index%2 ? 0 : 1)].evidence.startedAtMs); + }); + return validated; +} +export function noiseEnvelope(pairs) { + const validated = validatedPairs(pairs), chronological = validated.flat().sort((a,b) => a.evidence.startedAtMs - b.evidence.startedAtMs); + for (const [index,arm] of chronological.entries()) { + assert(Number.isSafeInteger(arm.evidence.startedAtMs) && Number.isSafeInteger(arm.evidence.finishedAtMs)); + assert(arm.evidence.finishedAtMs >= arm.evidence.startedAtMs); + if (index) assert(chronological[index-1].evidence.finishedAtMs < arm.evidence.startedAtMs, + 'A/A requires ten distinct non-overlapping chronological arms'); + } + const variants = new Set(chronological.map(arm => String(arm.evidence.variant).toLowerCase())); + assert.equal(variants.size, 1, 'A/A calibration must use one baseline variant'); + const baselineVariant = [...variants][0]; assert(['b0','b1'].includes(baselineVariant)); + const result = { schema: 'paperclip.native-ingestion.blocked-stop-noise.v1', protocol: PROTOCOL, baselineVariant, + workloadSha256: WORKLOAD_SHA256, precisionRule: PRECISION_RULE, precisionRuleSha256: PRECISION_RULE_SHA256, + sampleCounts: { pairs: 5, arms: 10 }, suitable: false, noiseFrozen: false, metrics: {}, + arms: chronological.map(arm => ({ reportSha256: hashReport(arm.evidence), startedAtMs: arm.evidence.startedAtMs, + finishedAtMs: arm.evidence.finishedAtMs, values: arm.values, observerBrackets: arm.brackets })), + rawArms: chronological.map(arm => arm.evidence), pairBindings: validated.map(pair => pair.map(arm => hashReport(arm.evidence))), + sourceDigestScope: 'SHA256 of JSON.stringify(parsed raw arm); owning raw-byte seals are separately required' }; + for (const metric of METRICS) { + const values = chronological.map(arm => arm.values[metric]), pooledMedianMs = median(values); + const baselineValues = validated.map(pair => pair[0].values[metric]), candidateValues = validated.map(pair => pair[1].values[metric]); + const baselineMedianMs = median(baselineValues), candidateMedianMs = median(candidateValues); + const differencesMs = validated.map(pair => pair[1].values[metric] - pair[0].values[metric]); + const maximumAbsolutePairedDifferenceMs = Math.max(...differencesMs.map(Math.abs)); + const firstFiveMedianMs = median(values.slice(0,5)), lastFiveMedianMs = median(values.slice(5)); + const absoluteChronologicalMedianDriftMs = Math.abs(lastFiveMedianMs - firstFiveMedianMs); + const precisionBudgetMs = Math.max(PRECISION_RULE.floorMs, PRECISION_RULE.pooledMedianFraction * pooledMedianMs); + const observerBracketWidthsMs = chronological.map(arm => arm.brackets[metric].widthMs); + const floorDominated = PRECISION_RULE.pooledMedianFraction * pooledMedianMs <= PRECISION_RULE.floorMs; + result.metrics[metric] = { sampleCount: 10, pairCount: 5, pooledMedianMs, baselineMedianMs, candidateMedianMs, + absoluteMedianDifferenceMs: Math.abs(candidateMedianMs - baselineMedianMs), + medianDifferencePercent: floorDominated ? null : 100 * (candidateMedianMs - baselineMedianMs) / baselineMedianMs, + ...spread(values), pairedDifferencesMs: differencesMs, maximumAbsolutePairedDifferenceMs, + firstFiveMedianMs, lastFiveMedianMs, absoluteChronologicalMedianDriftMs, precisionBudgetMs, + observerBracketWidthsMs, maximumObserverBracketMs: Math.max(...observerBracketWidthsMs), + suitable: maximumAbsolutePairedDifferenceMs <= precisionBudgetMs && absoluteChronologicalMedianDriftMs <= precisionBudgetMs + && observerBracketWidthsMs.every(width => width <= precisionBudgetMs), + noiseMs: null, floorDominated, resolutionLabel: floorDominated ? '100 ms resolution floor' : 'Observer brackets and budget reported', + subfloorOrP99ClaimsAllowed: false, percentageSpeedClaimsAllowed: !floorDominated }; + } + result.suitable = METRICS.every(metric => result.metrics[metric].suitable); + result.noiseFrozen = result.suitable; + if (result.suitable) for (const metric of METRICS) result.metrics[metric].noiseMs = result.metrics[metric].maximumAbsolutePairedDifferenceMs; + return result; +} +export function assertNoiseEnvelope(noise) { + assert.equal(noise?.schema, 'paperclip.native-ingestion.blocked-stop-noise.v1'); + assert.equal(noise.protocol, PROTOCOL); assert.equal(noise.workloadSha256, WORKLOAD_SHA256); + assert.equal(noise.precisionRuleSha256, PRECISION_RULE_SHA256); assert.deepEqual(noise.precisionRule, PRECISION_RULE); + assert.deepEqual(noise.sampleCounts, { pairs: 5, arms: 10 }); + assert.equal(noise.suitable, true, 'Unsuitable A/A cannot freeze a noise envelope'); assert.equal(noise.noiseFrozen, true); + assert.equal(noise.rawArms?.length, 10); + // Recompute suitability and all numeric limits from retained raw arms. A + // caller cannot inflate a budget or flip a failed suitability flag. + const byDigest = new Map(noise.rawArms.map(arm => [hashReport(arm), arm])); + assert.equal(noise.pairBindings?.length, 5); + const logical = noise.pairBindings.map(pair => { + assert.equal(pair.length, 2); + return pair.map(digest => { const raw = byDigest.get(digest); assert(raw, 'Frozen arm digest mismatch'); return raw; }); + }); + const recomputed = noiseEnvelope(logical); + assert.equal(recomputed.suitable, true); + assert.deepEqual(noise.metrics, recomputed.metrics, 'Frozen noise or precision limit changed'); + assert.equal(noise.baselineVariant, recomputed.baselineVariant); + assert.deepEqual(noise.arms, recomputed.arms, 'Frozen chronological arms changed'); + return noise; +} +export function comparePairs(pairs, noise) { + const validated = validatedPairs(pairs); assertNoiseEnvelope(noise); + for (const [baseline] of validated) assert.equal(String(baseline.evidence.variant).toLowerCase(), noise.baselineVariant, + 'Comparisons must use their own baseline calibration'); + return Object.fromEntries(METRICS.map(metric => { + const calibrated = noise.metrics[metric], deltas = validated.map(([baseline,candidate]) => candidate.values[metric] - baseline.values[metric]); + const baselineValues = validated.map(pair => pair[0].values[metric]), candidateValues = validated.map(pair => pair[1].values[metric]); + const baselineMedianMs = median(baselineValues), candidateMedianMs = median(candidateValues); + const observerBracketWidthsMs = validated.map(pair => pair.map(arm => arm.brackets[metric].widthMs)); + const floorDominated = calibrated.floorDominated || PRECISION_RULE.pooledMedianFraction * Math.min(baselineMedianMs,candidateMedianMs) <= PRECISION_RULE.floorMs; + return [metric, { sampleCounts: { pairs: 5, arms: 10 }, baselineMedianMs, candidateMedianMs, deltaMedianMs: median(deltas), + medianDifferencePercent: floorDominated ? null : 100 * (candidateMedianMs - baselineMedianMs) / baselineMedianMs, + deltasMs: deltas, noiseMs: calibrated.noiseMs, precisionBudgetMs: calibrated.precisionBudgetMs, + baselineSpread: spread(baselineValues), candidateSpread: spread(candidateValues), ...spread(deltas), + observerBracketWidthsMs, floorDominated, resolutionLabel: floorDominated ? '100 ms resolution floor' : 'Observer brackets and budget reported', + subfloorOrP99ClaimsAllowed: false, percentageSpeedClaimsAllowed: !floorDominated, + pass: deltas.every(delta => delta <= calibrated.noiseMs) && median(deltas) <= calibrated.noiseMs + && observerBracketWidthsMs.flat().every(width => width <= calibrated.precisionBudgetMs) }]; + })); +} +function assertOptionalShared(row, companyId, runId) { + assert.equal(row.company_id, companyId); assert.equal(row.run_id, runId); + const timestamp = value => typeof value === 'string' && /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z$/.test(value) && Number.isFinite(Date.parse(value)); + const execution = row.shared_execution_cancellation; + if (execution != null) { + assert(execution && typeof execution === 'object' && !Array.isArray(execution)); + assert(['requested','acknowledged'].includes(execution.state)); + if (execution.state === 'requested') assert(timestamp(execution.requestedAt)); + if (execution.state === 'acknowledged') assert(timestamp(execution.acknowledgedAt)); + if (execution.requestedAt != null) assert(timestamp(execution.requestedAt)); + if (execution.acknowledgedAt != null) assert(timestamp(execution.acknowledgedAt)); + if (execution.requestedAt != null && execution.acknowledgedAt != null) + assert(Date.parse(execution.requestedAt) <= Date.parse(execution.acknowledgedAt)); + } + const cancellation = row.run_cancellation; + if (cancellation != null) { + assert.equal(cancellation.source, 'operator'); assert.equal(cancellation.expected, true); + assert.equal(cancellation.initiator?.type, 'user'); + assert(typeof cancellation.reason === 'string' && cancellation.reason.length > 0); + assert(timestamp(cancellation.recordedAt)); + } +} +// Mirrors acknowledgedNativeStopExecutionHasStopped's two local identity states; +// independent PID/group absence is mandatory in either state, never inferred from ACK. +export function assertAuthoritativeRetirement(e) { + const before = e.authorityBefore?.rows?.[0], after = e.authorityAfter?.rows?.[0]; + for (const row of [before, after]) { + assert.equal(row?.company_id, e.companyId); assert.equal(row?.run_id, e.runId); + assert.equal(row.issue_id, e.issueId); assert.equal(row.runtime_mode, 'native'); + } + assert.equal(after.status, 'cancelled'); assert.equal(after.cancellation_dispatch_state, 'acknowledged'); + assertIntentAudit(e.ack); assert.equal(e.ack.intent.dispatchState, 'acknowledged'); + assert.equal(e.ack.company_id, e.companyId); assert.equal(e.ack.run_id, e.runId); assert.equal(e.ack.issue_id, e.issueId); + assert.deepEqual(after.native_cancellation, e.ack.intent, 'Snapshot must contain the same audited own ACK'); + assert.deepEqual([e.processesAfter?.runnerAbsent, e.processesAfter?.groupAbsent, + e.processesAfter?.providerAbsent, e.processesAfter?.providerGroupAbsent], [true,true,true,true]); + assert.equal(e.processesAfter.group, e.processesBefore.group); + assert.equal(e.processesAfter.providerGroup, e.processesBefore.providerGroup); + for (const process of [e.processesBefore.runner, e.processesBefore.provider]) { + assert(Number.isSafeInteger(process.pid) && process.pid > 1 && process.birth); + assert(Number.isSafeInteger(process.group) && process.group > 1); + } + assert.equal(e.processesBefore.runner.group, e.processesBefore.group); + assert.equal(e.processesBefore.group, before.process_group_id); + assert.equal(e.processesBefore.runner.pid, before.process_pid); + assert.equal(e.processesBefore.provider.group, e.processesBefore.providerGroup); + const birth = value => { + if (value === null) return null; + assert(value instanceof Date || typeof value === 'string'); + assert(Number.isFinite(new Date(value).getTime())); return new Date(value).toISOString(); + }; + const sequence = value => { assert(/^[1-9]\d*$/.test(String(value))); return BigInt(value); }; + const originalBirth = birth(before.process_started_at), identitySeq = sequence(before.process_identity_seq); + assert(originalBirth !== null, 'Missing captured authoritative original process birth'); + assert.equal(before.process_identity_source_event_id, null); + assert.equal(before.process_identity?.targetKind, 'local'); assert(before.process_identity.hostname); + assert.equal(before.process_identity.processPid, before.process_pid); + assert.equal(before.process_identity.processGroupId, before.process_group_id); + assert.equal(before.process_identity.processStartedAt, originalBirth); + assert.equal(after.process_identity_source_event_id, null); + assert.equal(sequence(after.process_identity_seq), identitySeq, 'A later identity is a different launch'); + assert.deepEqual(after.process_identity, before.process_identity); + assert.equal(e.lease?.heartbeatRunId, e.runId); assert.equal(e.lease.companyId, e.companyId); + assert.equal(e.lease.id, before.lease_id); assert(before.lease_id); + assert.equal(after.lease_id, before.lease_id); assert.equal(after.environment_id, before.environment_id); + assert.equal(e.lease.provider, 'local'); assert.equal(e.lease.providerLeaseId, null); + assert.equal(after.provider, 'local'); assert.equal(after.provider_lease_id, null); + for (const lease of [e.lease, { releasedAt: after.lease_released_at, status: after.lease_status, cleanupStatus: after.lease_cleanup_status }]) { + assert(lease.releasedAt && ['released','expired','failed'].includes(lease.status)); assert.notEqual(lease.cleanupStatus, 'failed'); + } + const cleared = after.process_pid === null && after.process_group_id === null; + if (cleared) assert.equal(after.process_started_at, null); + else { + assert.equal(after.process_pid, before.process_pid); assert.equal(after.process_group_id, before.process_group_id); + assert.equal(birth(after.process_started_at), originalBirth); + } + const noReceipt = [after.process_stopped_seq, after.process_stopped, after.process_stopped_source_event_id].every(value => value === null); + if (noReceipt) assert.equal(cleared, false, 'Cleared identity requires authoritative retirement evidence'); + else { + assert(sequence(after.process_stopped_seq) > sequence(after.process_identity_seq)); + assert.equal(after.process_stopped_source_event_id, null); + assert.deepEqual(after.process_stopped, { processPid: before.process_pid, processGroupId: before.process_group_id, + processStartedAt: originalBirth }); + } + assert.equal(after.process_lifecycle_source_event_id, null); + assert.equal(after.process_lifecycle_event_type, noReceipt ? 'native.process_identity_recorded' : 'native.local_process_stopped'); + assert.equal(sequence(after.process_lifecycle_seq), noReceipt ? identitySeq : sequence(after.process_stopped_seq), + 'A later process start or identity invalidates prior retirement evidence'); + return cleared ? 'cleared-identity-with-authoritative-retirement' : 'retained-identity-after-physical-exit'; +} +export function assertFunctional(e) { + metrics(e); observerBrackets(e); + assert.equal(e.prelude?.individualDeltas, WORKLOAD.prelude); assert.equal(e.prelude?.uniqueIdentities, WORKLOAD.prelude); + assert(e.prelude.verifiedBeforeEpochAtMs < e.epochMs); + assert(e.visibleBeforeLock?.length >= WORKLOAD.visibleBeforeLock); + assert.equal(e.lock?.id, e.runId); assert.equal(e.lock.company_id, e.companyId); + assert(e.appendBlockedBeforeStop?.observedAtMs < e.timing.requestAtMs); + assert(e.appendBlockedBeforeStop.rows.some(row => isAppendLockQuery(row.query) && row.blocking_pids.includes(e.lock.holder_pid))); + assert(e.heldSamples?.length >= 2); + for (const sample of e.heldSamples) { + assertHeldSample(sample, e.lock.holder_pid, e.processesBefore); assertOptionalShared(sample, e.companyId, e.runId); + assert(sample.queryStartedAtMs >= e.timing.requestAtMs && sample.observedAtMs <= e.timing.releaseRequestedAtMs); + } + assert(e.heldSamples.at(-1).emittedCount > e.heldSamples[0].emittedCount, 'Provider must still emit through the stall'); + assertIntentAudit(e.ack); assertOptionalShared(e.ack, e.companyId, e.runId); + assert.equal(e.ack.intent.dispatchState, 'acknowledged'); + if (e.sharedSettlement != null) { + assert.equal(e.sharedSettlement.runId, e.runId); assert.equal(e.sharedSettlement.companyId, e.companyId); + assert.equal(e.sharedSettlement.status, 'cancelled'); + assertOptionalShared({ company_id: e.sharedSettlement.companyId, run_id: e.sharedSettlement.runId, + shared_execution_cancellation: e.sharedSettlement.executionCancellation, run_cancellation: e.sharedSettlement.cancellation }, e.companyId, e.runId); + if (e.sharedSettlement.nativeCancellation != null) assert.equal(e.sharedSettlement.nativeCancellation.intentId, e.ack.intent.intentId); + } + assert.equal(e.ack.company_id, e.companyId); assert.equal(e.ack.run_id, e.runId); assert.equal(e.ack.issue_id, e.issueId); + assert.equal(e.stopHttpStatus, 200); + assert(e.timing.httpSettledAtMs >= e.timing.releaseRequestedAtMs); + assert.equal(e.authoritativeRetirementState, assertAuthoritativeRetirement(e)); + const transcript = e.transcript; + assert.equal(transcript?.persistedAfterReload, true); assert.equal(transcript?.ordered, true); + assert.equal(transcript.resultAcceptedEvents, 0); assert.equal(transcript.preludeCount, WORKLOAD.prelude); + assert.equal(transcript.individualDeltas, transcript.deltas.length); + assert.equal(transcript.uniqueSourceIdentities, transcript.deltas.length); + const identities = new Set(); + for (const [index,event] of transcript.deltas.entries()) { + assert.equal(event.runId, e.runId); assert.equal(event.eventType, 'item.delta'); + assert(event.sourceInstanceId && event.sourceEventId && Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0); + identities.add(`${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`); + assert.equal(event.payload.text, `[${e.clock.markerCode}${index.toString(36)}] `); + assert(e.clock.records.some(record => record.index === index && record.nonce === e.clock.nonce)); + } + assert.equal(identities.size, transcript.deltas.length); + for (const marker of transcript.seen) assert(marker.index >= 0 && marker.index < transcript.deltas.length); + assert(e.clock.ended || e.clock.aborted); assert(!e.clock.error); + assert.equal(e.measurement?.actionToAckIncludesForcedWait, true); + assert.equal(e.measurement?.recording?.trace, 'off'); assert.equal(e.measurement?.recording?.video, 'off'); + return metrics(e); +} diff --git a/tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.test.mjs b/tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.test.mjs new file mode 100644 index 00000000000..a224df5edf4 --- /dev/null +++ b/tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.test.mjs @@ -0,0 +1,333 @@ +import test from 'node:test'; +import assert from 'node:assert/strict'; +import { PROTOCOL, WORKLOAD as W, WORKLOAD_SHA256, METRICS, isAppendLockQuery, assertHeldSample, + assertIntentAudit, metrics, registration, noiseEnvelope, comparePairs, assertFunctional, assertAuthoritativeRetirement, observerBrackets, assertNoiseEnvelope, PRECISION_RULE, cleanupCancellationEligible, cleanupOwnedRunBeforeClockClose } from './native-ingestion-blocked-stop-protocol.mjs'; +const evidence = (shift = 0) => ({ protocol: PROTOCOL, workloadSha256: WORKLOAD_SHA256, complete: true, cleanup: { complete: true }, + timing: { actionAtMs: 1000, requestAtMs: 1005, releaseRequestedAtMs: 2005, releaseCommitAckAtMs: 2007, + absenceObservedAtMs: 2030 + shift, ackObservedAtMs: 2040 + shift } }); +const processes = () => ({ runner: { pid: 10, group: 10, birth: 'runner-birth' }, provider: { pid: 11, group: 11, birth: 'provider-birth' }, group: 10, providerGroup: 11 }); +const sample = () => ({ processes: { ...processes(), runnerGroupAlive: true, providerGroupAlive: true }, httpPending: true, intent: null, auditIds: [], processesAlive: true, clockOpen: true, blockers: [{ blocking_pids: [88] }] }); +const intentRow = () => ({ company_id: 'c', run_id: 'r', issue_id: 'i', intent: { + schema: 'paperclip.native-cancellation.v1', companyId: 'c', runId: 'r', issueId: 'i', + intentId: 'own', intentAuditId: 'a', dispatchState: 'acknowledged', acknowledgementAuditId: 'b', + scope: 'run', reasonCode: 'cancellation_run_only', dispatched: true, decisionId: null }, + audits: [{ id: 'a', action: 'native.cancellation_intent_recorded', details: { intentId: 'own' } }, + { id: 'b', action: 'native.cancellation_dispatch_acknowledged', details: { intentId: 'own', intentAuditId: 'a', dispatched: true, decisionId: null } }] }); + +test('geometry fits the unchanged 4,000 character bound, includes existing emitted history', () => { + const chars = '.'.repeat(W.prelude - 1) + ` PRELUDE_${'a'.repeat(32)}_READY ` + + Array.from({ length: W.events }, (_, index) => `[00${index.toString(36)}] `).join(''); + assert(chars.length < 4000); assert.equal(W.prelude, 1000); assert.equal(W.events, 256); assert.equal(W.intervalMs, 100); +}); +test('B0 and candidate append SQL match while cancellation full-row and unrelated SQL do not', () => { + for (const extra of ['', ', "next_event_seq", "event_cursor_projection_seq"']) assert(isAppendLockQuery( + `select "company_id", "agent_id"${extra} from "heartbeat_runs" where "heartbeat_runs"."id" = $1 limit $2 for update`)); + assert(!isAppendLockQuery('select "id", "company_id", "agent_id" from "heartbeat_runs" where "heartbeat_runs"."id" = $1 for update')); + assert(!isAppendLockQuery('select "company_id", "agent_id" from "heartbeat_runs" where "heartbeat_runs"."id" = $1')); +}); +test('held sample requires every barrier and physical liveness proof', () => { + assertHeldSample(sample(), 88, processes()); + for (const patch of [{ httpPending: false }, { intent: {} }, { auditIds: ['a'] }, { processesAlive: false }, { clockOpen: false }, { blockers: [] }]) + assert.throws(() => assertHeldSample({ ...sample(), ...patch }, 88, processes())); + assert.throws(() => assertHeldSample(sample(), 89, processes())); +}); +test('ACK correlates both committed audits to the same owned intent', () => { + assertIntentAudit(intentRow()); + for (const mutate of [row => { row.intent.runId = 'foreign'; }, row => { row.audits.pop(); }, + row => { row.audits[1].details.intentId = 'foreign'; }, row => { row.intent.intentAuditId = 'missing'; }]) { + const row = intentRow(); mutate(row); assert.throws(() => assertIntentAudit(row)); + } +}); +test('full action latency retains forced wait; release metrics use COMMIT ACK and also retain request-origin bounds', () => { + assert.deepEqual(metrics(evidence()), { actionToAckMs: 1040, releaseAckToAbsenceMs: 23, releaseAckToAckMs: 33, + releaseRequestToAbsenceMs: 25, releaseRequestToAckMs: 35 }); + for (const key of ['actionAtMs','requestAtMs','releaseRequestedAtMs','releaseCommitAckAtMs','absenceObservedAtMs','ackObservedAtMs']) { + const e = evidence(); delete e.timing[key]; assert.throws(() => metrics(e)); + } + const e = evidence(); e.timing.releaseRequestedAtMs--; assert.throws(() => metrics(e)); + e.timing.releaseRequestedAtMs++; e.complete = false; assert.throws(() => metrics(e)); +}); +test('registered companion is 50 separate alternating arms with fresh B0 and B1 A/A', () => { + const arms = registration(); assert.equal(arms.length, 50); + for (const stage of ['b0-aa','b1-aa','b0-b1','b1-b2','b0-b2']) { + const rows = arms.filter(a => a.stage === stage); assert.equal(rows.length, 10); + for (let pair = 1; pair <= 5; pair++) assert.equal(rows.filter(a => a.pair === pair).length, 2); + assert.deepEqual(rows.slice(0, 4).map(a => a.role), ['baseline','candidate','candidate','baseline']); + } +}); + +const functional = () => { + const e = evidence(), identity = { company_id: 'c', run_id: 'r', issue_id: 'i', runtime_mode: 'native', process_pid: 10, process_group_id: 10, + process_started_at: '2026-11-04T00:00:00.000Z', process_identity_seq: '1', process_identity_source_event_id: null, + process_identity: { targetKind: 'local', hostname: 'fixture-host', processPid: 10, processGroupId: 10, processStartedAt: '2026-11-04T00:00:00.000Z' }, + lease_id: 'lease', environment_id: 'environment', provider: 'local', provider_lease_id: null }; + return { ...e, variant: 'b0', startedAtMs: 100, finishedAtMs: 3000, companyId: 'c', runId: 'r', issueId: 'i', epochMs: 900, + prelude: { individualDeltas: 1000, uniqueIdentities: 1000, verifiedBeforeEpochAtMs: 850 }, + visibleBeforeLock: [0,1,2], lock: { id: 'r', company_id: 'c', holder_pid: 88 }, + appendBlockedBeforeStop: { observedAtMs: 1004, rows: [{ query: 'select "company_id", "agent_id" from "heartbeat_runs" where "heartbeat_runs"."id" = $1 for update', blocking_pids: [88] }] }, + heldSamples: [{ ...sample(), company_id: 'c', run_id: 'r', queryStartedAtMs: 1010, observedAtMs: 1015, emittedCount: 3 }, + { ...sample(), company_id: 'c', run_id: 'r', queryStartedAtMs: 1995, observedAtMs: 2004, emittedCount: 13 }], ack: intentRow(), stopHttpStatus: 200, + timing: { ...e.timing, httpSettledAtMs: 2042, lastAckNotObservedAtMs: 2036, lastProcessesPresentAtMs: 2026 }, + observerSamples: [{ kind: 'ack-query', startedAtMs: 2035, completedAtMs: 2036, durationMs: 1, value: false }, + { kind: 'ack-query', startedAtMs: 2039, completedAtMs: 2040, durationMs: 1, value: true }, + { kind: 'process-check', startedAtMs: 2025, completedAtMs: 2026, durationMs: 1, value: false }, + { kind: 'process-check', startedAtMs: 2029, completedAtMs: 2030, durationMs: 1, value: true }], processesBefore: processes(), processesAfter: { runnerAbsent: true, groupAbsent: true, providerAbsent: true, providerGroupAbsent: true, group: 10, providerGroup: 11 }, + authorityBefore: { rows: [{ ...identity }] }, + authorityAfter: { rows: [{ ...structuredClone(identity), status: 'cancelled', cancellation_dispatch_state: 'acknowledged', native_cancellation: intentRow().intent, + process_pid: null, process_group_id: null, process_started_at: null, process_stopped_seq: '20', + process_stopped_source_event_id: null, process_stopped: { processPid: 10, processGroupId: 10, processStartedAt: identity.process_started_at }, + process_lifecycle_seq: '20', process_lifecycle_event_type: 'native.local_process_stopped', process_lifecycle_source_event_id: null, + lease_status: 'released', lease_cleanup_status: 'success', lease_released_at: 'time' }] }, + authoritativeRetirementState: 'cleared-identity-with-authoritative-retirement', + lease: { id: 'lease', heartbeatRunId: 'r', companyId: 'c', provider: 'local', providerLeaseId: null, releasedAt: 'time', status: 'released', cleanupStatus: 'success' }, + transcript: { individualDeltas: 3, uniqueSourceIdentities: 3, persistedAfterReload: true, ordered: true, resultAcceptedEvents: 0, + preludeCount: 1000, seen: [{ index: 0 },{ index: 1 },{ index: 2 }], deltas: [0,1,2].map(index => ({ runId: 'r', eventType: 'item.delta', + sourceInstanceId: 'source', sourceEventId: `id${index}`, sourceSeq: index+1, payload: { text: `[00${index}] ` } })) }, + clock: { markerCode: '00', nonce: 'nonce', ended: true, records: [0,1,2].map(index => ({ index, nonce: 'nonce' })) }, + measurement: { actionToAckIncludesForcedWait: true, recording: { trace: 'off', video: 'off' } } }; +}; +test('functional oracle accepts complete correlated proof and rejects plausible missing or wrong outcomes', () => { + assertFunctional(functional()); + const mutations = [e => { delete e.ack; }, e => { e.ack.intent.intentId = 'other'; }, + e => { e.appendBlockedBeforeStop.rows[0].blocking_pids = []; }, e => { e.prelude.individualDeltas = 999; }, + e => { e.authorityAfter.rows[0].process_stopped.processPid = 99; }, e => { e.processesAfter.providerAbsent = false; }, + e => { e.heldSamples[1].emittedCount = 3; }, e => { e.transcript.deltas[1].payload.text = '[000] '; }, + e => { e.transcript.deltas[1].sourceEventId = 'id0'; e.transcript.deltas[1].sourceSeq = 1; }, + e => { e.transcript.persistedAfterReload = false; }, e => { e.measurement.recording.trace = 'on'; }]; + for (const mutate of mutations) { const e = functional(); mutate(e); assert.throws(() => assertFunctional(e)); } +}); + +const retained = (e = functional(), receipt = false) => { + const before = e.authorityBefore.rows[0], after = e.authorityAfter.rows[0]; + Object.assign(after, { process_pid: before.process_pid, process_group_id: before.process_group_id, process_started_at: before.process_started_at }); + if (!receipt) Object.assign(after, { process_stopped_seq: null, process_stopped: null, process_stopped_source_event_id: null, + process_lifecycle_seq: before.process_identity_seq, process_lifecycle_event_type: 'native.process_identity_recorded' }); + e.authoritativeRetirementState = 'retained-identity-after-physical-exit'; return e; +}; +test('retained original identity with NULL retirement receipt passes only with independent exit, own ACK and released lease', () => { + const e = retained(); + assert.equal(assertAuthoritativeRetirement(e), 'retained-identity-after-physical-exit'); assertFunctional(e); + assert.equal(e.authorityAfter.rows[0].process_stopped, null); +}); +test('cleared identity requires bound server retirement; a bound optional receipt also validates for retained identity', () => { + const cleared = functional(); assert.equal(assertAuthoritativeRetirement(cleared), 'cleared-identity-with-authoritative-retirement'); + assertFunctional(cleared); assertFunctional(retained(functional(), true)); +}); +test('both states require valid captured authoritative original birth, including before JSON serialization', () => { + for (const factory of [functional, retained]) { + const valid = factory(), before = valid.authorityBefore.rows[0], after = valid.authorityAfter.rows[0]; + before.process_started_at = new Date(before.process_started_at); + if (after.process_started_at !== null) after.process_started_at = new Date(after.process_started_at); + assertFunctional(valid); assertFunctional(JSON.parse(JSON.stringify(valid))); + for (const mutate of [e => { + const original = e.authorityBefore.rows[0], current = e.authorityAfter.rows[0]; + original.process_started_at = null; original.process_identity.processStartedAt = null; + current.process_identity.processStartedAt = null; current.process_started_at = null; + if (current.process_stopped) current.process_stopped.processStartedAt = null; + }, e => { delete e.authorityBefore.rows[0].process_started_at; }, + e => { e.authorityBefore.rows[0].process_started_at = 'not-a-birth'; e.authorityBefore.rows[0].process_identity.processStartedAt = 'not-a-birth'; }]) { + const invalid = factory(); mutate(invalid); + assert.throws(() => assertFunctional(invalid)); + } + } +}); +test('cleared identity rejects absent, unbound, partial or later-invalidated retirement evidence', () => { + const mutations = [ + row => { row.process_stopped_seq = null; row.process_stopped = null; }, + row => { row.process_stopped_seq = null; }, row => { row.process_stopped_source_event_id = 'provider-event'; }, + row => { row.process_stopped.processPid = 99; }, row => { row.process_stopped.processGroupId = 99; }, + row => { row.process_stopped.processStartedAt = '2026-11-04T00:00:01.000Z'; }, row => { delete row.process_stopped.processStartedAt; }, + row => { row.process_stopped_seq = '1'; }, row => { row.process_started_at = '2026-11-04T00:00:00.000Z'; }, + row => { row.process_lifecycle_seq = '21'; row.process_lifecycle_event_type = 'native.process_start_requested'; }, + row => { row.process_identity_seq = '21'; }, row => { row.process_lifecycle_seq = '21'; }, + row => { row.process_lifecycle_source_event_id = 'provider-event'; }]; + for (const mutate of mutations) { const e = functional(); mutate(e.authorityAfter.rows[0]); assert.throws(() => assertAuthoritativeRetirement(e)); } +}); +test('retained identity rejects changed PID, group, birth, server identity or later launch and verifies optional receipts', () => { + const mutations = [row => { row.process_pid = null; }, row => { row.process_group_id = null; }, + row => { row.process_pid = 99; }, row => { row.process_group_id = 99; }, + row => { row.process_started_at = '2026-11-04T00:00:01.000Z'; }, row => { row.process_started_at = null; }, + row => { row.process_identity.processStartedAt = 'other'; }, row => { row.process_identity_seq = '2'; }, + row => { row.process_lifecycle_seq = '2'; row.process_lifecycle_event_type = 'native.process_start_requested'; }, + row => { row.process_identity_source_event_id = 'provider-event'; }, row => { delete row.process_stopped; }]; + for (const mutate of mutations) { const e = retained(); mutate(e.authorityAfter.rows[0]); assert.throws(() => assertAuthoritativeRetirement(e)); } + const optional = retained(functional(), true); optional.authorityAfter.rows[0].process_stopped.processPid = 99; + assert.throws(() => assertAuthoritativeRetirement(optional)); +}); +test('neither retirement state bypasses physical process/group absence, own audited ACK or exact lease cleanup', () => { + const mutations = [e => { delete e.processesAfter.providerGroupAbsent; }, e => { e.processesAfter.groupAbsent = false; }, + e => { e.processesAfter.runnerAbsent = false; }, e => { e.processesAfter.providerAbsent = false; }, + e => { e.authorityAfter.rows[0].native_cancellation.intentId = 'other'; }, e => { e.ack.audits.pop(); }, + e => { e.authorityAfter.rows[0].status = 'failed'; }, e => { e.lease.id = 'other'; }, + e => { e.authorityAfter.rows[0].lease_id = 'other'; }, e => { e.lease.releasedAt = null; }, + e => { e.authorityAfter.rows[0].lease_released_at = null; }, e => { e.authorityAfter.rows[0].lease_cleanup_status = 'failed'; }, + e => { e.authorityAfter.rows[0].provider = 'remote'; }]; + for (const factory of [functional, retained]) for (const mutate of mutations) { + const e = factory(); mutate(e); assert.throws(() => assertAuthoritativeRetirement(e)); + } + const forged = retained(); forged.authoritativeRetirementState = 'cleared-identity-with-authoritative-retirement'; + assert.throws(() => assertFunctional(forged)); +}); + +test('shared decision is correlated when present and absent run-only decisions stay valid', () => { + const row = intentRow(); assertIntentAudit(row); + row.intent.decisionId = 'decision'; row.audits[1].details.decisionId = 'decision'; + row.shared_decision = { id: 'decision', companyId: 'c', runId: 'r', issueId: 'i' }; assertIntentAudit(row); + row.shared_decision.runId = 'foreign'; assert.throws(() => assertIntentAudit(row)); +}); + +const arm = (index, shift = 0, variant = 'b0') => { + const offset = index * 10000; + const evidence = JSON.parse(JSON.stringify(functional()), (key,value) => { + if (value === 'r') return `r${index}`; if (value === 'c') return `c${index}`; if (value === 'i') return `i${index}`; + return typeof value === 'number' && (key.endsWith('AtMs') || key === 'epochMs') ? value + offset : value; + }); + evidence.variant = variant; + for (const key of ['ackObservedAtMs','absenceObservedAtMs','lastAckNotObservedAtMs','lastProcessesPresentAtMs','httpSettledAtMs']) + evidence.timing[key] += shift; + for (const sample of evidence.observerSamples) { sample.startedAtMs += shift; sample.completedAtMs += shift; } + return evidence; +}; +const pairs = (shift = () => 0) => Array.from({ length: 5 }, (_, pair) => [ + arm(2*pair + (pair % 2), shift(pair,0)), arm(2*pair + (pair % 2 ? 0 : 1), shift(pair,1))]); + +test('noise artifact freezes actual suitable A/A limits with counts, spread, chronological drift and resolution labels', () => { + const noise = noiseEnvelope(pairs((pair,side) => side ? pair+1 : 0)); + assert.equal(noise.suitable, true); assert.equal(noise.noiseFrozen, true); assertNoiseEnvelope(noise); + assert.deepEqual(noise.sampleCounts, { pairs: 5, arms: 10 }); assert.equal(noise.rawArms.length, 10); + for (const metric of METRICS) { + const row = noise.metrics[metric]; assert.equal(row.noiseMs, 5); assert.equal(row.sampleCount, 10); + assert.equal(row.precisionBudgetMs, Math.max(2*W.pollIntervalMs, 0.10*row.pooledMedianMs)); + assert(row.maximumObserverBracketMs <= row.precisionBudgetMs); + } + assert.equal(noise.metrics.releaseAckToAckMs.resolutionLabel, '100 ms resolution floor'); + assert.equal(noise.metrics.releaseAckToAckMs.medianDifferencePercent, null); + assert.equal(noise.metrics.releaseAckToAckMs.percentageSpeedClaimsAllowed, false); +}); +test('every calibration and comparison arm must pass the complete functional oracle', () => { + const noise = noiseEnvelope(pairs()); + const mutations = [e => { delete e.lock; }, e => { delete e.ack; }, e => { delete e.processesAfter; }, + e => { delete e.lease; }, e => { delete e.transcript; }, e => Object.assign(e, evidence()), + e => { e.transcript.persistedAfterReload = false; }]; + for (const mutate of mutations) { + const invalid = pairs(); mutate(invalid[4][1]); + assert.throws(() => noiseEnvelope(invalid)); assert.throws(() => comparePairs(invalid, noise)); + } + const forged = pairs(); forged[0][0] = evidence(); + assert.throws(() => noiseEnvelope(forged)); assert.throws(() => comparePairs(forged, noise)); +}); +test('paired jitter and chronological drift outside prospective budget retain all arms and cannot freeze noise', () => { + for (const data of [pairs((pair,side) => side ? 150 : 0), pairs(pair => pair * 60)]) { + const noise = noiseEnvelope(data); + assert.equal(noise.suitable, false); assert.equal(noise.noiseFrozen, false); assert.equal(noise.rawArms.length, 10); + assert(METRICS.every(metric => noise.metrics[metric].noiseMs === null)); + assert.throws(() => assertNoiseEnvelope(noise)); assert.throws(() => comparePairs(pairs(), noise)); + } + const drift = noiseEnvelope(pairs(pair => pair * 60)); + assert.equal(drift.metrics.releaseAckToAckMs.maximumAbsolutePairedDifferenceMs, 0); + assert(drift.metrics.releaseAckToAckMs.absoluteChronologicalMedianDriftMs > drift.metrics.releaseAckToAckMs.precisionBudgetMs); +}); +test('raw observer brackets retain actual query/check durations and reject missing or forged timing', () => { + const e = functional(), brackets = observerBrackets(e); + assert.equal(brackets.releaseAckToAckMs.widthMs, 5); + assert.equal(brackets.releaseAckToAckMs.negativeDurationMs, 1); + assert.equal(brackets.releaseAckToAbsenceMs.successfulDurationMs, 1); + e.observerSamples[0].durationMs = 0; assert.throws(() => observerBrackets(e)); + delete e.observerSamples; assert.throws(() => assertFunctional(e)); +}); +test('a wide observer bracket invalidates A/A suitability and comparison despite stable latency medians', () => { + const data = pairs(() => 120), e = data[0][0]; + e.observerSamples = e.observerSamples.filter(sample => sample.kind !== 'ack-query' || sample.value); + delete e.timing.lastAckNotObservedAtMs; + const unsuitable = noiseEnvelope(data); + assert.equal(unsuitable.metrics.releaseAckToAckMs.maximumAbsolutePairedDifferenceMs, 0); + assert.equal(unsuitable.suitable, false); assert.throws(() => assertNoiseEnvelope(unsuitable)); + const noise = noiseEnvelope(pairs(() => 120)); + const result = comparePairs(data, noise); + assert.equal(result.releaseAckToAckMs.pass, false); + assert.equal(result.releaseAckToAckMs.noiseMs, 0); +}); +test('comparison uses unchanged frozen noise and budget; one paired regression or inflated artifact fails', () => { + const calibration = pairs((pair,side) => side ? pair+1 : 0), noise = noiseEnvelope(calibration); + assert(Object.values(comparePairs(calibration, noise)).every(row => row.pass)); + const regression = pairs((pair,side) => side ? 6 : 0); + assert(Object.values(comparePairs(regression, noise)).every(row => !row.pass)); + const forged = structuredClone(noise); forged.metrics.releaseAckToAckMs.noiseMs = 999; + assert.throws(() => comparePairs(pairs(), forged)); + const inflated = structuredClone(noise); inflated.metrics.releaseAckToAckMs.precisionBudgetMs = 999; + assert.throws(() => comparePairs(pairs(), inflated)); + assert.throws(() => noiseEnvelope(pairs().slice(1))); +}); +test('optional shared cancellation accepts absent/requested B0 metadata and validates present state, time, attribution and run scope', () => { + const when = '2026-11-04T10:00:00.000Z'; + const e = functional(); assertFunctional(e); + e.ack.shared_execution_cancellation = { state: 'requested', requestedAt: when }; + e.ack.run_cancellation = { source: 'operator', expected: true, initiator: { type: 'user' }, reason: 'Operator Stop', recordedAt: when }; + assertFunctional(e); + e.ack.shared_execution_cancellation = { state: 'acknowledged', acknowledgedAt: when }; assertFunctional(e); + for (const patch of [{ state: 'requested' }, { state: 'acknowledged', acknowledgedAt: 'invalid' }, + { state: 'other', requestedAt: when }]) { + const bad = structuredClone(e); bad.ack.shared_execution_cancellation = patch; assert.throws(() => assertFunctional(bad)); + } + const wrong = structuredClone(e); wrong.ack.run_cancellation.source = 'provider'; assert.throws(() => assertFunctional(wrong)); + const foreign = structuredClone(e); foreign.sharedSettlement = { companyId: 'c', runId: 'foreign', status: 'cancelled' }; + assert.throws(() => assertFunctional(foreign)); +}); + +test('calibration and comparison reject reused runs, overlapping arms and nonalternating chronology', () => { + const noise = noiseEnvelope(pairs()); + for (const corrupt of [data => { data[4][1] = data[0][0]; }, + data => { data[0][1].startedAtMs = data[0][0].finishedAtMs; }, + data => { data[1].reverse(); }]) { + const invalid = pairs(); corrupt(invalid); + assert.throws(() => noiseEnvelope(invalid)); assert.throws(() => comparePairs(invalid, noise)); + } +}); + +test('distinct provider group is valid, same-group topology is also valid, and both exact owned groups must be proved', () => { + const distinct = functional(); assert.notEqual(distinct.processesBefore.group, distinct.processesBefore.providerGroup); assertFunctional(distinct); + const shared = functional(); shared.processesBefore.provider.group = 10; shared.processesBefore.providerGroup = 10; + shared.processesAfter.providerGroup = 10; + for (const held of shared.heldSamples) { held.processes.provider.group = 10; held.processes.providerGroup = 10; } + assertFunctional(shared); + for (const mutate of [e => { delete e.processesBefore.providerGroup; }, e => { e.processesBefore.providerGroup = 99; }, + e => { e.processesBefore.provider.group = 0; }, e => { delete e.processesAfter.providerGroupAbsent; }, + e => { e.processesAfter.providerGroupAbsent = false; }, e => { e.processesAfter.providerGroup = 99; }, + e => { e.authorityBefore.rows[0].process_group_id = 99; }, e => { e.processesBefore.runner.group = 99; }, + e => { e.heldSamples[1].processes.provider.birth = 'different-birth'; }, e => { e.heldSamples[1].processes.runner.pid = 99; }, + e => { e.heldSamples[1].processes.providerGroupAlive = false; }, e => { e.heldSamples[1].processes.runnerGroupAlive = false; }, + e => { delete e.heldSamples[1].processes; }]) { + const e = functional(); mutate(e); assert.throws(() => assertFunctional(e)); + } +}); +test('cleanup eligibility follows supported active/scheduled and native retryable phases only', () => { + for (const status of ['queued','running','scheduled_retry']) assert(cleanupCancellationEligible({ status })); + assert(cleanupCancellationEligible({ status:'failed', runtimeMode:'native', nativePhase:'retryable_failure' })); + for (const run of [{ status:'failed', runtimeMode:'legacy', nativePhase:'retryable_failure' }, + { status:'failed', runtimeMode:'native', nativePhase:'terminal_failure' }, { status:'failed', runtimeMode:'native' }, + { status:'succeeded' }, { status:'cancelled' }, { status:'timed_out' }]) assert.equal(cleanupCancellationEligible(run), false); +}); +test('owned cleanup records public cancellation before closing its clock, including retryable failure', async () => { + for (const before of [{ status:'running', runtimeMode:'native' }, { status:'failed', runtimeMode:'native', nativePhase:'retryable_failure' }]) { + const order=[],record={}; let open=true,run={ id:'r',companyId:'c',...before }; + await cleanupOwnedRunBeforeClockClose({ companyId:'c',runId:'r',record, + readRun:async()=>{ assert(open);order.push('read');return run; }, + cancelRun:async()=>{ assert(open);order.push('cancel');run={...run,status:'cancelled',resultJson:{nativeCancellation:{dispatchState:'acknowledged'}}};return {status:200,run}; }, + closeClock:async()=>{ order.push('close');open=false; } }); + assert.deepEqual(order,['read','cancel','read','close']); assert.equal(record.before.status,before.status); + assert.equal(record.cancel.httpStatus,200); assert.equal(record.after.status,'cancelled'); assert.equal(record.after.nativeCancellation.dispatchState,'acknowledged'); + assert.equal('providerAbsent' in record,false,'Cancellation response is not physical Stop evidence'); + } +}); +test('cleanup does not cancel terminal/foreign runs and closes clock after a failed cancellation attempt', async () => { + for (const mode of ['terminal','foreign','cancel-error']) { + const order=[],record={};let cancelled=false; + const action=()=>cleanupOwnedRunBeforeClockClose({companyId:'c',runId:'r',record, + readRun:async()=>{order.push('read');return {id:'r',companyId:mode==='foreign'?'other':'c',status:mode==='terminal'?'succeeded':'running'};}, + cancelRun:async()=>{cancelled=true;order.push('cancel');throw Error('controlled cancellation error');}, + closeClock:async()=>{order.push('close');} }); + if(mode==='terminal'){await action();assert.equal(record.cancel.attempted,false);assert.equal(cancelled,false);} + else {await assert.rejects(action());assert(record.error);assert.equal(cancelled,mode==='cancel-error');} + assert.equal(order.at(-1),'close');assert(record.clockClosedAtMs); + } +}); diff --git a/tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs b/tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs new file mode 100644 index 00000000000..e6eb8383dd6 --- /dev/null +++ b/tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs @@ -0,0 +1,22 @@ +// Read-only validation of an independently extracted raw attachment; no app/DB/process execution. +import { readFile, writeFile } from 'node:fs/promises'; +import { createHash } from 'node:crypto'; +import { pathToFileURL } from 'node:url'; +import { assertFunctional, PROTOCOL, WORKLOAD_SHA256, registration } from './native-ingestion-blocked-stop-protocol.mjs'; +export async function validateFile(file) { + const bytes = await readFile(file), evidence = JSON.parse(bytes.toString('utf8')); + const metrics = assertFunctional(evidence); + return { protocol: PROTOCOL, workloadSha256: WORKLOAD_SHA256, sourceFile: file, + sourceSha256: createHash('sha256').update(bytes).digest('hex'), sourceRevision: evidence.sourceRevision, + runId: evidence.runId, metrics, functionalPass: true, + qualification: 'Functional gate only; owning Playwright, frozen preflight, cleanup and paired noise/comparison campaign remain separately required' }; +} +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + if (process.argv[2] === '--registration') process.stdout.write(`${JSON.stringify(registration(), null, 2)}\n`); + else { + if (!process.argv[2]) throw new Error('Pass extracted raw companion evidence JSON, or --registration'); + const result = await validateFile(process.argv[2]); + if (process.argv[3]) await writeFile(process.argv[3], `${JSON.stringify(result, null, 2)}\n`, { flag: 'wx' }); + else process.stdout.write(`${JSON.stringify(result, null, 2)}\n`); + } +} diff --git a/tests/e2e/fixtures/native-ingestion-blocked-stop.md b/tests/e2e/fixtures/native-ingestion-blocked-stop.md new file mode 100644 index 00000000000..0fb45d663f8 --- /dev/null +++ b/tests/e2e/fixtures/native-ingestion-blocked-stop.md @@ -0,0 +1,257 @@ +# Existing-history blocked Stop companion (prospective instrumentation v3) + +Status: v3 local source and pure oracle calibration only. No v3 application, browser, +PostgreSQL, VM or quantitative arm has run. No runtime qualification or performance +claim follows from the pure tests. Execution needs lead release after independent +source review. This is separate from the immutable H5c 50-arm campaign. + +## Preserved v1/v2 qualification failures and prospective corrections + +The one v1 qualification at `200396d5` failed before the emission epoch because +the probe assumed the provider shared the Runner process group. Actual Runner +PID/group was 2284214; provider PID/group was 2284218. Separate groups are allowed. +No held-append, Stop timing or calibration data was collected. The failed raw +qualification remains retained and cannot supply a v2 timing arm or noise sample. + +Normal API cleanup returned 200 and the port was released, but strict physical +quiet failed: both exact owned processes remained sleeping after the app closed. +The old finally block closed the clock before cancelling the run and did not +attempt cancellation for failed native retryable phases. Those observations do not establish a product +cause or successful physical termination from the API response. + +This v2 source records independent provider PID birth and process group, proves +both groups alive during the hold and absent after Stop, and keeps the server +termination receipt correlated to the Runner group. Every held sample retains +both original process births. Shared-group topology remains valid when observed. +Cleanup releases its owned lock, attempts supported public cancellation while the +clock is still available, records before/HTTP response/after status and native +cancellation metadata, and then closes the clock. No manual process kill belongs +to a passed functional oracle; exact owned cleanup after a failed qualification +is separate negative evidence. New source review and runtime qualification are +required. Protocol name, workload, precision rule and all deadlines are unchanged. + +The one v2 qualification at `6e5349cd` reached acknowledged native cancellation, +cancelled settlement and independently observed Runner/provider PID and group +absence. It then failed the probe's unconditional requirement for an immediate +`native.local_process_stopped` event: the row still retained original PID/group +2286265, its latest server identity sequence was 19, and the stop receipt was NULL. +Transcript, reload and the later lease gate were not reached. That requirement +exceeded the supported product contract. The failed trial and +its observed timings remain negative evidence, cannot supply a v3 timing arm or +noise sample, and are not reclassified as a qualified measurement. + +V3 reads final authority after the existing lease-release poll in one existing +READ ONLY REPEATABLE READ snapshot, including the native cancellation intent, +lease cleanup and latest server-authored process lifecycle fence. It adds no +poll, wait budget or timing shift: ACK and physical-absence observations retain +their original boundaries. `assertAuthoritativeRetirement` records either +`retained-identity-after-physical-exit` or +`cleared-identity-with-authoritative-retirement`. Both require cancelled state, +the same own audited ACK, the original owned local lease released without failed +cleanup, and independently observed absence of both exact PIDs and both groups. +Retained mutable PID/group/birth and latest server identity must exactly match +the captured original identity. An absent retirement receipt is valid only in +that retained state. If a receipt is present, it must still bind to the original +PID/group/birth. Cleared mutable identity requires that bound server-authored +retirement receipt, later than the latest identity, with no later process start +or identity event. Partial, changed or relaunched identity is rejected. + +The authoritative product basis is +`server/src/services/acknowledged-native-stop.ts`, +`acknowledgedNativeStopExecutionHasStopped`: after local lease cleanup, retained +PID/group use the host's ESRCH observations; cleared fields require +`hasNativeLocalProcessStop`. In `native-local-process-stop.ts`, +`recordNativeLocalProcessStop` records the original PID/group/birth before identity +clearing, while `hasNativeLocalProcessStop` rejects a later server start/identity. +`releaseSettledNativeRunProcess` separately gates retirement on coordinator, +provider and lease state. Native cancellation ACK persistence in +`native-session-executor.ts` does not itself retire process identity. The event is +authoritative identity-clearing evidence, not a universal immediate Stop receipt +or a substitute for physical process observation. This is a probe correction; +it includes no product change or attribution of a product defect. + +## Registration and ownership + +Explicit config: `tests/e2e/playwright-native-ingestion-blocked-stop.config.ts`. +Explicit source: `tests/e2e/native-event-ingestion-blocked-stop.journey.ts`. +Protocol: `blocked-stop-history-v1`; workload digest comes from the exact exported +`WORKLOAD` JSON in `native-ingestion-blocked-stop-protocol.mjs`. + +The config inherits the unchanged H5c production-built app bootstrap, local +Codex command fixture, normal local environment, and shipped UI. It selects only +the companion source and fixes a 900-second test budget. There is one owned +control agent and one task created and assigned through visible normal controls. +Provider execution is a deterministic existing Codex leaf fixture, not a live +model or a real Cloud provider. This companion grants no Cloud Stop qualification. +No resolver override, HTTP/API change, persisted outcome seeding or app/product +code change is included. Authority/gate/paint code is a bounded copy from H5c +`e431ec63d41fe90c6fd68bd2fc2bb969feeca917`; the original nine measured files stay +unchanged. The new files have their own source identity and review. + +## Fixed workload and functional gate + +All variants emit 1,000 actual prelude deltas, independently read back through +normal event APIs before the common gate releases. They then emit 256 compact +markers at 100 ms, fitting the unchanged 4,000 character Rust text bound together +with the prelude. Readiness is bounded at 300 seconds, measured streaming at +120 seconds, terminal settlement at 180 seconds; original later 30-second ACK +and lease observations remain after settlement. Setup stays inside the owning +900-second test budget. Server bootstrap retains its existing separate limit. + +After at least three visible markers and before Stop, a separately owned +PostgreSQL connection acquires `FOR UPDATE` on the exact company/run row. A +second connection reads `pg_stat_activity` and `pg_blocking_pids` and must see +an actual append query waiting on that holder. Its SELECT projection prefix is +checked against the B0/B1/B2 source signatures, distinguishing cancellation's +full-row SELECT. Raw SQL metadata is retained privately; no product hook is used. + +The ordinary visible Stop button must cause the actual matching public POST. +For at least 1,000 ms after that observed request, repeated independent snapshots +must show pending HTTP, no committed native cancellation intent or audit, the +same runner/provider PID births alive in their exact group, and an open fixture +clock. The fixture closes that clock on `turn/interrupt`. Provider emissions must +continue across the hold. The hold is not subtracted from action-to-ACK latency. +A late sample/query/timer can overrun the minimum; actual boundaries stay visible. + +Release records both COMMIT request and COMMIT response acknowledgement. Neither +is claimed to be the exact PostgreSQL commit instant. Independent observations +then require the cancellation intent and its own matching committed audit, +native acknowledged dispatch and its own matching acknowledgement audit, +normal cancelled settlement, successful Stop HTTP, independently absent +runner/group/provider/provider-group, and the exact owned local lease released +without failed cleanup. The coherent final snapshot must meet the retained or +cleared identity contract above; cleared fields require the bound server-authored +`native.local_process_stopped` retirement receipt. Transcript checks require complete compact payloads, unique source +identities, ordered canonical prefix, original emission clocks, zero accepted +results, and persistence after reload. The raw transport recovery policy stays +unchanged: raw fixture emissions after the canonical prefix are not represented +as universally database-durable. + +The native receipt, shared `executionCancellation` metadata (including absence) +and public cancelled run are recorded separately. The native dispatch audit must +match the same intent and intent audit, `dispatched: true`, and nullable decision +ID. A non-null decision is independently correlated through the owned +`status_decisions` company/issue/run row. Run-only Stop may have no decision; a +missing optional decision or shared field is not silently invented. A shared ACK field is not synthesized from +row `recordedAt` or a provider clock. Source/runtime review must confirm which +shared receipt contract the serving revision exposes before releasing the probe. +No universal 30-second click-to-ACK gate is introduced. + +Every actual ACK query and process check through its first success retains its +start, completion, duration and boolean observation. An uncertainty bracket runs +from the last negative query/check start (conservative for snapshot-read timing) +or release-COMMIT-ACK when no negative was seen, through first successful +completion. Both negative and successful operation durations stay explicit. + +Timing includes action-to-observed-native-ACK, release-COMMIT-ACK-to-independently +observed absence, and release-COMMIT-ACK-to-observed-native-ACK. Both release +request-origin values also remain. Last unsuccessful and successful-query/check +bounds are retained with a 50-ms poll interval; query duration and scheduling add +uncertainty. Screenshots and reload happen after those timing observations. + +Any failure, timeout, censored observation, query/clock fault or cleanup error +makes the arm ineligible. Partial raw Playwright results and the final attachment +remain even after failure. Cleanup first releases/rolls back only the owned lock, +closes both connections, settles only its owned run while its clock is available, +then retires the clock and restores owned experimental settings. Supported public +phases are queued/running/scheduled_retry and failed native retryable_failure; +other terminal phases are not presumed cancelable or physically stopped. This +cleanup subset follows `server/src/services/heartbeat/recovery.ts` active statuses +and `native-cancellation-request.ts` retryable eligibility, reached through the +normal board POST in `routes/agents.ts`. The server rechecks durable coordinator +eligibility and owns cancellation barriers; the probe does not grant authority. + The external operator must independently prove the +server, database, runner, provider and browser are gone before the next arm. + +## Separate quantitative registration (50 short arms) + +Generate the explicit registration without executing an app: + +```sh +node tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs --registration +``` + +Order is B0 A/A five fresh alternating pairs, B1 A/A five fresh alternating pairs, +then B0→B1, B1→B2 and B0→B2 five alternating pairs each. Each pair is two fresh +owned invocations with matching data/load/payload/resource/cache conditions. +There are no repeats hidden inside an invocation. Keep the original main 50 arms +and their noise files untouched. + +Before any companion data, freeze this exact suitability rule independently per +metric. The precision budget is `max(2 * 50 ms, 0.10 * pooled ten-arm A/A metric +median)`. Both the maximum absolute paired difference and the absolute median +drift between the first five and last five chronological arms must fit that +budget. Every raw ACK/absence observer bracket must also fit the corresponding +metric budget; action-to-ACK uses the ACK observer bracket. This is a +measurement repeatability rule, not a product Stop deadline. + +Only suitable calibration freezes its observed maximum absolute paired +difference as noise. B0 comparisons use B0 A/A; B1→B2 uses B1 A/A. Every paired +candidate difference and their median must remain within that own frozen noise, +and all comparison observer brackets must fit the frozen baseline precision +budget. No unblocked Stop noise is reused. An unsuitable ten-arm calibration +retains every raw arm and produces no usable noise envelope: no silent +replacement, budget increase or inflated envelope follows from noisy data. + +`noiseEnvelope(pairs)` takes five logical `[baseline, candidate]` raw-report +pairs whose chronological order alternates baseline-first then candidate-first. +It validates the full functional oracle for every arm, requires ten distinct +owned runs/companies with nonoverlapping arm times, and returns the structured +`paperclip.native-ingestion.blocked-stop-noise.v1` artifact. This contains +sample counts, all raw arms and report digests, logical pair bindings, actual +absolute/percentage median differences, spread, chronological drift, budgets, +query/check uncertainty brackets, suitability, and nullable noise values. +Percentage fields are null when the 100-ms precision floor determines the +budget. Those metrics are labeled **100 ms resolution floor** and cannot support +subfloor, p99 or percentage speed claims. Ten observed arms do not establish p99. + +`assertNoiseEnvelope(artifact)` revalidates all retained functional proofs and +recomputes suitability and numeric limits; unsuitable or inflated artifacts +throw. `comparePairs(pairs, artifact)` validates every comparison arm's complete +functional proof and returns per-metric `.pass` results using only that +baseline's unchanged frozen noise and budget. It rejects reused runs, +nonalternating/overlapping arms, missing lock/ACK/process/lease/transcript proof, +and forged `complete` flags. Preserve all failures and censored arms outside +these successful artifacts as well. A single functional probe is not +quantitative nonregression evidence. + +## Reproduction and execution gates + +Credential-free local checks, with Node 24: + +```sh +node --test tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.test.mjs +node --check tests/e2e/native-event-ingestion-blocked-stop.journey.ts +node --check tests/e2e/playwright-native-ingestion-blocked-stop.config.ts +node tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs --registration +``` + +Before runtime, the lead must review/freeze the companion source, workload digest, +registration, validator and companion-only release descriptor. Keep the reviewed +H5c private runner `6b6376b01de6cdc15ca05343a0bdd5c00424f1d6c546b9b4d61cd76716b253c4` +unchanged. A companion orchestrator should import its checked exported +`validateVariantBuild`, `validateActualInventory`, `assertNoPreferredUiShadow`, +`physicalQuietGuard`, `sealArmArtifacts` and `verifyRawArtifactSeals`; it must not +invoke or alter the original stage plan. Bind each arm to exact source/build/UI, +qualified receipt, full inventory manifest, Node/dependency/native pins, unchanged +frozen fixture hashes, companion hashes, recording-off flags and a clean source. +Use original B0 `bac53ec3`, B1 `07061a60`, unoptimized B2 `ca7cba17` only where +explicitly released; a future optimized B2 needs its own qualified descriptor. +The existing build receipt `a1259ba7…`, full inventory `d3766fc4…` and native binary +`abb65bdc…` are provenance, not substitutes for current artifact validation. + +Only the performance owner may stage/execute on the sole benchmark VM, after +lead release. First do discovery, narrow static typecheck, and one smallest +functional qualification with raw Playwright, server/banner, PostgreSQL blocker, +process/clock and external cleanup evidence. Validate its extracted attachment: + +```sh +node tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs RAW.json RECEIPT.json +``` + +Do not start the 50-arm registration until the source and runtime probe qualify. +Capture quiet before/after-arm receipts, owning Playwright exit/status/duration, +raw API/SQL/process evidence, immutable report seals, and all cleanup outcomes. +The source validator is an additional functional oracle; it cannot establish the +owning Playwright pass, infrastructure cleanup, build authenticity or performance +on its own. Freeze separate B0/B1 A/A noise before nonregression comparisons. diff --git a/tests/e2e/fixtures/native-ingestion-h5.md b/tests/e2e/fixtures/native-ingestion-h5.md index 94c16439628..a503f1cdca8 100644 --- a/tests/e2e/fixtures/native-ingestion-h5.md +++ b/tests/e2e/fixtures/native-ingestion-h5.md @@ -305,3 +305,165 @@ queue/backlog, delay and memory with identical capture hooks. Aborted profiles remain diagnostic-only and cannot establish complete CPU/event costs. No CPU requirement or profiling is added to this timing comparator. Local results do not establish deployed improvement. + +## Prospective H5d oracle instrumentation and verified-completion gate + +H5d adds oracle instrumentation and a prospective completion gate from frozen H5c +`e431ec63d41fe90c6fd68bd2fc2bb969feeca917`. It keeps the exact H5/H4e mode, +protocol, registration, workload and deadline constants. The new driver commit +and changed nine-file measurement digest distinguish the experiment. No product +optimization, new buffering, setup/control flow, payload bound, provider fixture, +transport, public API, P0 ordering or raw-ACK durability policy changes here. +All fifty main arms, including fresh B0/B1 A/A, must be rerun on the qualified new +source; old noise or the failed H5c attempt cannot qualify H5d. + +The retained H5c FULL B0 attempt reached the unchanged 1,200,000-ms observation +ceiling while fetching/validating independent durable events. Root's retained +raw evidence records all 100 runs succeeded within 75.926 seconds of the same +180-second settlement window, all 100 clock uploads ended with 51,200 emissions +and no clock errors, and only roughly 30 of 100 per-run durable reads completed +before timeout. Foreground completion remained 283.391 seconds, with only +66/512 markers at the original streaming deadline. This owning attempt failed; +complete in-journey transcript/results verification and later cases were not +finished. A separate stopped read-only supplement cannot turn it into an arm +pass. Product improvement is not established by that failure or this correction. + +Frozen H5c performed five Playwright assertions for every measured normalized +delta and one Playwright deep-equality assertion for its original clock join. +It also asserted each prelude payload and each visible clock through Playwright. +H5d uses the same actual paginated API rows and provider clock maps, then validates +those retained observations through Node assertions. Only verified per-run +summaries enter the existing durable evidence array. The active failure record +also identifies the current run, pagination cursor and returned row count. + +| Frozen check | H5d equivalent | +| --- | --- | +| Exact complete compact payload for every measured delta | `assertDurableRunEvents` checks the same full text, rejecting coalescing or added bytes | +| Exact run, truthy source-instance/event IDs, positive safe integer sequence in H5 | Same assertions under the unchanged `requireNativeIdentity: h5` condition | +| Original nonce, marker code and index for every durable event | Same tuple deep equality against the original provider clock map | +| Unique normalized identity tuple and complete ordered indices | Same set count and exact ordered index array; Stop permits only its canonical prefix | +| Every visible Stop marker is durable; normal result one / Stop result zero | Same prefix membership and exact accepted-result count | +| Complete normal source-clock count and closed uploads | Existing clock-readiness/closure polls remain; Node oracle also checks the normal count | +| Prelude exact dots/final readiness text, count and unique identity tuples | `assertPreludeEvents` checks the same filtered prelude rows before epoch release | +| Every visible marker has an original clock; complete normal unique visible count | `assertVisibleMarkerClocks` checks the same observations and preserves the original two-frame timestamp | + +The same `/events?afterSeq=…&limit=1000` pages, advancing cursor checks and final +empty-page read remain. The 30-second per-run provider-clock availability poll +is unchanged and still occurs before the Node assertion. The 120/180/900-second +original gates, 600/180/1,200-second observation ceilings, original late Stop ACK +and lease polls, and outer resource watchdog remain unchanged. No per-event +check is dropped to make the invocation finish. + +The frozen comparator9e01 validated `h5.independentDrain` correctness but omitted +its elapsed duration from quantitative gates. A synthetic RED counterexample +actually passed that comparator with faster browser/terminal delivery while full +verified completion regressed from180,010 to200,000 ms. The prospective comparator +adds `h5.independentDrain.elapsedFromEpochMs` to every non-Stop H5 case's existing +measurement, own A/A noise and nonregression loops. For loaded burst512, final B2 +must also improve that duration beyond its own noise in the median and at least +four of five pairs, alongside the existing browser p95 and terminal-duration +gates. Three improving pairs do not pass. Cursor nonregression includes the same +verified-completion metric. All previous gates and timing limits remain. + +Reports also derive measured normalized deltas per second from equal matched +counts divided by that independently verified completion upper bound. For the +loaded burst, N is51,200 complete measured deltas, with all100 accepted results +independently checked. The duration includes oracle, reload, screenshot, normal +API and lease verification overhead; it is not the exact last database-commit +instant. This conservative observed completion rate is descriptive: the existing +noise gate applies to duration, and no separate rate-noise threshold is invented. +Equal counts make a shorter fully verified duration monotonically higher rate. +Missing/tampered completion proof still fails the existing functional oracle. +No accepted A/A exists. The following finite browser repeatability rule is +chosen prospectively; source freezing still requires the combined root/peer review. + +### Prospective main-browser A/A suitability (H5 only) + +Before any B0 or B1 calibration data, choose exactly five alternating pairs and +apply these rules independently to every declared case/metric after all existing +functional, owning-report and cleanup checks. With pair durations a,b: + +- Every paired difference must satisfy `abs(a-b) <= max(0.20*a, 2 ms)`. +- Sort the ten actual owning invocations by validated start. First-five versus + last-five median drift must satisfy `abs(lastMedian-firstMedian) <= + max(0.10*firstMedian, 2 ms)`. +- The two five-sample role medians must also satisfy `abs(candidateMedian- + baselineMedian) <= max(0.10*baselineMedian, 2 ms)`. + +Require parseable owner start/duration, nonoverlap, registered baseline-first then +candidate-first alternation, distinct owning reports, companies and run sets. +A comparison must follow all calibration owners and reuse none of their evidence +or owned data. Synthetic oracle fixtures represent this chronology and freshness +explicitly; they do not establish that an application or provider ran. + +Retain all raw samples and owner chronology. Freeze the actual maximum paired +absolute difference AND actual maximum paired relative difference. The 2-ms +floor applies only to suitability and never replaces either observed envelope. +Comparisons recompute these envelopes and suitability diagnostics from retained +samples and reject altered envelopes, rules, flags or timestamps. Unsuitable A/A +exits unsuccessfully after writing its complete diagnostic report; retain all ten +arms and diagnose before a newly reviewed prospective observation. Do not inflate +caps or silently replace an arm. + +If the pooled ten-value median is at most20 ms, label that metric +representation-limited, suppress percentage-improvement claims and use its actual +frozen absolute noise for nonregression: `median(candidate-baseline) <= observed +absolute noise`. In particular, zero observed noise permits no positive regression +even when the suitability floor was2 ms. Longer metrics retain the existing +relative-noise nonregression rule. Loaded improvement still requires its existing +relative-noise median and at least4/5 individual wins; each counted win and the +median absolute improvement must additionally exceed2 ms before a numerical gain +is claimed. No sub-2ms, p99 or percentage gain is claimed for representation-limited +metrics. All original hard product/control deadlines remain unchanged. + +The floor describes integer-millisecond timestamp contrast representation only. +It is not a frame period, paint guarantee, clock-accuracy bound or a bound on the +settlement/Stop observer. The unchanged Playwright polls can wait1000 ms after a +callback and incur API/scheduler time as well; no blanket1–2second tolerance is +introduced. The blocked-Stop companion keeps its separate frozen100-ms policy. +The default H4e comparator and journey semantics remain unchanged. + +### Deferred diagnostic design: Playwright assertion cost only + +This diagnostic has **not run**. After root review and an explicit quiet-window +release, create an owned temporary standalone Playwright test/config outside the +measured source. Do not inherit a config with `webServer`. Request no `page`, +`browser` or `request` fixture; use one Node worker, no retries, trace/video off, +and a bounded 300-second diagnostic timeout. It must start no app, database, +provider or browser. Record the temporary source/config, Node and Playwright +versions, source hashes, owning test status, raw durations and all failures. + +Generate 1,000 deterministic synthetic normalized rows and a matching clock map +inside that test process. They are CPU-oracle inputs only: no provider output, +application event or persisted outcome is seeded. Use the same six checks for +both branches on the same objects: + +```ts +// Planned scratch test functions, not an executed measurement. +const checkPw = (event, original, index) => { + expect(event.payload.text).toBe(`[q7${index.toString(36)}] `); + expect(event.runId).toBe("diagnostic-run"); + expect(event.sourceInstanceId).toBeTruthy(); + expect(event.sourceEventId).toBeTruthy(); + expect(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0).toBe(true); + expect({ nonce: original.nonce, markerCode: original.markerCode, index: original.index }) + .toEqual({ nonce: "diagnostic-source", markerCode: "q7", index }); +}; +const checkNode = (event, original, index) => { + assert.equal(event.payload.text, `[q7${index.toString(36)}] `); + assert.equal(event.runId, "diagnostic-run"); + assert.ok(event.sourceInstanceId); + assert.ok(event.sourceEventId); + assert.ok(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0); + assert.deepEqual({ nonce: original.nonce, markerCode: original.markerCode, index: original.index }, + { nonce: "diagnostic-source", markerCode: "q7", index }); +}; +``` + +Warm each branch once on 100 inputs outside timing. Then record five alternating +pairs of all 1,000 inputs using monotonic `performance.now()` boundaries, retaining +each elapsed time and six-check count. Do not disable normal Playwright assertion +steps in its branch. Compare total completion duration and spread; keep diagnostic +profiles separate. This tests assertion instrumentation overhead, not application +ingestion throughput, browser responsiveness or acceptance. Full H5d qualification +and the fresh frozen fifty-arm protocol remain required regardless of its result. diff --git a/tests/e2e/fixtures/native-ingestion-h5.mjs b/tests/e2e/fixtures/native-ingestion-h5.mjs index 3ba362667bc..b2efeda1713 100644 --- a/tests/e2e/fixtures/native-ingestion-h5.mjs +++ b/tests/e2e/fixtures/native-ingestion-h5.mjs @@ -334,6 +334,79 @@ export function assertRecordedFailures(proof, item, companyId) { recordedErrorClasses: Object.fromEntries([...classes].sort(([a], [b]) => a.localeCompare(b)).map(([code, ids]) => [code, ids.size])) }; } +// H5d only changes post-observation oracle instrumentation. These functions +// check the same retained API rows and original provider clocks without making +// a Playwright step for every normalized delta. They never drive the SUT. +export function collectDurableRunEvents(rows, markerCode) { + const deltas = []; let resultAcceptedEvents = 0; + for (const row of rows) { + const event = row.payload?.prpEvent; + if (row.eventType === 'run.result.accepted' || event?.eventType === 'run.result.accepted') resultAcceptedEvents++; + if (event?.eventType !== 'item.delta') continue; + for (const match of String(event.payload?.text ?? '').matchAll(new RegExp(`\\[${markerCode}([a-z0-9]{1,3})\\]`, 'g'))) { + const index = Number.parseInt(match[1], 36); + deltas.push({ index, identity: `${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`, event }); + } + } + return { deltas, resultAcceptedEvents }; +} + +export function assertDurableRunEvents({ collected, runId, token, markerCode, events, stop, sourceClocks, visible, + requireNativeIdentity }) { + const { deltas, resultAcceptedEvents } = collected; + for (const [ordinal,delta] of deltas.entries()) { + const { index, event } = delta, context = `Durable run ${runId} source ${token} delta ${ordinal} marker ${markerCode}:${index}`; + assert.equal(event.payload.text, `[${markerCode}${index.toString(36)}] `, `${context}: complete normalized payload`); + if (requireNativeIdentity) { + assert.equal(event.runId, runId, `${context}: exact run identity`); + assert.ok(event.sourceInstanceId, `${context}: nonempty source instance identity`); + assert.ok(event.sourceEventId, `${context}: nonempty source event identity`); + assert.ok(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0, `${context}: positive integer source sequence`); + } + const original = sourceClocks.get(index); + assert.ok(original, `${context}: missing original provider clock`); + assert.deepEqual({ nonce: original.nonce, markerCode: original.markerCode, index: original.index }, + { nonce: token, markerCode, index }, `${context}: exact original emission clock join`); + } + const identities = new Set(deltas.map(delta => delta.identity)), indices = deltas.map(delta => delta.index); + assert.equal(identities.size, deltas.length, `Durable run ${runId}: duplicate normalized source identity tuple`); + assert.deepEqual(indices, Array.from({ length: stop ? deltas.length : events }, (_, index) => index), + `Durable run ${runId}: complete ordered ${stop ? 'canonical Stop prefix' : 'measured events'}`); + if (stop) { + const present = new Set(indices); + for (const item of visible) assert.ok(present.has(item.index), + `Durable run ${runId}: visible Stop marker ${item.index} must remain durable`); + } else assert.equal(sourceClocks.size, events, `Durable run ${runId}: complete source clock count`); + assert.equal(resultAcceptedEvents, stop ? 0 : 1, `Durable run ${runId}: exact accepted result count`); + return { runId, token, markerCode, individualDeltas: deltas.length, uniqueSourceIdentities: identities.size, + matchedSourceClocks: deltas.length, ordered: true, resultAcceptedEvents }; +} + +export function assertPreludeEvents({ rows, runId, token, expectedEvents }) { + const identities = new Set(); let count = 0; + for (const row of rows) { + const event = row.payload?.prpEvent; + if (event?.eventType !== 'item.delta' || !String(event.itemId ?? '').startsWith('prelude-')) continue; + assert.equal(event.payload?.text, count === expectedEvents - 1 ? ` PRELUDE_${token}_READY ` : '.', + `Prelude run ${runId} delta ${count}: complete normalized payload`); + count++; identities.add(`${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`); + } + assert.equal(count, expectedEvents, `Prelude run ${runId}: exact individually persisted delta count`); + assert.equal(identities.size, count, `Prelude run ${runId}: unique source identity tuples`); + return { individualDeltas: count, uniqueSourceIdentities: identities.size }; +} + +export function assertVisibleMarkerClocks({ seenMarkers, sourceClocks, markerCode, expectedEvents, stop }) { + const visible = seenMarkers.map(marker => { + const original = sourceClocks.get(marker.index); + assert.ok(original, `Missing original provider clock for visible ${marker.markerCode}:${marker.index}`); + return { ...original, ...marker }; // Retain the original two-frame paint timestamp. + }); + if (!stop) assert.equal(new Set(visible.map(item => item.index)).size, expectedEvents, + `Visible marker ${markerCode}: complete unique foreground count`); + return visible; +} + export function assessH5Arm(report, owner) { assert.equal(report.protocolVersion, H5_PROTOCOL); assert.equal(report.observationMode, H5_MODE); diff --git a/tests/e2e/native-event-ingestion-blocked-stop.journey.ts b/tests/e2e/native-event-ingestion-blocked-stop.journey.ts new file mode 100644 index 00000000000..7123b988199 --- /dev/null +++ b/tests/e2e/native-event-ingestion-blocked-stop.journey.ts @@ -0,0 +1,318 @@ +import { randomUUID, createHash } from "node:crypto"; +import { execFileSync } from "node:child_process"; +import { readFile } from "node:fs/promises"; +import path from "node:path"; +import { expect, test, type Request } from "@playwright/test"; +import { json } from "./agent-chat.shared"; +import { runtimeAuthority, processAbsent } from "./fixtures/native-ingestion-blocked-stop-authority"; +import { openClock } from "./fixtures/native-ingestion-blocked-stop-clock"; +import { ownedDatabase, cancellationSnapshot, blockers } from "./fixtures/native-ingestion-blocked-stop-db"; +import { PROTOCOL, WORKLOAD as W, WORKLOAD_SHA256, assertHeldSample, assertIntentAudit, isAppendLockQuery, metrics, assertFunctional, assertAuthoritativeRetirement, cleanupOwnedRunBeforeClockClose } + from "./fixtures/native-ingestion-blocked-stop-protocol.mjs"; +import { observeRunSettlement } from "./fixtures/native-ingestion-h5.mjs"; + +const delay = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); +const birth = (pid: number) => { + const value = execFileSync("ps", ["-p", String(pid), "-o", "pid=,pgid=,lstart="], { encoding: "utf8" }).trim(); + const match = value.match(/^(\d+)\s+(\d+)\s+(.+)$/); + if (!match || Number(match[1]) !== pid) throw new Error("Missing exact owned process birth identity"); + return { pid, group: Number(match[2]), birth: match[3] }; +}; + +test("existing-history Stop preserves its durable barrier behind a real blocked append", async ({ page, request }, info) => { + const appRoot = path.resolve(process.env.PAPERCLIP_INGESTION_APP_ROOT ?? ""); + expect(process.env.PAPERCLIP_INGESTION_SERVER_ARTIFACT).toBe("compiled"); + expect(process.env.PAPERCLIP_INGESTION_COMPANION_PROTOCOL).toBe(PROTOCOL); + const report: any = { protocol: PROTOCOL, workload: W, workloadSha256: WORKLOAD_SHA256, complete: false, + variant: process.env.PAPERCLIP_INGESTION_VARIANT, startedAtMs: Date.now(), timing: {}, heldSamples: [], observerSamples: [], + observedClock: "Independent observer wall clock; PostgreSQL row recordedAt is not a commit timestamp", + measurement: { pollIntervalMs: W.pollIntervalMs, releaseOrigin: "Owned COMMIT response acknowledgement", + releaseCommitBracket: "releaseRequestedAtMs..releaseCommitAckAtMs; exact server commit instant is unknown", + absenceAndAckUncertainty: "Last unsuccessful observation through successful query/process observation; polling plus query duration", + actionToAckIncludesForcedWait: true, noUniversalClickAckDeadline: true, + recording: { trace: info.project.use.trace, video: info.project.use.video }, quantitativeAcceptance: "Requires separately frozen companion A/A and five paired arms per comparison" } }; + const clock = await openClock(page); + const company = await json(await request.post("/api/companies", { data: { name: `Blocked Stop ${Date.now()}` } })); + report.companyId = company.id; + const original = await json(await request.get("/api/instance/settings/experimental")); + let holder: any, observer: any, held = false, companyRunId: string | undefined; + let stopRequest: Request | undefined, stopSettled = false; + const nonce = randomUUID().replaceAll("-", ""), key = randomUUID().replaceAll("-", ""), code = clock.nextMarkerCode(); + const stopFinished = (req: Request) => { + if (req === stopRequest) { stopSettled = true; report.timing.httpSettledAtMs = Date.now(); } + }; + page.on("requestfinished", stopFinished); + page.on("requestfailed", stopFinished); + try { + const files = ["native-event-ingestion-blocked-stop.journey.ts", "playwright-native-ingestion-blocked-stop.config.ts", + "fixtures/native-ingestion-blocked-stop-authority.ts", "fixtures/native-ingestion-blocked-stop-clock.ts", + "fixtures/native-ingestion-blocked-stop-db.ts", "fixtures/native-ingestion-blocked-stop-protocol.mjs", + "native-event-ingestion.journey.ts", "playwright-native-ingestion.config.ts", "fixtures/in-feed-codex.mjs", + "fixtures/native-ingestion-bin/codex", "fixtures/composer-stop-codex-loader.mjs", "fixtures/native-ingestion-bootstrap.mjs", + "fixtures/native-ingestion-remote-server.mjs", "fixtures/native-ingestion-protocol-qualification.mjs", "fixtures/native-ingestion-h5.mjs"]; + report.sourceHashes = Object.fromEntries(await Promise.all(files.map(async file => [file, + createHash("sha256").update(await readFile(path.join(import.meta.dirname, file))).digest("hex")]))); + const harnessRoot = path.resolve(import.meta.dirname, "../.."); + report.harnessProvenance = { + sourceRevision: execFileSync("git", ["rev-parse", "HEAD"], { cwd: harnessRoot, encoding: "utf8" }).trim(), + sourceStatus: execFileSync("git", ["status", "--porcelain", "--", "tests/e2e"], { cwd: harnessRoot, encoding: "utf8" }).trim(), + driverLockSha256: createHash("sha256").update(await readFile(path.join(harnessRoot, "pnpm-lock.yaml"))).digest("hex"), + inheritedH5cRevision: "e431ec63d41fe90c6fd68bd2fc2bb969feeca917", + }; + report.sourceRevision = execFileSync("git", ["rev-parse", "HEAD"], { cwd: appRoot, encoding: "utf8" }).trim(); + report.buildInfo = JSON.parse(await readFile(path.join(appRoot, "server/dist/build-info.json"), "utf8")); + expect(report.buildInfo.commit).toBe(report.sourceRevision); + report.serverEntrySha256 = createHash("sha256").update(await readFile(path.join(appRoot, "server/dist/index.js"))).digest("hex"); + report.health = await json(await request.get("/api/health")); + await json(await request.patch("/api/instance/settings/experimental", { data: { enableNativeRunner: true, enableWorktreeRunExecution: true } })); + await json(await request.post(`/api/companies/${company.id}/user-secret-definitions`, { data: { key: "OPENAI_API_KEY", name: "Fixture credential" } })); + await json(await request.post(`/api/companies/${company.id}/me/user-secrets`, { data: { definitionKey: "OPENAI_API_KEY", value: "not-a-real-model-key" } })); + const environments = await json(await request.get(`/api/companies/${company.id}/environments?driver=local`)); + expect(environments).toHaveLength(1); + const environment = environments[0]; expect(environment.driver).toBe("local"); + const agent = await json(await request.post(`/api/companies/${company.id}/agents`, { timeout: 120_000, data: { + name: "Blocked Stop local control", role: "engineer", adapterType: "paperclip_runner", defaultEnvironmentId: environment.id, + adapterConfig: { provider: "codex", model: "gpt-5.1-codex-mini", env: { + OPENAI_API_KEY: { type: "user_secret_ref", key: "OPENAI_API_KEY", version: "latest", required: true } } }, + runtimeConfig: { heartbeat: { enabled: false, wakeOnDemand: true, maxConcurrentRuns: 50 } }, + } })); + await expect.poll(async () => (await json(await request.get(`/api/agents/${agent.id}`))).lifecycleState, + { timeout: 60_000 }).toBe("ready"); + report.setup = { environmentId: environment.id, agentId: agent.id, + selection: "Normal local environment and control agent; existing deterministic Codex leaf command; no resolver override" }; + const text = ".".repeat(W.prelude - 1) + ` PRELUDE_${nonce}_READY ` + + Array.from({ length: W.events }, (_, index) => `[${code}${index.toString(36)}] `).join(""); + expect([...text].length).toBeLessThan(4000); + const state = { events: W.events, intervalMs: W.intervalMs, streams: new Map(), errors: [] } as any; + state.streams.set(nonce, { nonce, markerCode: code, opened: false, hello: false, ended: false, aborted: false, + partialBytes: 0, records: new Map(), expectedLauncher: "host-local" }); + clock.clocks.set(key, state); clock.pending.set(key, new Map()); + const prompt = `Measure native streaming for ${nonce.slice(0, 8)}.\n\nINGESTION_PERF nonce=${nonce} events=${W.events} intervalMs=${W.intervalMs} prelude=${W.prelude} markerCode=${code} gate=${clock.base}${key}`; + const before = new Set((await json(await request.get(`/api/companies/${company.id}/issues`))).map((row: any) => row.id)); + await page.goto(`/${company.issuePrefix}/agents/${agent.id}/overview`); + await page.getByRole("link", { name: "Tasks", exact: true }).click(); + await page.getByRole("button", { name: "New Task", exact: true }).last().click(); + const dialog = page.getByRole("dialog"); + await dialog.getByRole("textbox", { name: "editable markdown" }).fill(prompt); + await dialog.getByRole("button", { name: "Select assignee", exact: true }).click(); + await page.getByRole("listbox", { name: "Assignees", exact: true }).getByRole("option") + .filter({ has: page.getByText(agent.name, { exact: true }) }).click(); + await expect(dialog.getByRole("button", { name: "Select assignee" })).toContainText(agent.name); + report.setup.taskCreateActionAtMs = Date.now(); + await dialog.getByRole("button", { name: "Create task", exact: true }).click(); + let issue: any; + await expect.poll(async () => { + issue = (await json(await request.get(`/api/companies/${company.id}/issues`))) + .find((row: any) => !before.has(row.id) && row.description?.includes(nonce)); return Boolean(issue); + }).toBe(true); + report.issueId = issue.id; + await page.goto(`/${company.issuePrefix}/issues/${issue.identifier}`); + await expect.poll(() => clock.pending.get(key)!.has(nonce) && state.streams.get(nonce).hello, + { timeout: W.readinessMs, intervals: [W.pollIntervalMs] }).toBe(true); + await expect(page.getByText(`PRELUDE_${nonce}_READY`, { exact: false }).first()).toBeVisible({ timeout: W.streamMs }); + let run: any; + await expect.poll(async () => { + run = (await json(await request.get(`/api/companies/${company.id}/heartbeat-runs?limit=1000`))) + .find((row: any) => (row.issueId ?? row.nativeIssueId ?? row.contextSnapshot?.issueId ?? row.contextSnapshot?.taskId) === issue.id); + return Boolean(run); + }).toBe(true); + companyRunId = run.id; report.runId = run.id; + const readEvents = async () => { + const result: any[] = []; let afterSeq = 0; + for (;;) { + const rows = await json(await request.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=1000`)); + if (!rows.length) return result; + result.push(...rows); + const next = Math.max(...rows.map((row: any) => row.seq)); expect(next).toBeGreaterThan(afterSeq); afterSeq = next; + } + }; + const prelude = (await readEvents()).map(row => row.payload?.prpEvent) + .filter(event => event?.eventType === "item.delta" && String(event.itemId).startsWith("prelude-")); + expect(prelude).toHaveLength(W.prelude); + prelude.forEach((event, index) => { + expect(event.payload.text).toBe(index === W.prelude - 1 ? ` PRELUDE_${nonce}_READY ` : "."); + expect(event.sourceInstanceId).toBeTruthy(); expect(event.sourceEventId).toBeTruthy(); + expect(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0).toBe(true); + }); + expect(new Set(prelude.map(event => `${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`)).size).toBe(W.prelude); + report.prelude = { individualDeltas: prelude.length, uniqueIdentities: prelude.length, verifiedBeforeEpochAtMs: Date.now(), + method: "Normal event API read of actual emitted and committed deltas; no outcome prepopulation" }; + const authority = await runtimeAuthority({ appRoot, companyId: company.id, environmentId: environment.id, + issueIds: [issue.id], transport: "local" }); + const row = authority.rows[0], stream = state.streams.get(nonce); + expect(row.run_id).toBe(run.id); + expect(row.process_identity.processPid).toBe(row.process_pid); + expect(row.process_identity.processGroupId).toBe(row.process_group_id); + const provider = birth(stream.providerPid); + const processes = { runner: birth(row.process_pid), provider, group: row.process_group_id, providerGroup: provider.group }; + expect(processes.runner.group).toBe(processes.group); + expect(Number.isSafeInteger(processes.providerGroup) && processes.providerGroup > 1).toBe(true); + const alive = () => !processAbsent(processes.runner.pid) && !processAbsent(-processes.group) + && !processAbsent(processes.provider.pid) && !processAbsent(-processes.providerGroup); + expect(alive()).toBe(true); report.authorityBefore = authority; report.processesBefore = processes; + state.epochMs = Date.now() + 1000; report.epochMs = state.epochMs; + clock.pending.get(key)!.get(nonce)!.writeHead(200, { "Content-Type": "application/json" }).end(JSON.stringify({ epochMs: state.epochMs })); + const visible = () => page.evaluate(markerCode => (window as any).nativeIngestionVisible.filter((item: any) => item.markerCode === markerCode), code); + await expect.poll(async () => (await visible()).length, { timeout: W.streamMs }).toBeGreaterThanOrEqual(W.visibleBeforeLock); + report.visibleBeforeLock = await visible(); + holder = await ownedDatabase(appRoot, "blocked_stop_owned_lock"); + observer = await ownedDatabase(appRoot, "blocked_stop_independent_observer"); + await holder.unsafe("BEGIN"); held = true; + const owned = await holder`select id, company_id, pg_backend_pid() as holder_pid from heartbeat_runs + where company_id = ${company.id} and id = ${run.id} for update`; + expect(owned).toHaveLength(1); const holderPid = owned[0].holder_pid; + report.lock = { ...owned[0], acquiredAtMs: Date.now(), scope: "One owned company/run row; no persisted mutation" }; + await expect.poll(async () => { + const rows = await blockers(observer, holderPid); + report.appendBlockedBeforeStop = { observedAtMs: Date.now(), rows }; + return rows.some((candidate: any) => isAppendLockQuery(candidate.query)); + }, { timeout: 10_000, intervals: [W.pollIntervalMs] }).toBe(true); + const button = page.getByRole("button", { name: "Stop", exact: true }); await expect(button).toBeVisible(); + const stop = page.waitForRequest(req => { + const matched = req.method() === "POST" && new URL(req.url()).pathname === `/api/heartbeat-runs/${run.id}/cancel`; + if (matched) report.timing.requestAtMs = Date.now(); + return matched; + }); + report.timing.actionAtMs = Date.now(); + const click = button.click(); + stopRequest = await stop; await click; + const emissionsAtRequest = stream.records.size; + do { + const snapshot = await cancellationSnapshot(observer, company.id, run.id); + const sample = { ...snapshot, httpPending: !stopSettled, auditIds: snapshot.audits.map((audit: any) => audit.id), + processesAlive: alive(), processes: { runner: birth(processes.runner.pid), provider: birth(processes.provider.pid), + group: processes.group, providerGroup: processes.providerGroup, + runnerGroupAlive: !processAbsent(-processes.group), providerGroupAlive: !processAbsent(-processes.providerGroup) }, + clockOpen: !stream.ended && !stream.aborted, + emittedCount: stream.records.size, blockers: await blockers(observer, holderPid) }; + report.heldSamples.push(sample); assertHeldSample(sample, holderPid, processes); + if (Date.now() >= report.timing.requestAtMs + W.heldAfterActualRequestMs) break; + await delay(Math.min(W.pollIntervalMs, report.timing.requestAtMs + W.heldAfterActualRequestMs - Date.now())); + } while (true); + expect(stream.records.size, "Real provider continues emitting while transcript writes and Stop await the run lock").toBeGreaterThan(emissionsAtRequest); + report.timing.releaseRequestedAtMs = Date.now(); + await holder.unsafe("COMMIT"); held = false; report.timing.releaseCommitAckAtMs = Date.now(); + let acknowledged = false, absent = false; + const observationStartedAtMs = Date.now(); + const observe = async () => { + const snapshot = await cancellationSnapshot(observer, company.id, run.id); + if (snapshot.intent) { assertIntentAudit(snapshot); report.intentObserved ??= snapshot; } + if (!acknowledged) { + const value = snapshot.intent?.dispatchState === "acknowledged"; + report.observerSamples.push({ kind: "ack-query", startedAtMs: snapshot.queryStartedAtMs, + completedAtMs: snapshot.observedAtMs, durationMs: snapshot.observedAtMs - snapshot.queryStartedAtMs, value }); + if (value) { + assertIntentAudit(snapshot); acknowledged = true; report.timing.ackObservedAtMs = snapshot.observedAtMs; + report.ack = snapshot; report.timing.ackSuccessfulQueryStartedAtMs = snapshot.queryStartedAtMs; + } else report.timing.lastAckNotObservedAtMs = snapshot.observedAtMs; + } + if (!absent) { + const checkStartedAtMs = Date.now(); + const value = processAbsent(processes.runner.pid) && processAbsent(-processes.group) + && processAbsent(processes.provider.pid) && processAbsent(-processes.providerGroup); + const completedAtMs = Date.now(); + report.observerSamples.push({ kind: "process-check", startedAtMs: checkStartedAtMs, + completedAtMs, durationMs: completedAtMs - checkStartedAtMs, value }); + if (value) { + absent = true; report.timing.absenceObservedAtMs = completedAtMs; + report.timing.absenceSuccessfulCheckStartedAtMs = checkStartedAtMs; + report.processesAfter = { runnerAbsent: true, groupAbsent: true, providerAbsent: true, providerGroupAbsent: true, + group: processes.group, providerGroup: processes.providerGroup, checkedAtMs: completedAtMs }; + } else report.timing.lastProcessesPresentAtMs = completedAtMs; + } + }; + await observe(); + const settlement = await observeRunSettlement({ expectedRuns: 1, + readRuns: async () => { await observe(); return [await json(await request.get(`/api/heartbeat-runs/${run.id}`))]; }, + readRun: async () => json(await request.get(`/api/heartbeat-runs/${run.id}`)), + poll: (read: () => Promise, timeout: number) => expect.poll(read, { timeout, intervals: [W.pollIntervalMs] }).toBe(true), + onOutcome: (outcome: unknown) => { report.settlement = outcome; } }); + expect(settlement.runs[0].status).toBe("cancelled"); report.settledAtMs = Date.now(); + const sharedDetail = await json(await request.get(`/api/heartbeat-runs/${run.id}`)); + report.sharedSettlement = { runId: sharedDetail.id, companyId: sharedDetail.companyId, status: sharedDetail.status, + cancellation: sharedDetail.resultJson?.cancellation ?? null, + executionCancellation: sharedDetail.resultJson?.executionCancellation ?? null, + nativeCancellation: sharedDetail.resultJson?.nativeCancellation ?? null, observedAtMs: Date.now() }; + await expect.poll(async () => { await observe(); return acknowledged; }, + { timeout: W.acknowledgementAfterSettlementMs, intervals: [W.pollIntervalMs] }).toBe(true); + await expect.poll(async () => { await observe(); return absent; }, + { timeout: W.acknowledgementAfterSettlementMs, intervals: [W.pollIntervalMs] }).toBe(true); + report.timing.postReleaseObservationStartedAtMs = observationStartedAtMs; + await expect.poll(() => stopSettled, { timeout: W.acknowledgementAfterSettlementMs, + intervals: [W.pollIntervalMs] }).toBe(true); + const response = await stopRequest.response(); expect(response?.ok()).toBe(true); report.stopHttpStatus = response!.status(); + let lease: any; + await expect.poll(async () => { + lease = (await json(await request.get(`/api/environments/${environment.id}/leases`))) + .find((candidate: any) => candidate.heartbeatRunId === run.id); + return Boolean(lease?.releasedAt && ["released", "expired", "failed"].includes(lease.status) && lease.cleanupStatus !== "failed"); + }, { timeout: W.leaseAfterSettlementMs }).toBe(true); + expect(lease.provider).toBe("local"); expect(lease.providerLeaseId).toBeNull(); + report.lease = lease; + report.authorityAfter = await runtimeAuthority({ appRoot, companyId: company.id, environmentId: environment.id, + issueIds: [issue.id], transport: "local", afterStop: true }); + report.authoritativeRetirementState = assertAuthoritativeRetirement(report); + const seen = await visible(), events = await readEvents(); + const deltas = events.map(event => event.payload?.prpEvent).filter(event => event?.eventType === "item.delta" + && String(event.payload?.text).startsWith(`[${code}`)); + expect(deltas.length).toBeGreaterThanOrEqual(W.visibleBeforeLock); + deltas.forEach((event, index) => { + expect(event.payload.text).toBe(`[${code}${index.toString(36)}] `); expect(event.runId).toBe(run.id); + expect(event.sourceInstanceId).toBeTruthy(); expect(event.sourceEventId).toBeTruthy(); + expect(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0).toBe(true); + expect(stream.records.has(index)).toBe(true); + }); + expect(new Set(deltas.map(event => `${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`)).size).toBe(deltas.length); + seen.forEach((marker: any) => expect(marker.index).toBeLessThan(deltas.length)); + expect(events.filter(event => event.eventType === "run.result.accepted" || event.payload?.prpEvent?.eventType === "run.result.accepted")).toHaveLength(0); + await page.screenshot({ path: info.outputPath("blocked-stop-existing-history.png"), fullPage: true }); + await page.reload(); + await expect(page.getByRole("button", { name: "Stop", exact: true })).toHaveCount(0); + await expect(page.getByText(`[${code}${seen.at(-1).index.toString(36)}]`, { exact: false }).first()).toBeVisible(); + const transcriptRows = (rows: any[]) => rows.filter(row => row.payload?.prpEvent?.eventType === "item.delta") + .map(row => ({ id: row.id, seq: row.seq, event: row.payload.prpEvent })); + expect(transcriptRows(await readEvents())).toEqual(transcriptRows(events)); + await expect.poll(() => stream.ended || stream.aborted, { timeout: 30_000 }).toBe(true); + expect(state.errors).toEqual([]); + report.transcript = { individualDeltas: deltas.length, uniqueSourceIdentities: deltas.length, ordered: true, + persistedAfterReload: true, resultAcceptedEvents: 0, seen, deltas, preludeCount: prelude.length }; + report.clock = { ...stream, records: [...stream.records.values()] }; + report.complete = true; + } catch (error) { + report.failure = { message: String(error), stack: error instanceof Error ? error.stack : undefined, atMs: Date.now() }; + throw error; + } finally { + const cleanupErrors: unknown[] = []; + const cleanup = async (step: string, action: () => Promise) => { + try { await action(); } catch (error) { cleanupErrors.push({ step, error: String(error) }); } + }; + // Always release our exact owned lock before invoking any public cancellation cleanup. + if (held) await cleanup("rollback-owned-lock", async () => { await holder.unsafe("ROLLBACK"); held = false; }); + if (holder) await cleanup("close-holder", () => holder.end({ timeout: 5 })); + if (observer) await cleanup("close-observer", () => observer.end({ timeout: 5 })); + report.cleanupSettlement = {}; + await cleanup("settle-owned-run-before-clock-close", () => cleanupOwnedRunBeforeClockClose({ + companyId: company.id, runId: companyRunId, + readRun: companyRunId ? async () => json(await request.get(`/api/heartbeat-runs/${companyRunId}`)) : undefined, + cancelRun: async () => { const response = await request.post(`/api/heartbeat-runs/${companyRunId}/cancel`); + return { status: response.status(), run: await json(response) }; }, + closeClock: async () => { + for (const pending of clock.pending.values()) for (const response of pending.values()) if (!response.writableEnded) response.writeHead(503).end(); + clock.gate.closeAllConnections(); await new Promise(resolve => clock.gate.close(() => resolve())); + }, record: report.cleanupSettlement, + })); + await cleanup("restore-experimental-settings", async () => json(await request.patch("/api/instance/settings/experimental", { + data: { enableNativeRunner: original.enableNativeRunner, enableWorktreeRunExecution: original.enableWorktreeRunExecution } }))); + report.cleanup = { complete: cleanupErrors.length === 0, errors: cleanupErrors }; + report.complete &&= report.cleanup.complete; report.finishedAtMs = Date.now(); + let validationError: unknown; + if (report.complete) { + try { report.metrics = metrics(report); assertFunctional(report); } + catch (error) { validationError = error; report.complete = false; report.validationFailure = String(error); } + } + await info.attach("native-ingestion-blocked-stop-evidence", { body: JSON.stringify(report, null, 2), contentType: "application/json" }); + if (validationError) throw validationError; + if (!report.failure) expect(cleanupErrors).toEqual([]); + page.off("requestfinished", stopFinished); page.off("requestfailed", stopFinished); + } +}); diff --git a/tests/e2e/native-event-ingestion.journey.ts b/tests/e2e/native-event-ingestion.journey.ts index bcb49eedda3..d30e5efad1d 100644 --- a/tests/e2e/native-event-ingestion.journey.ts +++ b/tests/e2e/native-event-ingestion.journey.ts @@ -8,7 +8,8 @@ import path from "node:path"; import { expect, test } from "@playwright/test"; import { json } from "./agent-chat.shared"; import { h5Enabled, H5_MODE, H5_PROTOCOL, H4E_PROTOCOL, H5_TOPOLOGY, H5_LIMITS, H5_WORKLOAD_SHA256, ORIGINAL_BUDGETS, - observeStreaming, observeRunSettlement, originalTerminal, prepareParticipantAgents, assertParticipantTopology, assertRecordedFailures } from "./fixtures/native-ingestion-h5.mjs"; + observeStreaming, observeRunSettlement, originalTerminal, prepareParticipantAgents, assertParticipantTopology, assertRecordedFailures, collectDurableRunEvents, assertDurableRunEvents, + assertPreludeEvents, assertVisibleMarkerClocks } from "./fixtures/native-ingestion-h5.mjs"; type SeenMarker = { markerCode: string; index: number; visibleMs: number }; type Emission = { nonce: string; markerCode: string; index: number; intendedMs: number; emittedMs: number; receivedAtMs: number }; @@ -511,23 +512,18 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy const listed = await json(await request.get(`/api/companies/${company.id}/heartbeat-runs?limit=1000`)); const foreground = listed.find((run: any) => (run.issueId ?? run.nativeIssueId ?? run.contextSnapshot?.issueId ?? run.contextSnapshot?.taskId) === issue.id); expect(foreground).toBeTruthy(); - const identities = new Set(); let count = 0, afterSeq = 0; + const preludeRows: any[] = []; let afterSeq = 0; while (true) { const rows = await json(await request.get(`/api/heartbeat-runs/${foreground.id}/events?afterSeq=${afterSeq}&limit=1000`)); if (!rows.length) break; - for (const row of rows) { - const event = row.payload?.prpEvent; - if (event?.eventType !== "item.delta" || !String(event.itemId ?? "").startsWith("prelude-")) continue; - expect(event.payload?.text).toBe(count === workload.prelude - 1 ? ` PRELUDE_${nonce}_READY ` : "."); - count++; identities.add(`${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`); - } + preludeRows.push(...rows); const next = Math.max(...rows.map((row: { seq: number }) => row.seq)); expect(next).toBeGreaterThan(afterSeq); afterSeq = next; } - preludeProof = { runId: foreground.id, individualDeltas: count, uniqueSourceIdentities: identities.size, + const checked = assertPreludeEvents({ rows: preludeRows, runId: foreground.id, token: nonce, expectedEvents: workload.prelude }); + preludeProof = { runId: foreground.id, ...checked, retainedGlobalHead: afterSeq, verifiedAtMs: Date.now(), method: "Independent normal event API read before common emission gate release" }; active.preludeProof = preludeProof; - expect(count).toBe(workload.prelude); expect(identities.size).toBe(count); } const controllerAuthority = await runtimeAuthority({ appRoot, companyId: company.id, environmentId: caseEnvironment.id, issueIds, transport: caseTransport, @@ -645,12 +641,8 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy ? seenMarkers.every(marker => state.streams.get(nonce)!.records.has(marker.index)) : stream.records.size === workload.events && stream.ended)), { timeout: 30_000 }).toBe(true); expect(state.errors).toEqual([]); - const visible: Visible[] = seenMarkers.map(marker => { - const original = state.streams.get(nonce)!.records.get(marker.index); - expect(original, `Missing original provider clock for visible ${marker.markerCode}:${marker.index}`).toBeTruthy(); - return { ...original!, ...marker }; // Keep the original paint timestamp; never restamp after clock arrival. - }); - if (!stop) expect(new Set(visible.map(item => item.index)).size).toBe(workload.events); + const visible: Visible[] = assertVisibleMarkerClocks({ seenMarkers, sourceClocks: state.streams.get(nonce)!.records, + markerCode, expectedEvents: workload.events, stop }); // Terminal drainage has finished. Derive paint gaps from the retained // observer timestamps without adding work to the streaming observer. const paintTimes = [...new Set(visible.map(item => item.visibleMs))].sort((a, b) => a - b); @@ -693,39 +685,23 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy const token = issuedTokens.get(taskId)!; const code = state.streams.get(token)!.markerCode; expect(token).toBeTruthy(); - const deltas: Array<{ index: number; identity: string }> = []; - let afterSeq = 0, resultAcceptedEvents = 0; + const persistedRows: any[] = []; + let afterSeq = 0; + active.verifyingRun = { runId: run.id, nonce: token, markerCode: code, afterSeq, rows: 0 }; while (true) { const rows = await json(await request.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=1000`)); if (!rows.length) break; - for (const row of rows) { - const event = row.payload?.prpEvent; - if (row.eventType === "run.result.accepted" || event?.eventType === "run.result.accepted") resultAcceptedEvents++; - if (event?.eventType !== "item.delta") continue; - for (const match of String(event.payload?.text ?? "").matchAll(new RegExp(`\\[${code}([a-z0-9]{1,3})\\]`, "g"))) { - const index = Number.parseInt(match[1], 36); - expect(event.payload.text, "Every durable normalized delta retains its complete compact payload").toBe(`[${code}${index.toString(36)}] `); - if (h5) { expect(event.runId).toBe(run.id); expect(event.sourceInstanceId).toBeTruthy(); - expect(event.sourceEventId).toBeTruthy(); expect(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0).toBe(true); } - deltas.push({ index, identity: `${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}` }); - } - } + persistedRows.push(...rows); const next = Math.max(...rows.map((row: { seq: number }) => row.seq)); expect(next).toBeGreaterThan(afterSeq); afterSeq = next; + active.verifyingRun = { runId: run.id, nonce: token, markerCode: code, afterSeq, rows: persistedRows.length }; } - active.latestDurableRead = { runId: run.id, nonce: token, markerCode: code, deltas }; - await expect.poll(() => deltas.every(delta => state.streams.get(token)!.records.has(delta.index)), { timeout: 30_000 }).toBe(true); - for (const delta of deltas) { - const original = state.streams.get(token)!.records.get(delta.index)!; - expect({ nonce: original.nonce, markerCode: original.markerCode, index: original.index }) - .toEqual({ nonce: token, markerCode: code, index: delta.index }); - } - expect(new Set(deltas.map(item => item.identity)).size).toBe(deltas.length); - expect(deltas.map(item => item.index)).toEqual(Array.from({ length: stop ? deltas.length : workload.events }, (_, index) => index)); - if (stop) for (const item of visible) expect(deltas.some(delta => delta.index === item.index)).toBe(true); - expect(resultAcceptedEvents).toBe(stop ? 0 : 1); - durable.push({ runId: run.id, token, markerCode: code, individualDeltas: deltas.length, uniqueSourceIdentities: deltas.length, - matchedSourceClocks: deltas.length, ordered: true, resultAcceptedEvents }); + const collected = collectDurableRunEvents(persistedRows, code); + active.latestDurableRead = { runId: run.id, nonce: token, markerCode: code, + deltas: collected.deltas.map(({ index, identity }) => ({ index, identity })) }; + await expect.poll(() => collected.deltas.every(delta => state.streams.get(token)!.records.has(delta.index)), { timeout: 30_000 }).toBe(true); + durable.push(assertDurableRunEvents({ collected, runId: run.id, token, markerCode: code, events: workload.events, stop, + sourceClocks: state.streams.get(token)!.records, visible, requireNativeIdentity: h5 })); } await expect.poll(() => [...state.streams.values()].every(stream => stream.ended || stream.aborted), { timeout: 30_000 }).toBe(true); expect(state.errors).toEqual([]); diff --git a/tests/e2e/playwright-native-ingestion-blocked-stop.config.ts b/tests/e2e/playwright-native-ingestion-blocked-stop.config.ts new file mode 100644 index 00000000000..3446dbe0af1 --- /dev/null +++ b/tests/e2e/playwright-native-ingestion-blocked-stop.config.ts @@ -0,0 +1,5 @@ +import { defineConfig } from "@playwright/test"; +import base from "./playwright-native-ingestion.config"; +process.env.PAPERCLIP_INGESTION_COMPANION_PROTOCOL = "blocked-stop-history-v1"; +export default defineConfig({ ...base, testMatch: "native-event-ingestion-blocked-stop.journey.ts", timeout: 900_000, + retries: 0, workers: 1, use: { ...base.use, trace: "off", video: "off" } });