Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-11 05:11:49 +01:00
Code Issues Packages Projects Releases Wiki Activity

test(native-ingestion): retain qualified browser and Stop fixtures

Browse Source
This commit is contained in:
Dotta committed 2026-11-04 04:00:59 -06:00
1 parent 6757908c19
commit eed0a08e17
29 files changed
+4318 -78
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

doc/benchmarks/2026-11-04-native-event-ingestion-application.md
+294
View File
@@ -0,0 +1,294 @@
# Native ingestion application acceptance
Status at 2026-11-04 10:00 UTC: **application acceptance remains open**. Source
append measurements passed their registered gates. No production-application
A/A calibration, paired browser improvement, or deployed improvement is claimed.
Both delivery PRs remain drafts.
This report records the application inputs and qualification evidence. The
[source report](2026-11-03-native-event-ingestion-interleaved.md) measures active
append-service work; it does not establish application burst drainage or browser
latency. The dependent batching PR owns incremental and combined results.
## Matched application builds
All three inputs normally merge upstream
`4a83ba4050f6e6d334fe4cb6e3ace050097530ce`. They use independent normal locked
application installs and production builds. The browser driver has a separate
frozen install; the application checkouts' test files are not the executed driver.
| Arm | Immutable application revision | Role |
| --- | --- | --- |
| B0 | `bac53ec3bfeaf6546e7aaeaee516487b30534654` | Matched baseline |
| B1 | `07061a601cdc20603c8c7a46f9ddbad7ac5f5939` | Cursor change |
| B2 base | `ca7cba171cd5198ed569d75012edc858041cbc44` | Unoptimized batching base |
All three normal builds exited zero. Their shared native inputs have Rust tree
`bbe29bba574ce8d5cfddf2f7461ae0eb5983df07` and binary SHA256
`abb65bdc073991e428bd37f9809603ee7564543cca0ed0305dc231137c93070f`.
Native target reuse was limited to the same 15 native build inputs. Each
application has its own Node dependencies and compiled output.
The qualified-build receipt SHA256 is
`a1259ba760f3cbbe44817e333f9f620bcf3769fb0a6c16f9ce2f0f5b2ed60d94`.
The inventory-summary SHA256 is
`d3766fc4206d569d10fd02224433f1e9a2caa6992a7c62216e80af368aa4e44d`.
Inventories cover `server/dist`, `ui/dist`, `packages/db/src`, and
`packages/shared/src`: 12,884 / 12,898 / 12,906 files respectively. Every measured
arm must validate its actual files, preferred-UI-shadow absence, native binary,
clean source and compiled revision against the qualified inventory again.
Source/build-input comparisons match the frozen B1/B2 to the delivery branches;
this excludes test/tooling differences rather than executing the delivery lock
as the browser driver.
These are build qualifications, not performance results. A further Rust change
needs its own immutable revision, normal build, binary and qualified inventory.
The existing B2 base cannot stand in for that candidate.
## Retained full-load qualification failures
Earlier H4c application attempts missed their original streaming deadline;
[their report](2026-11-03-native-event-ingestion-h4c.md) remains retained. Later
qualification failures are also excluded from paired performance samples.
| Driver registration | Application baseline | Workload authority | Original 120s streaming observation | Eventual foreground completion | Final observation before teardown |
| --- | --- | --- | --- | --- | --- |
| H5a, `7388e24d` | Historical `a36567f9` | Two agents, 50 runs each | 54 of 512 foreground markers | 512 at 296.555s from poll start | 86 succeeded; 14 retryable failures |
| H5b, `179fe6fa6da1b24ca08c1162301538d3e3788af6` | Matched `bac53ec3` | 100 independent agents, one run each | 61 of 512 foreground markers | 512 at 282.734s from poll start | 83 succeeded; 17 retryable failures |
Both attempts independently verified all 51,200 durable deltas against original
emission identities, payloads and clocks, and 100 accepted results. Those counts
do not establish final settlement or lease cleanup. Existing-history and Stop
cases were not reached.
H5a's 14 failures encountered 30-second merge locks on the two shared agent
directories. The later independent-agent experiment does not establish a product
fix for that topology. A separate PostgreSQL deadlock on a succeeded run does
not establish the failed runs' wait cycle.
H5b's 17 failures were different: their logs recorded suspension but no timely
provider-drain receipt after the five-second wait. The reason the receipts were
late remains unproved. The observer treated failed native `retryable_failure`
runs as terminal and stopped the application before their first supported retry.
The first retry was due at 05:40:32 UTC; teardown occurred at 05:40:26 UTC.
The global scheduler was enabled. Eligibility and recovery execution were not
observed, so this failure does not establish either recovery success or failure.
The complete raw attempts, stopped-database read-only checks and source logs
remain sealed privately. Public summaries include only the fields above; raw
process, configuration and provider logs are not wholesale publication inputs.
## H5c qualification
The frozen driver is
[`e431ec63d41fe90c6fd68bd2fc2bb969feeca917`](https://github.com/paperclipai/paperclip/tree/e431ec63d41fe90c6fd68bd2fc2bb969feeca917),
tag `native-ingestion-h5c`. Its nine-file measurement SHA256 is
`2265aabe9c7344f5d4b9c8b08a5c6bbd5571a6e2de9ae7b72a8d1733250e609f`.
The [registered protocol](../../tests/e2e/fixtures/native-ingestion-h5.md) and
[exact execution helpers](../../scripts/native-ingestion/h5c/README.md) provide
reproduction commands, dependency provenance, bounded resource ownership and
pass/fail gates.
H5c observes retryable finalization inside the same single 180-second terminal
window. It records observed transitions and current/retained recovery errors.
Final success, complete events/results, and lease cleanup remain mandatory.
The retained recovery marker contains the last failure; it is not an exhaustive
history of every retry. Comparisons reject new recorded error classes and retry
rates outside freshly frozen baseline noise. No manual retry or new provider
turn is introduced.
The original 120-second streaming and 900-second invocation limits remain.
Extended observation preserves original deadline misses as failures. The final
candidate must pass every original limit in every arm and later pass a separate
default-mode original-SLA invocation with the same source and qualified build.
The SMALL B0 functional qualification passed one Playwright attempt without
retry: 49.6 seconds including hooks, 33.0 seconds in the test body. Three remote
fixture runs retained 48 deltas with unique identities and original clocks,
three accepted results, reload persistence, and successful lease cleanup. A
separate normal local Stop run retained eight canonical deltas, zero accepted
results, native cancellation acknowledgement, independent runner/group/provider
absence and a released lease. Both cases recorded zero retries. This small case
does not exercise recovery under the 100-run load and does not qualify real
cloud Stop.
The small report SHA256 is
`1c1ec8a0163e09fd75cabf3616374e8b185165a23c425f57794840c72b40782c`;
the 25-file seal manifest is
`6db1d2d41375a965fefa266a1d2bd47ddf9ad6f8d07773d7bde8a7bb8786b959`;
the sealed archive is
`50837389f0d2737a7f7ec249e75760da0aee366eeea881a9ce58f99025f20f83`.
Independent lead review verified those bytes, the owning attempt's attached
report, original clocks, recorded-failure proofs and external process/port
cleanup. The legacy raw `performanceAcceptanceEligible` flag denotes an
unprofiled run; `qualificationOnly: true` is independently rejected by the frozen
performance assessor. No flag or raw report was rewritten to change eligibility.
## H5c full failure and independently verified persistence
Full H5c B0 qualification failed at its unchanged 1,200-second observation
limit during independent durable-event verification. The foreground showed
66 of 512 markers near the original 120-second deadline and all 512 at
283.391 seconds after the epoch. All 100 runs succeeded within 75.926 seconds
of the original single 180-second terminal window. Fourteen transient provider
transport failures recovered; no nonretryable failure remained.
Independent lead review verified the owning report, 56 raw files and eight
read-only stopped-PostgreSQL supplement files. All 51,200 durable deltas match
the original emission identities, payloads, ordering and clocks; all 100 results
were accepted, finalizers committed and leases released with successful cleanup.
This does not establish that those events committed before the original
streaming deadline. The owning Playwright attempt timed out after verifying
only 33 of 100 runs through its API oracle. Existing-history and Stop cases
were not reached. API cleanup encountered a disposed request context; separate
physical-quiet and port checks passed. The attempt remains failed and ineligible.
The failed report SHA256 is
`4946558e5a04377650661cbe2c0d5d45b755ef77f5064399e4df8889c3f7b595`;
the raw seal manifest is
`2adb8afed5a10c6dc2091d7946a2e2d3bfc8ad6677613b46c20b1190d5ce47c8`.
## H5d prospective correction
The separately frozen driver is
[`2df9897e1ce2ca90be29fa097d776c3766e217ad`](https://github.com/paperclipai/paperclip/tree/2df9897e1ce2ca90be29fa097d776c3766e217ad),
tag `native-ingestion-h5d`. Its nine-file measurement SHA256 is
`e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4`;
the comparator SHA256 is
`09daef50fad5eef4dbda4e63a191749d0d167d3ae9d38bfd22df8721e2aac4a3`.
Plain assertions replace individual Playwright steps in bulk readback while
preserving payload, identity, original-clock, ordering, result and Stop-prefix
checks. Workloads, browser observation, APIs and deadlines stay fixed. A
separate no-application diagnostic measures oracle cost; it establishes no
product performance gain. H5d has now completed its own small and full functional observation. Fresh
A/A calibration remains separate; qualification attempts are not reused.
Before observing A/A data, the comparator registers suitability limits for
paired difference and chronological/role drift. It freezes actual absolute and
relative noise; its two-millisecond timestamp representation floor never pads
a regression allowance. Short measurements cannot support subfloor, percentage
speedup or p99 claims. Fully verified completion now has its own quantitative
gate: faster browser delivery or terminal status cannot hide slower final
drainage, missing readback or unclosed leases. Its conservative completion rate
includes verification, reload and lease observation; it is not the exact last
commit time. All comparison arms must be fresh, and every final-candidate arm
must meet the original limits as well as a later same-source original-SLA run.
## H5d qualification and calibration status
SMALL passed one owning attempt: 48 normal deltas across three remote fixture
runs, three accepted results, reload persistence, and a separate local Stop with
nine canonical deltas, zero results, acknowledged cancellation, Runner PID/group
and provider PID absence, a bound server retirement receipt and released lease.
Lead review verified all 40 sealed files and the exact owning attachment.
FULL passed one owning test attempt without retry in 853.339 seconds; the whole
command took 870.711 seconds. Independent readback completed all six cases:
| Case | Measured deltas | Accepted results | Fully verified drainage from epoch |
| --- | ---: | ---: | ---: |
| Paced single stream | 128 | 1 | 8.662s |
| 16-event burst, one run | 16 | 1 | 6.589s |
| 128-event burst, ten runs | 1,280 | 10 | 15.287s |
| 512-event burst, 100 runs | 51,200 | 100 | 402.584s |
| Foreground history under ten concurrent streams | 1,280 | 10 | 17.088s |
| Local Stop | 9 | 0 | 3.948s |
The history case is **one foreground run with 1,000 persisted prelude events
under nine fresh concurrent streams**. The frozen workload explicitly gives
background streams no prelude. A normal event-API read verified the foreground's
1,000 unique prelude events and retained global head 1062 before the common
epoch. It does not establish that all ten runs have 1,000-event histories.
The baseline still failed the original loaded-streaming limit: 41 of 512
foreground markers were visible at the last observation near 120 seconds.
Foreground completion took 287.750 seconds; terminal settlement then took
103.326 seconds, followed by 12.483 seconds of verification. Forty-three
recorded provider transport failures recovered; all 100 runs succeeded and their
leases closed successfully. The original 900-second owning-test gate passed.
This is complete functional observation with a retained responsiveness failure,
not passing application performance acceptance.
The full report SHA256 is
`d12ad9128d6a30bc335a69761d6004b2b3618fc275a07a84ef5434cfdb00b1e1`.
Lead review verified 38 sealed files, exact owning attachment, all source-clock
prefix joins, counts, results and leases, and reran the frozen assessment on
those bytes. An independent postreader initially assumed the foreground was the
first element of an API run list. The actual foreground was element 9. The
preserved failure was corrected in a separate offline reader by joining the
foreground token, durable proof, provider clock, activation and authority row;
event-count and timing checks remain. The owning attempt, raw seal, measurement
source and workload were unchanged. No failed application attempt was regraded.
A separate no-application diagnostic confirmed assertion-framework overhead for
synthetic 1,000-event inputs: five samples each gave medians of 3,859.190ms for
the old Playwright assertions and 1.954ms for plain assertions. Playwright
samples increased chronologically from 1,843.546 to 5,655.339ms; plain assertions
ranged from 1.363 to 3.017ms. This is neither a stable product-cost distribution
nor an exact attribution of the earlier full-run timeout.
Fresh B0 A/A began at 09:30 UTC. Two of ten invocations have completed and the
third is running. No noise envelope or paired application gain has passed.
## Blocked-write Stop qualification
The separate companion emits its own 1,000-event prelude through normal
ingestion, confirms a real append blocked by an owned run-row lock, then presses
Stop through the UI. During a fixed hold after the actual HTTP request it
requires no committed cancellation intent or audit, a pending response, ongoing
emissions and both birth-identified process groups alive. Public Stop must
commit its own intent under the same run lock; postrelease measurements retain
that durable boundary rather than treating a blocked intent as dispatch delay.
Two qualification failures remain excluded. V1 failed before the measured
epoch because it incorrectly assumed the provider shared the Runner process
group. Its cleanup left live processes; separately recorded manual cleanup
retired those owned processes. Manual cleanup is not successful Stop evidence.
V2 measured the held-write boundary, its own acknowledged intent and audit
records, and physical exit of both independent groups. It then failed an
unsupported unconditional `native.local_process_stopped` assertion. Transcript,
reload and lease checks after that assertion were not reached. Normal cleanup,
port and strict physical-quiet checks passed without manual signals.
The product supports two termination-proof states: retained original process
identities with verified physical absence, or cleared identities with a
server-authored retirement event bound to the latest launch/identity. The
retirement event is authoritative evidence for identity clearing; immediate
cancellation acknowledgement does not universally require it. The separately frozen
v3 oracle checks this distinction in a coherent authority snapshot after the
existing lease poll, without adding a wait budget. It also retains the own-intent,
audit, physical-group, lease, transcript and reload requirements. V2 cannot be
regraded or reused as an accepted arm.
V2 source is `6e5349cdbd5e3a197877f1082f84a17e866c03e7`; its failed report
SHA256 is `0ab6c383a07c57f5eb16f226d0e458c70b6d6800e3d9dce051e3d2ca53bba20e`
and 42-file manifest is
`f81168c2d0c2218d658e1c6422b074fe80d10de9d88ad752c0b04fc54b1cde65`.
Independent lead review verified those bytes and the owning attempt's attachment.
Those two failed trials remain excluded. V3 at
`bc33f09cae26404352848169d92a20c89480a5f5` subsequently passed one independent
qualification. Fifteen held samples covered 1,010ms after the actual Stop POST
with no committed intent/audit, response pending, and both original groups
alive. After release it verified the same own native intent and audits, both
identified PIDs and both independent process groups absent, cleared mutable
identity backed by the latest bound server receipt, and the exact local lease
released without failed cleanup. The 1,000-event prelude, twelve canonical Stop
deltas, zero results and persisted reload were verified. API cleanup and strict
physical quiet passed without manual signals.
Lead review independently verified all 49 files, the ten arm-artifact seals,
the exact owning Playwright attachment, and the frozen functional oracle. V3 raw
report SHA256 is
`66ebb86acfb5de5a3140ec4a5db104775265fc296c8e831d2964d065359a739b`.
This qualifies that functional path; control-latency acceptance still needs its
own fresh baseline noise and paired comparisons, distinct from the main matrix.
## Remaining acceptance
No application A/A or paired browser improvement has passed. No proposed Rust
optimization is implemented. Remaining gates are fresh B0/B1 application and companion noise; all three paired comparisons; loaded browser
and complete-drainage improvement; sparse/control nonregression; complete-window
resource accounting; original-SLA confirmation; affected live-provider and real
cloud Stop journeys on the final runtime; and current-head checks/review.
Local evidence establishes no deployed improvement.
scripts/compare-native-ingestion-browser.ts
+140 -7
View File
@@ -107,6 +107,10 @@ const workloads = [
];
const names = workloads.map(item => item.name);
const median = (values: number[]) => [...values].sort((a, b) => a - b)[Math.floor(values.length / 2)]!;
const pooledMedian = (values: number[]) => {
const sorted = [...values].sort((a, b) => a - b), middle = Math.floor(sorted.length / 2);
return sorted.length % 2 ? sorted[middle]! : (sorted[middle - 1]! + sorted[middle]!) / 2;
};
const first = pairs[0];
const sha256 = (value: unknown) => typeof value === "string" && /^[a-f0-9]{64}$/.test(value);
const uuid = (value: unknown) => typeof value === "string" && /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i.test(value);
@@ -463,27 +467,112 @@ if (h5 && phase === "compare" && candidatePolicy === "final-b2") {
}
}
const metric = (item: any, key: string): number => key.split(".").reduce((value, part) => value[part], item);
// Prospective main-campaign rule. The 2ms floor is timestamp contrast
// representation only: it never pads observed noise or a regression gate.
const browserAaRule = Object.freeze({ pairCount: 5, pairedFraction: 0.20, medianDriftFraction: 0.10,
contrastRepresentationFloorMs: 2, representationLimitedMedianMs: 20 });
function assertArmChronology(rows: any[]) {
assert.equal(rows?.length, 10, "Browser A/A/comparison requires ten actual owning invocations");
const sorted = [...rows].sort((a,b) => a.startedAtMs - b.startedAtMs);
const companies = new Set(), runIds = new Set(), owners = new Set();
for (const [index, row] of sorted.entries()) {
assert.ok(Number.isSafeInteger(row.startedAtMs) && row.startedAtMs > 0 && Number.isFinite(row.durationMs)
&& row.durationMs > 0, "Missing finite actual owning timestamp/duration");
assert.equal(row.finishedAtMs, row.startedAtMs + row.durationMs);
if (index) assert.ok(sorted[index - 1].finishedAtMs <= row.startedAtMs, "Browser owning invocations overlap");
assert.ok(sha256(row.ownerReportSha256) && !owners.has(row.ownerReportSha256), "Reused owning report"); owners.add(row.ownerReportSha256);
assert.ok(uuid(row.companyId) && !companies.has(row.companyId), "Each browser arm needs its fresh owned company"); companies.add(row.companyId);
assert.ok(Array.isArray(row.runIds) && row.runIds.length > 0);
for (const runId of row.runIds) {
assert.ok(uuid(runId) && !runIds.has(runId), "Browser arms reuse an owned run"); runIds.add(runId);
}
const pairIndex = Math.floor(index / 2), firstArm = pairIndex % 2 ? "candidate" : "baseline";
assert.equal(row.pairIndex, pairIndex, "Owner chronology must match registered pair order");
assert.equal(row.arm, index % 2 ? firstArm === "baseline" ? "candidate" : "baseline" : firstArm,
"Browser pairs must alternate baseline-first then candidate-first");
}
return sorted;
}
const armChronology = h5 ? assertArmChronology(pairs.flatMap((pair, pairIndex) => (["baseline", "candidate"] as const).map(arm => {
const report = pair[arm], owner = report.owningPlaywright, startedAtMs = Date.parse(owner.startedAt);
return { pairIndex, arm, startedAtMs, durationMs: owner.durationMs, finishedAtMs: startedAtMs + owner.durationMs,
ownerReportSha256: owner.reportSha256, companyId: report.companyId, runIds: report.cases.flatMap((item: any) => item.runIds) };
}))) : undefined;
function assessAaMetric(samples: any[], chronology: any[]) {
assert.equal(samples.length, 5);
for (const sample of samples) {
assert.ok(Number.isFinite(sample.baseline) && sample.baseline > 0 && Number.isFinite(sample.candidate) && sample.candidate >= 0);
assert.equal(sample.improvementMs, sample.baseline - sample.candidate, "A/A paired absolute contrast was forged");
assert.equal(sample.improvementRelative, sample.improvementMs / sample.baseline, "A/A paired relative contrast was forged");
}
const pairedDifferencesMs = samples.map(sample => Math.abs(sample.improvementMs));
const pairedLimitsMs = samples.map(sample => Math.max(browserAaRule.pairedFraction * sample.baseline, browserAaRule.contrastRepresentationFloorMs));
const baselineMedianMs = median(samples.map(sample => sample.baseline)), candidateMedianMs = median(samples.map(sample => sample.candidate));
const chronologicalValues = chronology.map(row => samples[row.pairIndex][row.arm]);
const firstFiveMedianMs = median(chronologicalValues.slice(0,5)), lastFiveMedianMs = median(chronologicalValues.slice(5));
const chronologicalMedianDriftMs = Math.abs(lastFiveMedianMs - firstFiveMedianMs);
const chronologicalDriftLimitMs = Math.max(browserAaRule.medianDriftFraction * firstFiveMedianMs, browserAaRule.contrastRepresentationFloorMs);
const roleMedianDriftMs = Math.abs(candidateMedianMs - baselineMedianMs);
const roleDriftLimitMs = Math.max(browserAaRule.medianDriftFraction * baselineMedianMs, browserAaRule.contrastRepresentationFloorMs);
const pooledMedianMs = pooledMedian(samples.flatMap(sample => [sample.baseline, sample.candidate]));
return { sampleCount: 10, pairCount: 5, pairedDifferencesMs, pairedLimitsMs, baselineMedianMs, candidateMedianMs,
pooledMedianMs, firstFiveMedianMs, lastFiveMedianMs, chronologicalMedianDriftMs, chronologicalDriftLimitMs,
roleMedianDriftMs, roleDriftLimitMs, representationLimited: pooledMedianMs <= browserAaRule.representationLimitedMedianMs,
observedNoiseAbsoluteMs: Math.max(...pairedDifferencesMs),
observedNoiseRelative: Math.max(...samples.map(sample => Math.abs(sample.improvementRelative))),
passed: pairedDifferencesMs.every((value,index) => value <= pairedLimitsMs[index])
&& chronologicalMedianDriftMs <= chronologicalDriftLimitMs && roleMedianDriftMs <= roleDriftLimitMs };
}
const measurements = names.flatMap(name => {
const observed = first.baseline.cases.find((item: any) => item.name === name);
const keys = ["firstVisibleAfterEmissionMs", "emissionToVisible.medianMs", "drainThroughTerminalMs",
...(name === "paced" ? ["interChunkGaps.maxMs"] : []),
...(observed.emissionToVisible.p95Ms === undefined ? [] : ["emissionToVisible.p95Ms", "intendedToVisible.p95Ms"]),
...(observed.stop ? ["stopThroughTerminalMs", "stopThroughAcknowledgementMs"] : [])];
...(observed.stop ? ["stopThroughTerminalMs", "stopThroughAcknowledgementMs"] : []),
...(h5 && !observed.stop ? ["h5.independentDrain.elapsedFromEpochMs"] : [])];
return keys.map(key => {
const samples = pairs.map(pair => {
const baseline = metric(pair.baseline.cases.find((item: any) => item.name === name), key);
const candidate = metric(pair.candidate.cases.find((item: any) => item.name === name), key);
assert.ok(Number.isFinite(baseline) && baseline > 0 && Number.isFinite(candidate) && candidate >= 0, `Invalid latency ${name}.${key}`);
return { baseline, candidate, improvementRelative: (baseline - candidate) / baseline };
return { baseline, candidate, improvementMs: baseline - candidate, improvementRelative: (baseline - candidate) / baseline };
});
const pooledMedianMs = pooledMedian(samples.flatMap(sample => [sample.baseline, sample.candidate]));
const representationLimited = h5 && pooledMedianMs <= browserAaRule.representationLimitedMedianMs;
const aaSuitability = h5 && phase === "aa" ? assessAaMetric(samples, armChronology!) : undefined;
return { name, metric: key, baselineMedianMs: median(samples.map(sample => sample.baseline)),
candidateMedianMs: median(samples.map(sample => sample.candidate)),
medianPairedImprovementRelative: median(samples.map(sample => sample.improvementRelative)),
medianPairedImprovementRelative: representationLimited ? null : median(samples.map(sample => sample.improvementRelative)),
...(h5 ? { pooledMedianMs, representationLimited, percentageGainClaimAllowed: !representationLimited,
sub2msOrP99GainClaimAllowed: false, medianPairedImprovementMs: median(samples.map(sample => sample.improvementMs)),
aaSuitability } : {}),
candidateWins: samples.filter(sample => sample.improvementRelative > 0).length,
aaNoiseEnvelopeRelative: phase === "aa" ? Math.max(...samples.map(sample => Math.abs(sample.improvementRelative))) : undefined,
...(h5 ? { aaNoiseEnvelopeAbsoluteMs: phase === "aa" ? Math.max(...samples.map(sample => Math.abs(sample.improvementMs))) : undefined } : {}),
samples };
});
});
// A descriptive rate from the independently verified completion upper bound.
// Gate duration against its own A/A noise; invent no separate rate threshold.
const verifiedCompletionRates = h5 ? names.filter(name => !first.baseline.cases.find((item: any) => item.name === name).stop).map(name => {
const samples = pairs.map(pair => {
const baseline = pair.baseline.cases.find((item: any) => item.name === name);
const candidate = pair.candidate.cases.find((item: any) => item.name === name);
const counts = [baseline, candidate].map(item => item.h5.independentDrain.deltaCount);
assert.equal(counts[0], counts[1], "Verified completion rates require equal matched measured-delta counts");
assert.ok([baseline, candidate].every(item => item.h5.independentDrain.elapsedFromEpochMs > 0),
"Verified completion rate needs positive observed duration");
return { measuredDeltas: counts[0], baselineElapsedMs: baseline.h5.independentDrain.elapsedFromEpochMs,
candidateElapsedMs: candidate.h5.independentDrain.elapsedFromEpochMs,
baselineDeltasPerSecond: 1000 * counts[0] / baseline.h5.independentDrain.elapsedFromEpochMs,
candidateDeltasPerSecond: 1000 * counts[1] / candidate.h5.independentDrain.elapsedFromEpochMs };
});
return { name, sampleCount: samples.length, samples,
baselineMedianDeltasPerSecond: median(samples.map(sample => sample.baselineDeltasPerSecond)),
candidateMedianDeltasPerSecond: median(samples.map(sample => sample.candidateDeltasPerSecond)),
gatedDurationMetric: "h5.independentDrain.elapsedFromEpochMs",
scope: "Matched measured normalized deltas / independently verified completion upper bound; includes oracle, reload, screenshot, API and lease verification; not exact last-commit throughput" };
}) : undefined;
const retryObservations = names.map(name => {
const samples = pairs.map(pair => Object.fromEntries((["baseline", "candidate"] as const).map(arm => {
const item = pair[arm].cases.find((value: any) => value.name === name);
@@ -525,6 +614,9 @@ const result: Record<string, any> = {
comparatorSha256: createHash("sha256").update(await readFile(new URL(import.meta.url))).digest("hex"),
manifestSha256: createHash("sha256").update(await readFile(manifestPath)).digest("hex"), measurements,
primaryLoadedWorkload: "burst512", complete: true, retryObservations,
...(h5 ? { verifiedCompletionRates, armChronology,
...(phase === "aa" ? { aaSuitability: { rule: browserAaRule, metricCount: measurements.length,
passed: measurements.every(item => item.aaSuitability?.passed) } } : {}) } : {}),
...(h5 ? { observationMode: H5_MODE, workloadSha256: H5_WORKLOAD_SHA256, candidatePolicy, h5Assessments,
artifactInventories: { baseline: first.baseline.armInventory, candidate: first.candidate.armInventory }, originalH4eConfirmation,
originalProtocolAcceptance: "Baseline SLA misses are retained; H5c complete observation is not an original-deadline PASS",
@@ -550,16 +642,55 @@ if (phase === "compare") {
assert.equal(noise.gatesSha256, result.gatesSha256, "Acceptance gates changed after A/A calibration");
assert.equal(noise.comparatorSha256, result.comparatorSha256, "Comparator changed after A/A calibration");
assert.ok(noise.pairCount >= 5);
if (h5) {
assert.equal(noise.pairCount, 5); assert.deepEqual(noise.aaSuitability?.rule, browserAaRule, "Browser A/A suitability rule changed");
const chronology = assertArmChronology(noise.armChronology);
assert.deepEqual(chronology, noise.armChronology, "Frozen A/A chronology was reordered");
assert.ok(armChronology![0].startedAtMs >= chronology.at(-1).finishedAtMs,
"Comparison must follow all owning calibration invocations");
const priorOwners = new Set(chronology.map(row => row.ownerReportSha256));
const priorCompanies = new Set(chronology.map(row => row.companyId));
const priorRuns = new Set(chronology.flatMap(row => row.runIds));
for (const arm of armChronology!) {
assert.ok(!priorOwners.has(arm.ownerReportSha256) && !priorCompanies.has(arm.companyId)
&& arm.runIds.every((runId: string) => !priorRuns.has(runId)), "Comparison reuses a calibration owner/company/run");
}
assert.equal(noise.measurements?.length, measurements.length, "Frozen A/A is missing a predeclared metric");
assert.equal(noise.aaSuitability.metricCount, measurements.length);
for (const item of measurements) {
const controls = noise.measurements.filter((entry: any) => entry.name === item.name && entry.metric === item.metric);
assert.equal(controls.length, 1, "Frozen A/A needs exactly one calibration per metric");
const control = controls[0], actual = assessAaMetric(control.samples, chronology);
assert.equal(actual.passed, true, "Unsuitable browser A/A cannot freeze noise");
assert.deepEqual(control.aaSuitability, actual, "Browser A/A suitability was forged");
assert.equal(control.aaNoiseEnvelopeAbsoluteMs, actual.observedNoiseAbsoluteMs, "Browser absolute noise was forged");
assert.equal(control.aaNoiseEnvelopeRelative, actual.observedNoiseRelative, "Browser relative noise was forged");
assert.equal(control.representationLimited, actual.representationLimited, "Browser representation flag was forged");
}
assert.equal(noise.aaSuitability.passed, true, "Unsuitable browser A/A cannot freeze noise");
}
result.gates = measurements.map(item => {
const control = noise.measurements.find((entry: any) => entry.name === item.name && entry.metric === item.metric);
assert.ok(control);
const improvementRequired = expectation === "improvement" && item.name === "burst512"
&& ["emissionToVisible.p95Ms", "intendedToVisible.p95Ms", "drainThroughTerminalMs"].includes(item.metric);
const noRegression = -item.medianPairedImprovementRelative <= control.aaNoiseEnvelopeRelative;
const paired = noiseExceedingPairedWins(item.samples.map(sample => sample.improvementRelative), control.aaNoiseEnvelopeRelative);
&& ["emissionToVisible.p95Ms", "intendedToVisible.p95Ms", "drainThroughTerminalMs",
"h5.independentDrain.elapsedFromEpochMs"].includes(item.metric);
const representationLimited = h5 && (control.representationLimited || item.representationLimited);
if (representationLimited) {
item.representationLimited = true; item.percentageGainClaimAllowed = false; item.medianPairedImprovementRelative = null;
}
const medianImprovementRelative = median(item.samples.map(sample => sample.improvementRelative));
const medianImprovementMs = median(item.samples.map(sample => sample.improvementMs));
const noRegression = representationLimited ? -medianImprovementMs <= control.aaNoiseEnvelopeAbsoluteMs
: -medianImprovementRelative <= control.aaNoiseEnvelopeRelative;
const paired = noiseExceedingPairedWins(item.samples.map(sample => h5 && sample.improvementMs <= 2 ? 0 : sample.improvementRelative), control.aaNoiseEnvelopeRelative);
const improvement = paired.pairedWinRequirementPassed
&& item.medianPairedImprovementRelative > control.aaNoiseEnvelopeRelative;
&& medianImprovementRelative > control.aaNoiseEnvelopeRelative && (!h5 || medianImprovementMs > 2);
return { name: item.name, metric: item.metric, improvementRequired, noiseEnvelopeRelative: control.aaNoiseEnvelopeRelative,
...(h5 ? { noiseEnvelopeAbsoluteMs: control.aaNoiseEnvelopeAbsoluteMs, representationLimited,
percentageGainClaimAllowed: !representationLimited, sub2msOrP99GainClaimAllowed: false,
nonregressionScale: representationLimited ? "observed-absolute-ms" : "observed-relative",
requiredAbsoluteImprovementAboveMs: improvementRequired ? 2 : undefined } : {}),
noiseExceedingPairedWins: paired.noiseExceedingPairedWins, requiredPairedWins: paired.requiredPairedWins,
passed: improvementRequired ? improvement : noRegression };
});
@@ -588,6 +719,8 @@ if (phase === "compare") {
}
if (arg("output")) await writeFile(arg("output")!, JSON.stringify(result, null, 2));
console.log(JSON.stringify(result, null, 2));
if (h5 && phase === "aa") assert.equal(result.aaSuitability.passed, true,
"Browser A/A suitability failed; retain all ten arms, diagnose and review a fresh calibration");
if (process.argv.includes("--require-performance")) {
assert.equal(h5, true, "H5c performance-only gating is explicit; default H4e keeps original deadlines");
assert.equal(phase, "compare");
scripts/native-ingestion-h5.test.mjs
+94 -13
View File
@@ -3,23 +3,20 @@ import { test } from 'node:test';
import { createRequire } from 'node:module';
import { readFile } from 'node:fs/promises';
import { h5Enabled, H5_MODE, observeStreaming, originalTerminal, requireH5Candidate,
H5_LIMITS, ORIGINAL_BUDGETS, prepareParticipantAgents, observeRunSettlement } from '../tests/e2e/fixtures/native-ingestion-h5.mjs';
H5_LIMITS, ORIGINAL_BUDGETS, prepareParticipantAgents, observeRunSettlement, collectDurableRunEvents, assertDurableRunEvents,
assertPreludeEvents, assertVisibleMarkerClocks } from '../tests/e2e/fixtures/native-ingestion-h5.mjs';
const timeout = milliseconds => Object.assign(new Error(`Timeout ${milliseconds}ms exceeded while waiting on the predicate`),
{ matcherResult: { name: 'toBeGreaterThanOrEqual', pass: false } });
test('the actual paginated result counter counts accepted results in default H4e and observation H5c', async () => {
const source = await readFile(new URL('../tests/e2e/native-event-ingestion.journey.ts', import.meta.url), 'utf8');
const statements = source.split('\n').filter(line => line.includes('resultAcceptedEvents++'));
assert.equal(statements.length, 1, 'Exercise the actual event-loop counter statement');
const count = new Function('h5', 'rows', `let resultAcceptedEvents = 0;
for (const row of rows) { const event = row.payload?.prpEvent; ${statements[0]} } return resultAcceptedEvents;`);
for (const h5 of [false, true]) {
assert.equal(count(h5, [{ eventType: 'run.result.accepted', payload: { prpEvent: { eventType: 'run.result.accepted' } } }]), 1);
assert.equal(count(h5, [{ payload: { prpEvent: { eventType: 'run.result.accepted' } } }]), 1);
assert.equal(count(h5, [{ eventType: 'run.result.proposed' }, { payload: { prpEvent: { eventType: 'item.delta' } } }]), 0);
assert.equal(count(h5, []), 0);
}
test('the actual durable collector counts each accepted row once in default H4e and observation H5c', () => {
for (const row of [
{ eventType: 'run.result.accepted', payload: { prpEvent: { eventType: 'run.result.accepted' } } },
{ payload: { prpEvent: { eventType: 'run.result.accepted' } } },
{ eventType: 'run.result.accepted' }]) assert.equal(collectDurableRunEvents([row], 'q7').resultAcceptedEvents, 1);
assert.equal(collectDurableRunEvents([{ eventType: 'run.result.proposed' },
{ payload: { prpEvent: { eventType: 'item.delta' } } }], 'q7').resultAcceptedEvents, 0);
assert.equal(collectDurableRunEvents([], 'q7').resultAcceptedEvents, 0);
});
test('real frozen Playwright tiny poll exposes the matcher deadline shape used by H5', async () => {
@@ -231,3 +228,87 @@ test('cursor inherits only paired exact known misses; final candidate inherits n
assert.throws(() => requireH5Candidate('final-b2', base, base), /every original SLA/);
assert.throws(() => requireH5Candidate('final-b2', slowInvocation, slowInvocation), /every original SLA/);
});
const durableFixture = ({ stop = false, count = 4 } = {}) => {
const runId = 'run-1', token = 'provider-1', markerCode = 'q7';
const rows = Array.from({ length: count }, (_, index) => ({ seq: index+1, payload: { prpEvent: {
eventType: 'item.delta', runId, sourceInstanceId: 'source-1', sourceEventId: `event-${index}`, sourceSeq: index+1,
payload: { text: `[${markerCode}${index.toString(36)}] ` } } } }));
if (!stop) rows.push({ seq: count+1, eventType: 'run.result.accepted' });
return { rows, input: { runId, token, markerCode, events: 4, stop, requireNativeIdentity: true,
sourceClocks: new Map(Array.from({ length: 4 }, (_, index) => [index, { nonce: token, markerCode, index,
intendedMs: 1000+index*25, emittedMs: 1000+index*25, receivedAtMs: 1001+index*25 }])),
visible: Array.from({ length: stop ? count : 4 }, (_, index) => ({ index, markerCode, visibleMs: 1003+index*25 })) } };
};
const checkDurable = fixture => assertDurableRunEvents({ ...fixture.input,
collected: collectDurableRunEvents(fixture.rows, fixture.input.markerCode) });
test('durable oracle retains complete per-run evidence in H4e and H5c and permits only the canonical Stop prefix', () => {
for (const requireNativeIdentity of [false,true]) {
const fixture = durableFixture(); fixture.input.requireNativeIdentity = requireNativeIdentity;
assert.deepEqual(checkDurable(fixture), { runId: 'run-1', token: 'provider-1', markerCode: 'q7', individualDeltas: 4,
uniqueSourceIdentities: 4, matchedSourceClocks: 4, ordered: true, resultAcceptedEvents: 1 });
}
const stopped = durableFixture({ stop: true, count: 2 });
assert.equal(checkDurable(stopped).individualDeltas, 2);
assert.equal(checkDurable(stopped).resultAcceptedEvents, 0);
assert.equal(stopped.input.sourceClocks.size, 4, 'Raw emissions outside the durable Stop prefix remain permitted');
});
test('durable oracle rejects corrupted payloads, coalescing, replay identity conflicts, gaps and foreign runs', () => {
const changes = [
fixture => { fixture.rows[0].payload.prpEvent.payload.text = '[q70] corruption'; },
fixture => { fixture.rows[0].payload.prpEvent.payload.text = '[q70] [q71] '; },
fixture => { fixture.rows[1].payload.prpEvent.sourceEventId = 'event-0'; fixture.rows[1].payload.prpEvent.sourceSeq = 1; },
fixture => { fixture.rows.splice(1,1); }, fixture => { fixture.rows.splice(1,0,fixture.rows[0]); },
fixture => { fixture.rows.reverse(); }, fixture => { fixture.rows[0].payload.prpEvent.runId = 'foreign-run'; },
fixture => { fixture.rows[0].payload.prpEvent.sourceInstanceId = ''; },
fixture => { fixture.rows[0].payload.prpEvent.sourceEventId = ''; },
fixture => { fixture.rows[0].payload.prpEvent.sourceSeq = 0; },
fixture => { fixture.rows[0].payload.prpEvent.sourceSeq = '1'; },
];
for (const change of changes) {
const fixture = durableFixture(); change(fixture);
assert.throws(() => checkDurable(fixture), /Durable run run-1/, 'Failure must identify its exact run');
}
});
test('durable oracle rejects wrong or missing original clock joins, wrong source counts and result counts', () => {
for (const change of [
fixture => { fixture.input.sourceClocks.delete(0); },
fixture => { fixture.input.sourceClocks.get(0).nonce = 'foreign-source'; },
fixture => { fixture.input.sourceClocks.get(0).markerCode = 'z9'; },
fixture => { fixture.input.sourceClocks.get(0).index = 1; },
fixture => { fixture.input.sourceClocks.set(9, { nonce: 'provider-1', markerCode: 'q7', index: 9 }); },
fixture => { fixture.rows.pop(); }, fixture => { fixture.rows.push({ eventType: 'run.result.accepted' }); },
]) { const fixture = durableFixture(); change(fixture); assert.throws(() => checkDurable(fixture), /Durable run run-1/); }
});
test('durable Stop oracle rejects visible markers outside the ordered durable prefix or an accepted result', () => {
const missing = durableFixture({ stop: true, count: 2 }); missing.input.visible.push({ index: 2 });
assert.throws(() => checkDurable(missing), /visible Stop marker 2/);
const gap = durableFixture({ stop: true, count: 2 }); gap.rows.shift(); assert.throws(() => checkDurable(gap), /canonical Stop prefix/);
const result = durableFixture({ stop: true, count: 2 }); result.rows.push({ eventType: 'run.result.accepted' });
assert.throws(() => checkDurable(result), /exact accepted result count/);
});
test('prelude oracle preserves exact setup payloads, source tuple uniqueness, count and prelude-only selection', () => {
const rows = [0,1,2].map(index => ({ payload: { prpEvent: { eventType: 'item.delta', itemId: 'prelude-turn',
sourceInstanceId: 'source-1', sourceEventId: `event-${index}`, sourceSeq: index+1,
payload: { text: index === 2 ? ' PRELUDE_provider-1_READY ' : '.' } } } }));
const check = selected => assertPreludeEvents({ rows: selected, runId: 'run-1', token: 'provider-1', expectedEvents: 3 });
assert.deepEqual(check([...rows,{ payload: { prpEvent: { eventType: 'item.delta', itemId: 'measured-turn' } } }]),
{ individualDeltas: 3, uniqueSourceIdentities: 3 });
assert.throws(() => check(rows.slice(1)), /Prelude run run-1/);
const bad = structuredClone(rows); bad[0].payload.prpEvent.payload.text = '..'; assert.throws(() => check(bad), /complete normalized payload/);
const replay = structuredClone(rows); Object.assign(replay[1].payload.prpEvent, { sourceEventId: 'event-0', sourceSeq: 1 });
assert.throws(() => check(replay), /unique source identity tuples/);
});
test('visible-clock oracle preserves original two-frame timestamps and rejects missing clocks or incomplete unique output', () => {
const fixture = durableFixture(), { visible: seenMarkers, sourceClocks } = fixture.input;
const check = (seen = seenMarkers, clocks = sourceClocks, stop = false) => assertVisibleMarkerClocks({
seenMarkers: seen, sourceClocks: clocks, markerCode: 'q7', expectedEvents: 4, stop });
const visible = check(); assert.deepEqual(visible.map(row => row.visibleMs), seenMarkers.map(row => row.visibleMs));
assert.deepEqual(visible.map(row => row.emittedMs), [...sourceClocks.values()].map(row => row.emittedMs));
assert.throws(() => check(seenMarkers, new Map()), /Missing original provider clock/);
assert.throws(() => check(seenMarkers.slice(1)), /complete unique foreground count/);
assert.throws(() => check([seenMarkers[0],seenMarkers[0],seenMarkers[2],seenMarkers[3]]), /complete unique foreground count/);
assert.equal(check(seenMarkers.slice(0,2), sourceClocks, true).length, 2);
});
scripts/native-ingestion-performance-gates.test.mjs
+155 -5
View File
@@ -234,7 +234,7 @@ test("H5 assessor retains an original SLA failure assessed before its nominal st
assert.throws(() => requireH5Candidate("final-b2", outcome, outcome), /every original SLA/);
});
test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires complete owning evidence", async () => {
test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires complete owning evidence", async context => {
const directory = await mkdtemp(path.join(os.tmpdir(), "native-ingestion-h5-gates-"));
const script = new URL("./compare-native-ingestion-browser.ts", import.meta.url);
const save = async (name, value) => { const file = path.join(directory, name); await writeFile(file, JSON.stringify(value)); return file; };
@@ -271,11 +271,17 @@ test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires
for (let index = 0; index < 5; index++) {
const entry = {};
for (const arm of ["baseline", "candidate"]) {
const report = h5Report(arm === "baseline" || sameRevision ? baselineRevision : candidateRevision, arm === "baseline" ? 180000 : candidateLatency);
const ordinal = 2 * index + (arm === "baseline" ? index % 2 : index % 2 ? 0 : 1);
const startedAtMs = Date.parse(sameRevision ? "2026-11-02T00:00:00.000Z" : "2026-11-03T00:00:00.000Z") + ordinal * 2000000;
const ownedIdPrefix = createHash("sha256").update(prefix).digest("hex").slice(0,6) + ordinal.toString(16).padStart(2,"0");
const report = JSON.parse(JSON.stringify(h5Report(arm === "baseline" || sameRevision ? baselineRevision : candidateRevision,
arm === "baseline" ? 180000 : candidateLatency)).replaceAll("00000000-", `${ownedIdPrefix}-`), (key, value) =>
typeof value === "number" && (key.endsWith("AtMs") || ["epochMs", "deadlineMs", "readinessDeadlineMs", "visibleMs", "intendedMs", "emittedMs"].includes(key)) ? value + startedAtMs : value);
report.bodyFinishedAtMs = startedAtMs + duration - 1;
const owner = { timeout: 1200000, expectedStatus: "passed", results: [{ status: "passed", retry: 0,
duration, startTime: "2026-11-03T00:00:00.000Z", errors: [], attachments: [] }] };
duration, startTime: new Date(startedAtMs).toISOString(), errors: [], attachments: [] }] };
if (index === 0 && arm === "candidate") mutate?.(report, owner);
mutateArm?.(report, { index, arm });
mutateArm?.(report, { index, arm, owner });
owner.results[0].attachments.push({ name: "native-ingestion-evidence", contentType: "application/json",
body: Buffer.from(JSON.stringify(report)).toString("base64") });
entry[`${arm}File`] = await save(`${prefix}-${index}-${arm}.json`, report);
@@ -299,11 +305,90 @@ test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires
const originalInventory = await inventory(proof, originalProof, "original-h4c");
const finalProofArgs = [`--original-h4e-proof=${originalProof}`, `--original-h4e-inventory=${originalInventory}`];
const noise = path.join(directory, "aa.json");
const aa = run(["--phase=aa", `--pairs=${await manifest("aa", 181000, { sameRevision: true })}`, `--output=${noise}`]);
const aaManifest = await manifest("aa", 181000, { sameRevision: true });
const aa = run(["--phase=aa", `--pairs=${aaManifest}`, `--output=${noise}`]);
assert.equal(aa.status, 0, aa.stderr);
const aaReport = JSON.parse(await readFile(noise, "utf8"));
assert.equal(aaReport.h5Assessments.length, 5);
assert.equal(aaReport.measurements.filter(item => item.metric === "h5.independentDrain.elapsedFromEpochMs").length, 5,
"Every non-Stop case needs its own verified-completion A/A noise");
assert(aaReport.measurements.find(item => item.name === "burst512" && item.metric === "h5.independentDrain.elapsedFromEpochMs").aaNoiseEnvelopeRelative > 0);
assert.deepEqual(aaReport.h5Assessments[0].baseline.misses, ["burst512:streaming120"]);
assert.equal(aaReport.aaSuitability.passed, true); assert.equal(aaReport.armChronology.length, 10);
assert.equal(new Set(aaReport.armChronology.map(arm => arm.companyId)).size, 10);
for (const [name, mutateArm, expectation] of [
["aa-paired-jitter", (report, { arm, index }) => { if (arm === "candidate" && index === 4) report.cases[0].firstVisibleAfterEmissionMs = 1300; }, "paired"],
["aa-chronological-drift", (report, { index }) => { report.cases[0].firstVisibleAfterEmissionMs = 1000 + index * 60; }, "chronological"],
["aa-role-bias", (report, { arm }) => { report.cases[0].firstVisibleAfterEmissionMs = arm === "candidate" ? 1120 : 1000; }, "role"],
]) {
const output = path.join(directory, `${name}.json`);
const attempt = run(["--phase=aa", `--pairs=${await manifest(name, 180000, { sameRevision: true, mutateArm })}`, `--output=${output}`]);
assert.notEqual(attempt.status, 0, name); assert.match(attempt.stderr, /A\/A suitability failed/);
const unsuitable = JSON.parse(await readFile(output, "utf8"));
assert.equal(unsuitable.aaSuitability.passed, false); assert.equal(unsuitable.armChronology.length, 10);
const metric = unsuitable.measurements.find(item => item.name === "paced" && item.metric === "firstVisibleAfterEmissionMs").aaSuitability;
if (expectation === "paired") assert(metric.pairedDifferencesMs.some((value,index) => value > metric.pairedLimitsMs[index]));
if (expectation === "chronological") { assert.equal(metric.observedNoiseAbsoluteMs, 0); assert(metric.chronologicalMedianDriftMs > metric.chronologicalDriftLimitMs); }
if (expectation === "role") assert(metric.roleMedianDriftMs > metric.roleDriftLimitMs);
}
for (const [name, mutateArm] of [
["aa-missing-owner-time", (_report, { arm, index, owner }) => { if (arm === "candidate" && index === 0) delete owner.results[0].startTime; }],
["aa-overlap", (_report, { arm, index, owner }) => { if (arm === "candidate" && index === 0) owner.results[0].startTime = "2026-11-02T00:00:00.001Z"; }],
["aa-wrong-order", (_report, { arm, index, owner }) => { if (index === 1) owner.results[0].startTime = new Date(Date.parse("2026-11-02T00:00:00.000Z") + (arm === "baseline" ? 4 : 5) * 1000000).toISOString(); }],
["aa-reused-company-run", (report) => { const prefix = report.companyId.slice(0,8); Object.assign(report, JSON.parse(JSON.stringify(report).replaceAll(`${prefix}-`, "00000000-"))); }],
]) {
const attempt = run(["--phase=aa", `--pairs=${await manifest(name, 180000, { sameRevision: true, mutateArm })}`]);
assert.notEqual(attempt.status, 0, name);
}
for (const [name, forge] of [
["forged-absolute-noise", noise => { noise.measurements[0].aaNoiseEnvelopeAbsoluteMs = 9999; }],
["forged-relative-noise", noise => { noise.measurements[0].aaNoiseEnvelopeRelative = 9; }],
["forged-representation", noise => { noise.measurements[0].representationLimited = !noise.measurements[0].representationLimited; }],
["forged-suitability", noise => { noise.measurements[0].aaSuitability.pairedLimitsMs[0] = 9999; }],
["forged-sample", noise => { noise.measurements[0].samples[0].improvementMs++; }],
["forged-rule", noise => { noise.aaSuitability.rule.contrastRepresentationFloorMs = 1000; }],
]) {
const forged = structuredClone(aaReport); forge(forged);
const attempt = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression",
`--pairs=${await manifest(name, 180000, { candidateRevision: "B1" })}`, `--noise-report=${await save(`${name}-noise.json`, forged)}`, "--require-performance"]);
assert.notEqual(attempt.status, 0, name); assert.match(attempt.stderr, /forged|rule changed/);
}
for (const repeat of [1,2]) {
const shortNoise = path.join(directory, `short-${repeat}-noise.json`);
const aa = run(["--phase=aa", `--pairs=${await manifest(`short-${repeat}-aa`, 180000, { sameRevision: true,
mutateArm: (report, { arm }) => { report.cases[0].firstVisibleAfterEmissionMs = arm === "candidate" ? repeat : 1; } })}`, `--output=${shortNoise}`]);
assert.equal(aa.status, 0, aa.stderr);
const calibrated = JSON.parse(await readFile(shortNoise, "utf8"));
const control = calibrated.measurements.find(item => item.name === "paced" && item.metric === "firstVisibleAfterEmissionMs");
assert.equal(control.representationLimited, true); assert.equal(control.percentageGainClaimAllowed, false);
assert.equal(control.medianPairedImprovementRelative, null); assert.equal(control.aaNoiseEnvelopeAbsoluteMs, repeat - 1);
const output = path.join(directory, `short-${repeat}-regression.json`);
const attempt = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression",
`--pairs=${await manifest(`short-${repeat}-regression`, 180000, { candidateRevision: "B1", mutateArm: (report, { arm }) => {
report.cases[0].firstVisibleAfterEmissionMs = repeat === 1 ? arm === "candidate" ? 2 : 1 : arm === "candidate" ? 12 : 10;
} })}`, `--noise-report=${shortNoise}`, `--output=${output}`, "--require-performance"]);
assert.notEqual(attempt.status, 0, "The suitability floor must not pad actual observed absolute noise");
const compared = JSON.parse(await readFile(output, "utf8"));
const gate = compared.gates.find(item => item.name === "paced" && item.metric === "firstVisibleAfterEmissionMs");
assert.equal(gate.noiseEnvelopeAbsoluteMs, repeat - 1); assert.equal(gate.representationLimited, true); assert.equal(gate.passed, false);
if (repeat === 2) {
const improving = path.join(directory, "short-calibrated-improvement.json");
const changed = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression",
`--pairs=${await manifest("short-calibrated-improvement", 180000, { candidateRevision: "B1", mutateArm: (report, { arm }) => {
report.cases[0].firstVisibleAfterEmissionMs = arm === "candidate" ? 1 : 100;
} })}`, `--noise-report=${shortNoise}`, `--output=${improving}`, "--require-performance"]);
assert.equal(changed.status, 0, changed.stderr);
const measured = JSON.parse(await readFile(improving, "utf8")).measurements.find(item => item.name === "paced" && item.metric === "firstVisibleAfterEmissionMs");
assert.equal(measured.representationLimited, true); assert.equal(measured.percentageGainClaimAllowed, false);
assert.equal(measured.medianPairedImprovementRelative, null, "Short calibration cannot turn into a percentage gain claim");
}
}
const replayedOwners = JSON.parse(await readFile(await manifest("reused-aa-owner", 180000, { candidateRevision: "B1" }), "utf8"));
const oldOwners = JSON.parse(await readFile(aaManifest, "utf8"));
for (const field of ["baselineFile", "baselinePlaywrightFile", "baselineInventoryFile"]) replayedOwners[0][field] = oldOwners[0][field];
const replay = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression",
`--pairs=${await save("reused-aa-owner-pairs.json", replayedOwners)}`, `--noise-report=${noise}`, "--require-performance"]);
assert.notEqual(replay.status, 0); assert.match(replay.stderr, /calibration|reus/);
const cursor = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression",
`--pairs=${await manifest("cursor", 180000, { candidateRevision: "B1" })}`, `--noise-report=${noise}`, "--require-acceptance"]);
assert.equal(cursor.status, 0, cursor.stderr);
@@ -318,6 +403,69 @@ test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires
const goodPairs = await manifest("good", 1000);
const good = run(["--candidate-policy=final-b2", `--pairs=${goodPairs}`, `--noise-report=${noise}`, ...finalProofArgs, "--require-acceptance"]);
assert.equal(good.status, 0, good.stderr);
const completionRegressionPairs = await manifest("verified-drain-regression", 1000, {
mutateArm: (report, { arm }) => {
if (arm !== "candidate") return;
const item = report.cases[3];
Object.assign(item.h5.independentDrain, { elapsedFromEpochMs: 200000, verifiedAtMs: item.epochMs + 200000 });
} });
const completionRegressionOutput = path.join(directory, "verified-drain-regression.json");
const completionRegression = run(["--candidate-policy=final-b2", `--pairs=${completionRegressionPairs}`,
`--noise-report=${noise}`, `--output=${completionRegressionOutput}`, "--require-performance"]);
const completionRegressionReport = JSON.parse(await readFile(completionRegressionOutput, "utf8"));
context.diagnostic(JSON.stringify({ regression: "browser and terminal faster but independently verified completion slower",
comparatorSha256: completionRegressionReport.comparatorSha256, exit: completionRegression.status,
performancePassed: completionRegressionReport.performancePassed, baselineVerifiedMs: 180010, candidateVerifiedMs: 200000,
verifiedCompletionGate: completionRegressionReport.gates.find(gate => gate.name === "burst512"
&& gate.metric === "h5.independentDrain.elapsedFromEpochMs") ?? null }));
assert.notEqual(completionRegression.status, 0, "Faster browser/terminal must not conceal slower fully verified completion");
const regressionGate = completionRegressionReport.gates.find(gate => gate.name === "burst512"
&& gate.metric === "h5.independentDrain.elapsedFromEpochMs");
assert.equal(regressionGate.improvementRequired, true); assert.equal(regressionGate.passed, false);
assert(completionRegressionReport.gates.filter(gate => gate.name === "burst512" &&
["emissionToVisible.p95Ms", "intendedToVisible.p95Ms", "drainThroughTerminalMs"].includes(gate.metric)).every(gate => gate.passed));
const goodReport = JSON.parse(good.stdout);
const rate = goodReport.verifiedCompletionRates.find(item => item.name === "burst512");
assert.equal(rate.samples[0].measuredDeltas, 51200);
assert.equal(rate.samples[0].baselineDeltasPerSecond, 51200000 / 180010);
assert.equal(rate.samples[0].candidateDeltasPerSecond, 51200000 / 1010);
assert(rate.candidateMedianDeltasPerSecond > rate.baselineMedianDeltasPerSecond);
for (const beyondNoise of [3,4]) {
const output = path.join(directory, `verified-drain-${beyondNoise}-wins.json`);
const attempt = run(["--candidate-policy=final-b2", `--pairs=${await manifest(`verified-drain-${beyondNoise}-wins`, 1000, {
mutateArm: (report, { arm, index }) => {
if (arm !== "candidate") return;
const item = report.cases[3], elapsed = index < beyondNoise ? 90000 : 179500;
Object.assign(item.h5.independentDrain, { elapsedFromEpochMs: elapsed, verifiedAtMs: item.epochMs + elapsed });
} })}`, `--noise-report=${noise}`, `--output=${output}`, "--require-performance"]);
assert.equal(attempt.status === 0, beyondNoise === 4, attempt.stderr);
const compared = JSON.parse(await readFile(output, "utf8"));
const gate = compared.gates.find(item => item.name === "burst512" && item.metric === "h5.independentDrain.elapsedFromEpochMs");
assert.equal(gate.noiseExceedingPairedWins, beyondNoise); assert.equal(gate.requiredPairedWins, 4);
assert.equal(gate.passed, beyondNoise === 4);
}
const zeroNoise = path.join(directory, "zero-noise.json");
const zeroAA = run(["--phase=aa", `--pairs=${await manifest("zero-noise-aa", 180000, { sameRevision: true })}`, `--output=${zeroNoise}`]);
assert.equal(zeroAA.status, 0, zeroAA.stderr);
for (const resolvedPairs of [0,3,4]) {
const output = path.join(directory, `timestamp-gain-${resolvedPairs}.json`);
const attempt = run(["--candidate-policy=final-b2", `--pairs=${await manifest(`timestamp-gain-${resolvedPairs}`, 1000, {
mutateArm: (report, { arm, index }) => {
if (arm !== "candidate") return;
const item = report.cases[3], gain = resolvedPairs === 0 ? 1 : index < resolvedPairs ? 3 : 2;
Object.assign(item.h5.independentDrain, { elapsedFromEpochMs: 180010 - gain, verifiedAtMs: item.epochMs + 180010 - gain });
} })}`, `--noise-report=${zeroNoise}`, `--output=${output}`, "--require-performance"]);
assert.equal(attempt.status === 0, resolvedPairs === 4, attempt.stderr);
const gate = JSON.parse(await readFile(output, "utf8")).gates.find(item => item.name === "burst512"
&& item.metric === "h5.independentDrain.elapsedFromEpochMs");
assert.equal(gate.noiseExceedingPairedWins, resolvedPairs); assert.equal(gate.requiredAbsoluteImprovementAboveMs, 2);
}
const cursorCompletionRegression = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression",
`--pairs=${await manifest("cursor-verified-drain-regression", 180000, { candidateRevision: "B1", mutateArm: (report, { arm }) => {
if (arm !== "candidate") return;
const item = report.cases[3]; Object.assign(item.h5.independentDrain, { elapsedFromEpochMs: 200000, verifiedAtMs: item.epochMs + 200000 });
} })}`, `--noise-report=${noise}`, "--require-performance"]);
assert.notEqual(cursorCompletionRegression.status, 0, "Cursor nonregression must include fully verified completion");
const wrongPolicy = run(["--expect=nonregression", "--candidate-policy=cursor-nonregression", `--pairs=${goodPairs}`,
`--noise-report=${noise}`, "--require-acceptance"]);
assert.notEqual(wrongPolicy.status, 0); assert.match(wrongPolicy.stderr, /Cursor policy requires B0→B1/);
@@ -447,6 +595,8 @@ test("H5 CLI retains baseline SLA failure, rejects final SLA misses and requires
["stop-recorded-provider-failure", report => { report.cases[5].recordedFailures.rows[0].current_error_code = "provider_transport_failed"; }, /recorded failed run must have recovered/],
["censored", report => { report.cases[3].h5.streaming.observation.censored = true; }, /false/],
["terminal-failed", report => { report.cases[3].terminalStatuses[99].status = "failed"; }, /successful drainage/],
["missing-independent-drain", report => { delete report.cases[3].h5.independentDrain; }, /independentDrain|verifiedAtMs|Cannot read/],
["tampered-independent-drain", report => { report.cases[3].h5.independentDrain.elapsedFromEpochMs++; }, /strictly equal/],
["result-missing", report => { report.cases[3].durable[99].resultAcceptedEvents = 0; }, /persisted result/],
["missing-delta", report => { report.cases[3].durable[99].individualDeltas--; }, /individualDeltas|result.events|512/],
["wrong-workload", report => { report.workloadSha256 = "old"; }, /registration changed/],
scripts/native-ingestion/README.md
+1 -1
View File
@@ -1,6 +1,6 @@
# Native ingestion experiment helpers
For fresh cursor results and the current interleaved source reproduction, see [the new source report](../../doc/benchmarks/2026-11-03-native-event-ingestion-interleaved.md). The frozen runner below preserves the earlier experiment. Current application performance requires the independently frozen [H5c driver and helpers](h5c/README.md), successful qualification and fresh comparisons; its gates remain open. The older H4c runner below remains historical.
For fresh cursor results and the current interleaved source reproduction, see [the new source report](../../doc/benchmarks/2026-11-03-native-event-ingestion-interleaved.md). The frozen runner below preserves the earlier experiment. Current application reproduction uses the separately frozen [H5d driver and helpers](h5d/README.md) and [v3 blocked-write Stop companion](blocked-stop/README.md). Each requires its own immutable checkout and locked install: main `2df9897e` uses measurement `e1d8930c…`, while companion `bc33f09c` inherits H5c measurement `2265aabe…`. This mixed delivery tree and its lock are not an executed driver. Functional qualifications passed; no application A/A or performance gain is established at the recorded 09:30 UTC cutoff. The versioned [H5c helpers](h5c/README.md), H5b and older H4c runner below retain historical evidence. Each current campaign has 50 fresh arms and independent noise gates; source reproduction commands are in its linked README.
These helpers reproduce the experiment registered on November 3, 2026. They
retain its immutable application revisions, harness revision, workload hashes,
scripts/native-ingestion/blocked-stop/README.md
+117
View File
@@ -0,0 +1,117 @@
# Frozen blocked-write Stop companion
These are exact source snapshots of the independently reviewed companion and
orchestrator. `sources.json` records every hash. Packaging itself proves no runtime
qualification, control nonregression or application performance gain. The exact
v3 helper is SHA256 `5748b6c1d271cf9ab270d5473873b85da295ffb03bb55ebc63693369c9b1a29d`;
source freeze receipt SHA256 is `210002a6699059ff9b5f74a49558c9f7f0f5866cd9ed3f786507b3ee12fc7b38`.
Use independent driver tag `native-ingestion-blocked-stop-v3`, annotated object
`a49c24dc2a690764d1cfcc080077f90485218df7`, commit
`bc33f09cae26404352848169d92a20c89480a5f5`. It inherits unchanged H5c `e431ec63`
and lock SHA256 `54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8`.
The delivery checkout has a different lock and must not replace that frozen driver.
Main H5d is the separate frozen `2df9897e` driver (measurement `e1d8930c…`);
this companion requires its inherited H5c measurement `2265aabe…`. The mixed
delivery checkout cannot execute either frozen protocol in place of its own driver. The [frozen protocol](../../../tests/e2e/fixtures/native-ingestion-blocked-stop.md)
defines the real row-lock, durable intent, physical Stop, transcript and noise gates.
The helper imports only six checked exports from the unchanged H5c helper,
SHA256 `6b6376b01de6cdc15ca05343a0bdd5c00424f1d6c546b9b4d61cd76716b253c4`:
qualified build/live inventory, preferred UI absence, physical quiet and raw seals.
It never invokes H5c's stage plan or release loader. Copy that exact dependency
from `../h5c/` to its recorded path before using a companion release.
## Recorded qualification and retained failures
As of 2026-11-04 09:30 UTC, ONE v3 functional qualification passed: 1,000 prelude
deltas, a 12-delta canonical Stop prefix with zero results, 1,010 ms held across
15 samples, own committed intent/ACK audits, both exact PIDs and both groups
absent, a bound latest server retirement receipt with cleared identity, released
lease, persisted reload and strict quiet. No manual signals were used. This is
not an accepted A/A or performance sample; the separate 50 fresh arms remain.
The immutable protocol document describes the prospective state at source freeze.
V1 failed before epoch on an unsupported shared-PGID assumption; API200 and port
release did not establish physical quiet, and exact owned cleanup was retained.
V2 failed an unconditional retirement-receipt assertion although retained original
identity after physical exit is supported. Its later transcript/reload/lease gate
was not reached. V3 accepts only the explicit retained or cleared identity contracts;
all independent physical, lease, audited ACK and original-birth gates remain.
Keep both failed trials, old tags/helpers and raw evidence. No failed timing data
is reused or regraded as qualification.
## Independent setup and source-only checks
The benchmark VM uses Node **v26.4.0** and pnpm **9.15.4**. Node 24 was used for
local Mac source checks; it is not the recorded VM runtime. Normally install and
build measured apps independently. Preserve actual qualified build receipts,
native inputs and all four inventoried artifact roots. No new B2 candidate or
qualified optimized build is supplied here; the old `ca7cba17` is rejected as final B2.
Run from this tools checkout on a disposable host. The driver path must be new:
```sh
TOOLS_CHECKOUT="$(git rev-parse --show-toplevel)"
DRIVER_ROOT=/home/ubuntu/native-ingestion/harness-blocked-stop-v3
HELPERS_ROOT=/home/ubuntu/native-ingestion/helpers
git -C "$TOOLS_CHECKOUT" show-ref --verify --quiet refs/tags/native-ingestion-blocked-stop-v3 ||
git -C "$TOOLS_CHECKOUT" fetch origin tag native-ingestion-blocked-stop-v3
test "$(git -C "$TOOLS_CHECKOUT" rev-parse 'native-ingestion-blocked-stop-v3^{commit}')" = bc33f09cae26404352848169d92a20c89480a5f5
git -C "$TOOLS_CHECKOUT" worktree add --detach "$DRIVER_ROOT" native-ingestion-blocked-stop-v3
pnpm --dir "$DRIVER_ROOT" install --frozen-lockfile --offline --ignore-scripts
mkdir -p "$HELPERS_ROOT"
cp "$TOOLS_CHECKOUT/scripts/native-ingestion/h5c/native-ingestion-run-h5c-browser.mjs" "$HELPERS_ROOT/native-ingestion-run-h5c-browser.mjs"
cp "$TOOLS_CHECKOUT/scripts/native-ingestion/blocked-stop/native-ingestion-run-blocked-stop-v3.mjs" "$HELPERS_ROOT/native-ingestion-run-blocked-stop-v3.mjs"
node "$HELPERS_ROOT/native-ingestion-run-blocked-stop-v3.mjs"
node "$DRIVER_ROOT/tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs" --registration
```
The last two commands are dry runs and start no app or arm. Verify both copied
helpers against `sources.json`. The release's `h5HelperFile` is the exact recorded
`/home/ubuntu/native-ingestion/helpers/native-ingestion-run-h5c-browser.mjs` path.
Keep the frozen driver install independent from every app's normal locked build.
Capture actual runtime versions and executable locations before preparing a
release: `node --version`, `node -p process.execPath`, `pnpm --version`,
`rustc --version`, `cargo --version`, and the actual PostgreSQL and Chromium
executables with `--version`. Use exact absolute executable, argument and output
values in `runtimeChecks`; do not guess provider/browser/database versions. Before
each arm, the helper additionally resolves bare PATH `node` under the exact app
cwd and launch environment and verifies its canonical executable and version.
## Explicit release and evidence gates
The dry run prints all required release fields. A release binds authorization,
stage/output, helper hash, exact driver revision and nine hashes, inherited H5c
hash/lock, workload and precision rules, actual runtime checks, qualified app
builds and earlier report/noise file hashes. Source validation is separate from
runtime qualification:
```sh
node "$HELPERS_ROOT/native-ingestion-run-blocked-stop-v3.mjs" \
--release="$RELEASE_JSON" --release-sha256="$RELEASE_SHA256" \
--stage=qualification-b0 --output-root="$EVIDENCE_ROOT" --validate-release
```
After the concrete release and quiet window are reviewed, replace
`--validate-release` with `--execute`. First run only `qualification-b0`: one
actual journey, owning Playwright pass, exact attached raw bytes, full functional
oracle, actual build/UI/native inventory, independent cleanup and sealed proof.
Inspect that evidence before separately releasing any calibration.
Then release `b0-aa`, `b1-aa`, `b0-b1`, `b1-b2`, `b0-b2` in order. Each has five
fresh alternating pairs, for 50 timing arms. Qualification is never reused as a
timing arm. Freeze both suitable A/A noise JSON files and their actual hashes
before further product optimization or comparisons. Unsuitable calibration
retains every raw arm and writes its noise artifact before stopping. No deadline
or precision budget is enlarged after observation. Each owning test keeps its
900-second limit; the 30-minute watchdog only bounds resources.
A passed arm requires immutable source, compiled/native/UI checks, one unretried
recording-off test, functional lock/intent/audit/process/lease/transcript proof,
exact raw/owner/build seals and physical cleanup. The helper revalidates prior
proofs and compares only against that companion's frozen noise. Preserve failed
or censored runs and stop the stage; it never kills unrelated observed processes.
This deterministic local-provider companion does not establish live Cloud Stop
acceptance or replace the main application's performance measurements.
scripts/native-ingestion/blocked-stop/native-ingestion-run-blocked-stop-v3.mjs
+233
View File
@@ -0,0 +1,233 @@
// DRAFT: no execution is released by this source. Default invocation is a dry run.
import assert from 'node:assert/strict';
import { readFile, writeFile, mkdir, open, realpath, lstat } from 'node:fs/promises';
import { spawn, execFileSync } from 'node:child_process';
import { createHash } from 'node:crypto';
import net from 'node:net';
import path from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
const SELF=fileURLToPath(import.meta.url), H5SHA='6b6376b01de6cdc15ca05343a0bdd5c00424f1d6c546b9b4d61cd76716b253c4';
const COMPANION_REVISION='bc33f09cae26404352848169d92a20c89480a5f5';
const INHERITED='e431ec63d41fe90c6fd68bd2fc2bb969feeca917', LOCK='54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8';
const H5MEASUREMENT='2265aabe9c7344f5d4b9c8b08a5c6bbd5571a6e2de9ae7b72a8d1733250e609f';
export const STAGES=['qualification-b0','b0-aa','b1-aa','b0-b1','b1-b2','b0-b2'];
const CORE=['native-event-ingestion-blocked-stop.journey.ts','playwright-native-ingestion-blocked-stop.config.ts',
'fixtures/native-ingestion-blocked-stop-authority.ts','fixtures/native-ingestion-blocked-stop-clock.ts','fixtures/native-ingestion-blocked-stop-db.ts','fixtures/native-ingestion-blocked-stop-protocol.mjs'];
const COMPANION=[...CORE,'fixtures/native-ingestion-blocked-stop-protocol.test.mjs','fixtures/native-ingestion-blocked-stop-validate.mjs','fixtures/native-ingestion-blocked-stop.md'];
const H5=['native-event-ingestion.journey.ts','playwright-native-ingestion.config.ts','fixtures/in-feed-codex.mjs','fixtures/native-ingestion-bin/codex',
'fixtures/composer-stop-codex-loader.mjs','fixtures/native-ingestion-bootstrap.mjs','fixtures/native-ingestion-remote-server.mjs','fixtures/native-ingestion-protocol-qualification.mjs','fixtures/native-ingestion-h5.mjs'];
const CONFIG='tests/e2e/playwright-native-ingestion-blocked-stop.config.ts', SCHEMA='paperclip.native-ingestion.blocked-stop-series.v1';
const AMBIENT=['DATABASE_URL','PAPERCLIP_RUNNER_STATE_DIR','PAPERCLIP_INGESTION_SERVER_COMMAND','PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_DIR',
'PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_FS','PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_INSPECTOR','PAPERCLIP_INGESTION_BROWSER_CAPTURE','NODE_OPTIONS',
'PAPERCLIP_INGESTION_SMOKE','PAPERCLIP_INGESTION_OBSERVATION_MODE'];
const hash=bytes=>createHash('sha256').update(bytes).digest('hex'), json=async file=>JSON.parse(await readFile(file,'utf8'));
const save=(file,value)=>writeFile(file,JSON.stringify(value,null,2)+'\n',{flag:'wx'});
const option=name=>process.argv.find(value=>value.startsWith(`--${name}=`))?.slice(name.length+3);
const absolute=value=>assert.ok(typeof value==='string'&&path.isAbsolute(value),'An absolute path is required');
const sha=value=>assert.match(value??'',/^[a-f0-9]{64}$/), revision=value=>assert.match(value??'',/^[a-f0-9]{40}$/);
const git=(root,...args)=>execFileSync('git',args,{cwd:root,encoding:'utf8'}).trim();
export async function pinned(file,expected){absolute(file);sha(expected);const bytes=await readFile(file);assert.equal(hash(bytes),expected);return bytes;}
export function schedule(stage,registered){
assert.ok(STAGES.includes(stage));if(stage==='qualification-b0')return [{stage,pair:0,index:0,variant:'b0',role:'qualification'}];
const rows=registered.filter(row=>row.stage===stage);assert.equal(rows.length,10);
const [a,b]=stage==='b0-aa'?['b0','b0']:stage==='b1-aa'?['b1','b1']:stage.split('-');
for(let pair=1;pair<=5;pair++)for(let index=0;index<2;index++){
const row=rows[(pair-1)*2+index],role=(pair%2===1?index===0:index===1)?'baseline':'candidate';
assert.deepEqual(row,{stage,pair,index,variant:role==='baseline'?a:b,role});
}return rows;
}
export function assertCompanionRevision(value){assert.equal(value,COMPANION_REVISION,'Use the exact frozen companionv3 revision');}
async function source(context){
const {r}=context;assertCompanionRevision(r.harnessRevision);await pinned(context.releaseFile,context.releaseSha256);await pinned(SELF,r.orchestratorSha256);await pinned(r.h5HelperFile,H5SHA);
assert.equal(git(r.harnessRoot,'rev-parse','HEAD'),r.harnessRevision);assert.equal(git(r.harnessRoot,'status','--short'),'');
execFileSync('git',['merge-base','--is-ancestor',INHERITED,r.harnessRevision],{cwd:r.harnessRoot});
assert.deepEqual(git(r.harnessRoot,'diff','--name-only',INHERITED,r.harnessRevision).split('\n').sort(),COMPANION.map(f=>'tests/e2e/'+f).sort());
assert.equal(hash(await readFile(path.join(r.harnessRoot,'pnpm-lock.yaml'))),LOCK);
assert.equal((await lstat(path.join(r.harnessRoot,'node_modules'))).isSymbolicLink(),false);
assert.deepEqual(Object.keys(r.companionFiles).sort(),[...COMPANION].sort());
for(const [file,digest]of Object.entries(r.companionFiles))await pinned(path.join(r.harnessRoot,'tests/e2e',file),digest);
assert.equal(hash(Buffer.concat(await Promise.all(H5.map(f=>readFile(path.join(r.harnessRoot,'tests/e2e',f)))))),H5MEASUREMENT);
}
export async function load(releaseFile,expectedSha,stage,out){
absolute(out);const bytes=await pinned(releaseFile,expectedSha),r=JSON.parse(bytes);revision(r.harnessRevision);assertCompanionRevision(r.harnessRevision);
assert.equal(r.authorized,true);assert.equal(r.stage,stage);assert.equal(r.outputRoot,out);assert.equal(r.h5HelperSha256,H5SHA);
assert.equal(r.resourceWatchdogMillis,1800000);assert.equal(r.inheritedH5cRevision,INHERITED);assert.equal(r.lockSha256,LOCK);
assert.equal(r.runtimeChecks?.node?.expected,'v26.4.0');assert.equal(r.runtimeChecks?.pnpm?.expected,'9.15.4');
const context={r,out,releaseFile,releaseSha256:hash(bytes)};await source(context);
const h=await import(pathToFileURL(r.h5HelperFile).href),p=await import(pathToFileURL(path.join(r.harnessRoot,'tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.mjs')).href);
assert.equal(p.PROTOCOL,'blocked-stop-history-v1');assert.equal(p.WORKLOAD_SHA256,r.workloadSha256);sha(r.precisionRuleSha256);assert.equal(p.PRECISION_RULE_SHA256,r.precisionRuleSha256);
assert.equal(typeof p.assertNoiseEnvelope,'function');context.h=h;context.p=p;context.rows=schedule(stage,p.registration());
const needed=STAGES.indexOf(stage)<2?['B0']:STAGES.indexOf(stage)<4?['B0','B1']:['B0','B1','B2'];
for(const variant of needed)await h.validateVariantBuild(variant,r.perVariantBuilds?.[variant]);
return context;
}
async function runtime(r){
assert.equal(process.version,'v26.4.0');const values={};
assert.deepEqual(Object.keys(r.runtimeChecks).sort(),['cargo','chromium','node','pnpm','postgres','rustc']);
for(const [name,check]of Object.entries(r.runtimeChecks)){
absolute(check.executable);assert.ok(Array.isArray(check.args)&&check.args.every(v=>typeof v==='string'));assert.ok(check.expected);
const actual=execFileSync(check.executable,check.args,{encoding:'utf8',timeout:10000}).trim();assert.equal(actual,check.expected);values[name]={...check,actual};
}assert.equal(await realpath(r.runtimeChecks.node.executable),await realpath(process.execPath));return values;
}
export async function assertPathNode(environment,approvedExecutable,workingDirectory){
absolute(workingDirectory);
const actual=JSON.parse(execFileSync('node',['-p','JSON.stringify({executable:process.execPath,version:process.version})'],{env:environment,cwd:workingDirectory,encoding:'utf8',timeout:10000}));
assert.equal(actual.version,'v26.4.0','PATH node version differs from approved runtime');
const canonicalExecutable=await realpath(actual.executable),approvedCanonicalExecutable=await realpath(approvedExecutable);
assert.equal(canonicalExecutable,approvedCanonicalExecutable,'PATH node executable differs from approved runtime');
assert.equal(approvedCanonicalExecutable,await realpath(process.execPath),'Approved node differs from orchestrator runtime');
return {...actual,canonicalExecutable,approvedCanonicalExecutable,PATH:environment.PATH,cwd:workingDirectory,canonicalCwd:await realpath(workingDirectory)};
}
async function buildCheck(context,variant){
const b=context.r.perVariantBuilds[variant],q=await context.h.validateVariantBuild(variant,b);
assert.equal(git(b.appRoot,'rev-parse','HEAD'),b.revision);assert.equal(git(b.appRoot,'status','--short'),'');
assert.equal((await json(path.join(b.appRoot,'server/dist/build-info.json'))).commit,b.revision);
const target=await realpath(path.join(b.appRoot,'packages/paperclip-runner/runner/target')),binary=path.join(target,'release/paperclip-runnerd');
assert.equal(target,q.receipt.nativeStageHashes.find(row=>row.variant===variant).targetRealPath);
assert.equal(await realpath(path.join(b.appRoot,'packages/paperclip-runner/runner/target/release/paperclip-runnerd')),binary);await pinned(binary,b.binarySha256);
for(const [file,object]of Object.entries(q.receipt.rustInputObjects))assert.equal(git(b.appRoot,'rev-parse',`${b.revision}:packages/paperclip-runner/${file}`),object);
return {qualified:q,live:await context.h.validateActualInventory(b,q.manifest),ui:await context.h.assertNoPreferredUiShadow(b.appRoot),binary};
}
async function portProbe(port=0){
const server=net.createServer();await new Promise((resolve,reject)=>{server.once('error',reject);server.listen(port,'127.0.0.1',resolve);});
const actual=server.address().port;await new Promise(resolve=>server.close(resolve));return actual;
}
async function launch(context,directory,env,launchEnvironment){
const record={startedAtMs:Date.now(),executable:process.execPath,args:['node_modules/@playwright/test/cli.js','test','--config',CONFIG],cwd:context.r.harnessRoot,environmentOverrides:env,resourceWatchdogMillis:1800000};
const stream=(await open(path.join(directory,'playwright.log'),'wx')).createWriteStream(),child=spawn(record.executable,record.args,{cwd:record.cwd,env:launchEnvironment,stdio:['ignore','pipe','pipe']});
record.childPid=child.pid;child.stdout.pipe(stream,{end:false});child.stderr.pipe(stream,{end:false});let force;
const interrupt=()=>child.kill('SIGINT'),terminate=()=>child.kill('SIGTERM');process.once('SIGINT',interrupt);process.once('SIGTERM',terminate);
const timer=setTimeout(()=>{record.watchdogExpired=true;child.kill('SIGTERM');force=setTimeout(()=>{record.forcedDriverKill=true;child.kill('SIGKILL');},30000);},1800000);
try{Object.assign(record,await new Promise((resolve,reject)=>{child.once('error',reject);child.once('close',(code,signal)=>resolve({code,signal}));}));}
catch(error){record.launchError=String(error);throw error;}finally{
clearTimeout(timer);clearTimeout(force);process.off('SIGINT',interrupt);process.off('SIGTERM',terminate);await new Promise(resolve=>stream.end(resolve));
record.finishedAtMs=Date.now();record.logSha256=hash(await readFile(path.join(directory,'playwright.log')));await save(path.join(directory,'command.json'),record);
}return record;
}
export async function owningEvidence(directory){
const bytes=await readFile(path.join(directory,'playwright.json')),report=JSON.parse(bytes),tests=[];
function visit(value){if(Array.isArray(value))for(const child of value)visit(child);else if(value&&typeof value==='object'){
if(Array.isArray(value.results)&&Object.hasOwn(value,'expectedStatus'))tests.push(value);for(const [key,child]of Object.entries(value))if(key!=='results')visit(child);
}}visit(report);assert.deepEqual(report.errors??[],[]);assert.equal(tests.length,1);const test=tests[0];assert.equal(test.expectedStatus,'passed');assert.equal(test.timeout,900000);assert.equal(test.results.length,1);
const result=test.results[0];assert.equal(result.status,'passed');assert.equal(result.retry,0);assert.deepEqual(result.errors??[],[]);assert.ok(!result.error);
assert.ok(Number.isFinite(report.stats?.duration)&&report.stats.duration>=0&&report.stats.duration<=900000,'Owning invocation exceeded original900s');
assert.ok(Number.isFinite(Date.parse(result.startTime))&&Number.isFinite(result.duration)&&result.duration>=0&&result.duration<=900000,'Missing owning attempt timing');
const attachments=result.attachments.filter(a=>a.name==='native-ingestion-blocked-stop-evidence');assert.equal(attachments.length,1);
let attached;if(attachments[0].body)attached=Buffer.from(attachments[0].body,'base64');else{
const target=await realpath(path.resolve(directory,attachments[0].path));assert.ok(target.startsWith((await realpath(directory))+path.sep),'Attachment escapes owning output');attached=await readFile(target);
}
return {bytes:attached,evidence:JSON.parse(attached),reportSha256:hash(bytes),owner:{startTime:result.startTime,durationMs:result.duration,status:result.status,retry:result.retry}};
}
export async function verifyQuietReceipts(directory){
for(const phase of ['pre-arm','post-arm']){
const q=await json(path.join(directory,phase==='pre-arm'?'physical-quiet-pre.json':'physical-quiet-post.json'));
assert.equal(q.schema,'paperclip.native-ingestion.physical-quiet.v1');assert.equal(q.platform,'linux');assert.equal(q.phase,phase);assert.equal(q.passed,true);
assert.equal(q.zombiePolicy,'Strict: matched zombie/dead rows remain blockers until absent; no automatic killing or silent zombie exclusion.');
assert.ok(q.observations?.length>0);assert.deepEqual(q.observations.at(-1).blockers,[]);
}
}
async function readArm(context,arm,expectedReleaseSha256=context.releaseSha256){
const {r,p}=context,directory=path.dirname(arm.file),attached=await owningEvidence(directory),raw=await readFile(arm.file);assert.equal(hash(raw),hash(attached.bytes));await verifyQuietReceipts(directory);
const preflight=await json(path.join(directory,'pre-arm-inventory-check.json')),port=await json(path.join(directory,'physical-http-port-check.json'));
assert.equal(preflight.releaseSha256,expectedReleaseSha256);assert.equal(preflight.variant,arm.variant);assert.equal(port.nodeBindAndListenPassed,true);
const e=attached.evidence,b=r.perVariantBuilds[arm.variant],sidecar=await json(arm.inventoryFile);p.assertFunctional(e);
assert.equal(e.variant,arm.variant);assert.equal(e.sourceRevision,b.revision);assert.equal(e.buildInfo.commit,b.revision);
assert.equal(e.serverEntrySha256,(await context.h.validateVariantBuild(arm.variant,b)).manifest.files.find(v=>v.path==='server/dist/index.js').sha256);
assert.deepEqual(e.harnessProvenance,{sourceRevision:r.harnessRevision,sourceStatus:'',driverLockSha256:LOCK,inheritedH5cRevision:INHERITED});
const expected={};for(const f of [...CORE,...H5])expected[f]=hash(await readFile(path.join(r.harnessRoot,'tests/e2e',f)));assert.deepEqual(e.sourceHashes,expected);
assert.equal(sidecar.variant,arm.variant);assert.equal(sidecar.sourceRevision,b.revision);assert.equal(sidecar.compiledBuildRevision,b.revision);assert.equal(sidecar.playwrightReportSha256,attached.reportSha256);
for(const [key,digest]of [['manifestFile','manifestFileSha256'],['inventorySummaryFile','inventorySummarySha256'],['qualifiedReceiptFile','qualifiedReceiptSha256']]){
assert.equal(sidecar[digest],b[digest]);assert.equal(sidecar[key],path.join('inventory',path.basename(b[key])));await pinned(path.resolve(directory,sidecar[key]),b[digest]);
}assert.equal(sidecar.aggregateSha256,b.aggregateSha256);assert.equal(sidecar.command.code,0);assert.equal(sidecar.command.signal,null);assert.equal(sidecar.command.watchdogExpired,undefined);
assert.ok(e.startedAtMs>=sidecar.command.startedAtMs&&e.finishedAtMs<=sidecar.command.finishedAtMs,'Raw proof must belong inside its owning command');
assert.equal(await realpath(sidecar.command.executable),await realpath(r.runtimeChecks.node.executable));
assert.equal(preflight.pathNode.version,'v26.4.0');assert.equal(preflight.pathNode.PATH,sidecar.command.environmentOverrides.PATH);
assert.equal(preflight.pathNode.cwd,b.appRoot);assert.equal(preflight.pathNode.canonicalCwd,await realpath(b.appRoot));
assert.equal(preflight.pathNode.canonicalExecutable,await realpath(r.runtimeChecks.node.executable));assert.equal(preflight.pathNode.approvedCanonicalExecutable,preflight.pathNode.canonicalExecutable);
assert.deepEqual(sidecar.command.args,['node_modules/@playwright/test/cli.js','test','--config',CONFIG]);assert.equal(sidecar.command.cwd,r.harnessRoot);assert.equal(sidecar.command.resourceWatchdogMillis,1800000);
assert.equal(sidecar.command.environmentOverrides.PAPERCLIP_INGESTION_APP_ROOT,b.appRoot);assert.equal(sidecar.command.environmentOverrides.PAPERCLIP_INGESTION_VARIANT,arm.variant);
assert.equal(sidecar.command.environmentOverrides.PAPERCLIP_INGESTION_EVIDENCE_DIR,directory);assert.equal(Number(sidecar.command.environmentOverrides.PAPERCLIP_E2E_PORT),port.port);
assert.equal(hash(await readFile(path.join(directory,'playwright.log'))),sidecar.command.logSha256);return e;
}
async function arm(context,row,stageDirectory){
const {r,h}=context,variant=row.variant.toUpperCase(),label=`${row.stage}-pair${row.pair}-${row.index}-${row.role}-${variant}`,directory=path.join(stageDirectory,label);await mkdir(directory);
await source(context);for(const key of AMBIENT)assert.ok(!process.env[key],`Ambient override rejected:${key}`);
const port=await portProbe(),env={PAPERCLIP_INGESTION_APP_ROOT:r.perVariantBuilds[variant].appRoot,PAPERCLIP_INGESTION_EVIDENCE_DIR:directory,
PAPERCLIP_E2E_PORT:String(port),PAPERCLIP_INGESTION_TRANSPORT:'sandbox-remote-with-local-stop',PAPERCLIP_INGESTION_VARIANT:variant,
PAPERCLIP_INGESTION_DRIVER_DEPENDENCIES:'Independent frozen companiondriver; independent normal qualified appbuild'};
await h.physicalQuietGuard({receiptFile:path.join(directory,'physical-quiet-pre.json'),phase:'pre-arm',armStage:label});
const versions=await runtime(r),checked=await buildCheck(context,variant);env.PAPERCLIP_RUNNER_BINARY=checked.binary;env.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH=checked.binary;
env.PATH=process.env.PATH;const launchEnvironment={...process.env,...env},pathNode=await assertPathNode(launchEnvironment,r.runtimeChecks.node.executable,r.perVariantBuilds[variant].appRoot);
const discovery=execFileSync(process.execPath,['node_modules/@playwright/test/cli.js','test','--config',CONFIG,'--list'],{cwd:r.harnessRoot,env:launchEnvironment,encoding:'utf8',timeout:30000});
assert.match(discovery,/Total: 1 test in 1 file/);assert.match(discovery,/native-event-ingestion-blocked-stop\.journey\.ts/);
await save(path.join(directory,'pre-arm-inventory-check.json'),{releaseSha256:context.releaseSha256,variant,versions,pathNode,discovery,live:checked.live,ui:checked.ui,checkedAtMs:Date.now()});
let command,failure;try{command=await launch(context,directory,env,launchEnvironment);assert.equal(command.code,0);assert.equal(command.signal,null);assert.equal(command.watchdogExpired,undefined);}catch(error){failure=error;}
const cleanupFailures=[];
try{await portProbe(port);await save(path.join(directory,'physical-http-port-check.json'),{port,nodeBindAndListenPassed:true,atMs:Date.now()});}
catch(error){cleanupFailures.push({step:'port',error:String(error)});failure??=error;}
try{await h.physicalQuietGuard({receiptFile:path.join(directory,'physical-quiet-post.json'),phase:'post-arm',armStage:label,primaryFailureAlreadyPresent:Boolean(failure)});}
catch(error){cleanupFailures.push({step:'physical-quiet',error:String(error)});failure??=error;}
if(cleanupFailures.length)await save(path.join(directory,'cleanup-failures.json'),cleanupFailures);if(failure)throw failure;
const attached=await owningEvidence(directory),file=path.join(directory,'blocked-stop-evidence.json');await writeFile(file,attached.bytes,{flag:'wx'});
const inventory=path.join(directory,'inventory');await mkdir(inventory);const b=r.perVariantBuilds[variant],copies={};
for(const [key,digest]of [['manifestFile','manifestFileSha256'],['inventorySummaryFile','inventorySummarySha256'],['qualifiedReceiptFile','qualifiedReceiptSha256']]){
const target=path.join(inventory,path.basename(b[key]));await writeFile(target,await pinned(b[key],b[digest]),{flag:'wx'});copies[key]=path.relative(directory,target);
}const inventoryFile=path.join(directory,'arm-inventory.json');await save(inventoryFile,{schema:'paperclip.native-ingestion.arm-inventory.v1',variant,sourceRevision:b.revision,compiledBuildRevision:b.revision,
playwrightReportSha256:attached.reportSha256,...copies,manifestFileSha256:b.manifestFileSha256,qualifiedReceiptSha256:b.qualifiedReceiptSha256,inventorySummarySha256:b.inventorySummarySha256,aggregateSha256:b.aggregateSha256,command});
const result={...row,variant,label,file,playwrightFile:path.join(directory,'playwright.json'),inventoryFile};await readArm(context,result);result.artifacts=await h.sealArmArtifacts(context.out,result);return result;
}
function identity(context){const r=context.r;return {harnessRevision:r.harnessRevision,companionFiles:r.companionFiles,workloadSha256:r.workloadSha256,precisionRuleSha256:r.precisionRuleSha256,h5HelperSha256:H5SHA,orchestratorSha256:r.orchestratorSha256,runtimeChecks:r.runtimeChecks};}
function pairManifest(arms){return Array.from({length:5},(_,index)=>{
const entry={pair:index+1};for(const role of ['baseline','candidate']){const arm=arms.find(a=>a.pair===index+1&&a.role===role);assert.ok(arm);
for(const [suffix,key]of [['File','file'],['PlaywrightFile','playwrightFile'],['InventoryFile','inventoryFile'],['Artifacts','artifacts']])entry[role+suffix]=arm[key];
}return entry;
});}
async function readSeries(context,stage,pin){
assert.equal(pin.file,path.join(context.out,stage+'.json'));const report=JSON.parse(await pinned(pin.file,pin.sha256));assert.equal(report.schema,SCHEMA);assert.equal(report.stage,stage);assert.equal(report.complete,true);assert.equal(report.passed,true);assert.deepEqual(report.identity,identity(context));
const ownerRelease=JSON.parse(await pinned(report.releaseFile,report.releaseSha256));assert.equal(ownerRelease.authorized,true);assert.equal(ownerRelease.stage,stage);assert.equal(ownerRelease.outputRoot,context.out);
assert.equal(report.releaseFile,path.join(context.out,stage,'root-release.json'));assert.deepEqual(identity({r:ownerRelease}),report.identity);
const expected=schedule(stage,context.p.registration());assert.equal(report.arms.length,expected.length);
const raw=[];for(const [i,a]of report.arms.entries()){for(const k of ['stage','pair','index','role'])assert.equal(a[k],expected[i][k]);assert.equal(a.variant,expected[i].variant.toUpperCase());
assert.deepEqual(await context.h.sealArmArtifacts(context.out,a),a.artifacts);raw.push(await readArm(context,a,report.releaseSha256));
}if(stage!=='qualification-b0'){assert.deepEqual(report.pairManifest,pairManifest(report.arms));await context.h.verifyRawArtifactSeals(context.out,report.pairManifest);}
if(stage.endsWith('-aa')){assert.equal(report.noiseFile,path.join(context.out,stage+'.noise.json'));
assert.deepEqual(JSON.parse(await pinned(report.noiseFile,report.noiseSha256)),report.noise);
assert.deepEqual(report.noise,context.p.noiseEnvelope(logicalPairs(report.arms,raw)));context.p.assertNoiseEnvelope(report.noise);
}
return {report,raw};
}
function logicalPairs(arms,raw){return Array.from({length:5},(_,i)=>['baseline','candidate'].map(role=>raw[arms.findIndex(a=>a.pair===i+1&&a.role===role)]));}
async function prerequisites(context){
const index=STAGES.indexOf(context.r.stage),prior={},companies=new Set(),runs=new Set();let completed=0;
for(const stage of STAGES.slice(0,index)){const value=await readSeries(context,stage,context.r.priorStageReports?.[stage]??{});assert.ok(value.report.completedAtMs>completed);completed=value.report.completedAtMs;prior[stage]=value.report;
if(stage!=='qualification-b0'&&!stage.endsWith('-aa')){const noiseStage=stage.startsWith('b1')?'b1-aa':'b0-aa';
const recomputed=context.p.comparePairs(logicalPairs(value.report.arms,value.raw),prior[noiseStage].noise);assert.deepEqual(value.report.comparison,recomputed);assert.ok(Object.values(recomputed).every(row=>row.pass));
}for(const e of value.raw){assert.ok(!companies.has(e.companyId)&&!runs.has(e.runId),'An earlier arm was reused');companies.add(e.companyId);runs.add(e.runId);}
}if(index>=3)for(const stage of ['b0-aa','b1-aa']){
const pin=context.r.noiseReports?.[stage];assert.equal(pin?.file,prior[stage].noiseFile);assert.equal(pin?.sha256,prior[stage].noiseSha256);
context.p.assertNoiseEnvelope(JSON.parse(await pinned(pin.file,pin.sha256)));
}
return {prior,companies,runs,lastCompletedAtMs:completed};
}
async function execute(context){
const {r,h,p}=context,prior=await prerequisites(context),directory=path.join(context.out,r.stage);await mkdir(directory);await writeFile(path.join(directory,'root-release.json'),await pinned(context.releaseFile,context.releaseSha256),{flag:'wx'});
const arms=[],raw=[];try{for(const row of context.rows){const a=await arm(context,row,directory),e=await readArm(context,a);
assert.ok(!prior.companies.has(e.companyId)&&!prior.runs.has(e.runId),'Every arm needs a fresh company and run');prior.companies.add(e.companyId);prior.runs.add(e.runId);arms.push(a);raw.push(e);await save(path.join(directory,`arm${arms.length}-sealed.json`),a);
}await source(context);let noise,comparison;const pairs=r.stage==='qualification-b0'?undefined:logicalPairs(arms,raw),manifest=pairs?pairManifest(arms):undefined;
if(manifest)await h.verifyRawArtifactSeals(context.out,manifest);else assert.deepEqual(await h.sealArmArtifacts(context.out,arms[0]),arms[0].artifacts);
if(r.stage.endsWith('-aa')){noise=p.noiseEnvelope(pairs);}else if(pairs){const baseline=r.stage.startsWith('b1')?'b1-aa':'b0-aa';comparison=p.comparePairs(pairs,prior.prior[baseline].noise);}
let noiseFile,noiseSha256;if(noise){noiseFile=path.join(context.out,r.stage+'.noise.json');await save(noiseFile,noise);noiseSha256=hash(await readFile(noiseFile));}
const passed=noise?noise.suitable&&noise.noiseFrozen:!comparison||Object.values(comparison).every(row=>row.pass),report={schema:SCHEMA,scope:'Separate blocked-write control qualification/nonregression only; not main application performance acceptance',stage:r.stage,
identity:identity(context),releaseFile:path.join(directory,'root-release.json'),releaseSha256:context.releaseSha256,complete:true,passed,arms,pairManifest:manifest,noise,noiseFile,noiseSha256,comparison,completedAtMs:Date.now()};
await save(path.join(context.out,r.stage+'.json'),report);if(noise)p.assertNoiseEnvelope(noise);assert.equal(passed,true,'Companion control nonregression failed');
}catch(error){await save(path.join(directory,'failure.json'),{stage:r.stage,complete:false,error:String(error),stack:error.stack,completedArms:arms,atMs:Date.now()});throw error;}
}
async function main(){
if(!process.argv.includes('--execute')&&!process.argv.includes('--validate-release')){console.log(JSON.stringify({draft:true,executionReleased:false,stages:STAGES,registeredTimingArms:50,qualificationArms:1,
note:'Exact companion source/revision, calibration rules and runtime command outputs are not frozen by this draft. No candidate or qualification result is invented.',
requiredReleaseFields:['authorized','stage','outputRoot','orchestratorSha256','h5HelperFile','h5HelperSha256','harnessRoot','harnessRevision','inheritedH5cRevision','lockSha256','companionFiles(9 exact SHA256)','workloadSha256','precisionRuleSha256','resourceWatchdogMillis:1800000','runtimeChecks(exact executable/args/expected for node26.4.0,pnpm9.15.4,postgres,chromium,rustc,cargo)','perVariantBuilds(actual qualified H5c-compatible descriptors)','priorStageReports(exact passed owner/raw/proof/seal reports by file/SHA256)','noiseReports(exact b0-aa and b1-aa noise JSON file/SHA256 pins before comparisons)' ]},null,2));return;}
const c=await load(option('release'),option('release-sha256'),option('stage'),option('output-root'));await prerequisites(c);
if(!process.argv.includes('--execute')){console.log(JSON.stringify({valid:true,sourceAndBindingsOnly:true,arms:c.rows.length,noRuntimeQualification:true}));return;}
await mkdir(c.out,{recursive:true});await execute(c);
}
if(process.argv[1]&&path.resolve(process.argv[1])===SELF)await main();
scripts/native-ingestion/blocked-stop/sources.json
+122
View File
@@ -0,0 +1,122 @@
{
"schema": "paperclip.native-ingestion.blocked-stop-helper-sources.v3",
"scope": "Exact frozen companion v3 sources and reviewed helper; one functional qualification is not a timing/noise/performance pass.",
"repository": "https://github.com/paperclipai/paperclip.git",
"deliveryLockSha256AtPackaging": "7540075e96f9a93924a1e54775c17346c5760dd5d02902d83fb5228f918996ba",
"deliveryCheckoutWasExecutedDriver": false,
"mixedDeliveryTreeCannotReplaceEitherFrozenDriver": true,
"recordedVmRuntime": {
"node": "v26.4.0",
"pnpm": "9.15.4",
"otherRuntimeOutputs": "Exact absolute executable, argument and output pins from sealed runtime receipts; never guessed."
},
"rawB2Role": "ca7cba171cd5198ed569d75012edc858041cbc44 is unoptimized B2 provenance, not a final candidate.",
"frozenDriver": {
"commit": "bc33f09cae26404352848169d92a20c89480a5f5",
"tree": "c84d96153d87ef3a31265c3de01568714688200c",
"parent": "6e5349cdbd5e3a197877f1082f84a17e866c03e7",
"tag": "native-ingestion-blocked-stop-v3",
"tagObject": "a49c24dc2a690764d1cfcc080077f90485218df7",
"tagPeeledCommit": "bc33f09cae26404352848169d92a20c89480a5f5",
"companionNineFileConcatOrder": "Lexicographically sorted repository-relative paths in files",
"companionNineFileSha256": "0dea53d3ec7bbf63c45c628e39c65983181d5e19f8bfae872cf78fa85e7831e8",
"frozenH5cNineFileSha256": "2265aabe9c7344f5d4b9c8b08a5c6bbd5571a6e2de9ae7b72a8d1733250e609f",
"driverLockSha256": "54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8",
"protocol": {
"PROTOCOL": "blocked-stop-history-v1",
"WORKLOAD_SHA256": "0fc383bca459682042be313e1c3243746b7428ab3b94c2a6ab3dc81eee252522",
"PRECISION_RULE_SHA256": "a93bb828c91eae6f6fcb3b5f5533465475d57aaa03c79485b51d879af5d14669",
"arms": 50
}
},
"sourceFreezeReceiptSha256": "210002a6699059ff9b5f74a49558c9f7f0f5866cd9ed3f786507b3ee12fc7b38",
"sourceFiles": [
{
"file": "tests/e2e/fixtures/native-ingestion-blocked-stop-authority.ts",
"sha256": "cd94f1f0dcd49a17ea341918954a115a1627357b0b78ea9d94905190a0973514",
"bytes": 10309,
"byteIdenticalToFrozenDriver": true
},
{
"file": "tests/e2e/fixtures/native-ingestion-blocked-stop-clock.ts",
"sha256": "530bee16cbbec6801d0efe78942917fad63b4e2e5410d273dd4510a62ea87721",
"bytes": 6976,
"byteIdenticalToFrozenDriver": true
},
{
"file": "tests/e2e/fixtures/native-ingestion-blocked-stop-db.ts",
"sha256": "8d9c637048280feb495f834e0dcd9e35cb4fbc98ded98a8e1d296525f1cbcc96",
"bytes": 2904,
"byteIdenticalToFrozenDriver": true
},
{
"file": "tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.mjs",
"sha256": "fcb83d72d6ab609fbd6a7e765a756989ece681f27d97675271f8d1f2c9c2ee0c",
"bytes": 28889,
"byteIdenticalToFrozenDriver": true
},
{
"file": "tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.test.mjs",
"sha256": "91e134430df46488ad19adcdb17591c688e069e747095e4a66361ece5fb33484",
"bytes": 27170,
"byteIdenticalToFrozenDriver": true
},
{
"file": "tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs",
"sha256": "383e43ce1e210a35ec920d8e3e5d763e7502c61db2057cf8a8d3ba8270b6dbd3",
"bytes": 1502,
"byteIdenticalToFrozenDriver": true
},
{
"file": "tests/e2e/fixtures/native-ingestion-blocked-stop.md",
"sha256": "e5fe99b4ba353b77e76507bae6f9095a4475f61c9badbfa27c75e6ca8470f77c",
"bytes": 16597,
"byteIdenticalToFrozenDriver": true
},
{
"file": "tests/e2e/native-event-ingestion-blocked-stop.journey.ts",
"sha256": "8b3f8b3a9704c77a6ddd14d996ebd5cf34a5fe76883c51060edee208e82211a0",
"bytes": 24850,
"byteIdenticalToFrozenDriver": true
},
{
"file": "tests/e2e/playwright-native-ingestion-blocked-stop.config.ts",
"sha256": "e9766cfd60c2eea5c1e14bedefa95706e5d87c03efebb8229a6536f00f441706",
"bytes": 383,
"byteIdenticalToFrozenDriver": true
}
],
"helper": {
"file": "native-ingestion-run-blocked-stop-v3.mjs",
"originalTaskFile": "/private/tmp/native-ingestion-run-blocked-stop-v3.mjs",
"bytes": 27973,
"sha256": "5748b6c1d271cf9ab270d5473873b85da295ffb03bb55ebc63693369c9b1a29d",
"packagingChanges": []
},
"checkedH5cDependency": {
"sha256": "6b6376b01de6cdc15ca05343a0bdd5c00424f1d6c546b9b4d61cd76716b253c4",
"packagedSource": "scripts/native-ingestion/h5c/native-ingestion-run-h5c-browser.mjs",
"recordedPath": "/home/ubuntu/native-ingestion/helpers/native-ingestion-run-h5c-browser.mjs",
"purpose": "Six checked build/inventory/UI/quiet/raw-seal exports only; original stage plan and loadRelease are not invoked."
},
"mainCampaignBoundary": "Companion bc33 inherits H5c e431/main9=2265; main H5d2df/main9=e1 is separate. Install each immutable driver independently.",
"priorFailedQualifications": {
"v1": "200396/936d failed unsupported shared PGID assumption before epoch; API200/port did not prove physical quiet; failed raw and exact owned cleanup retained.",
"v2": "6e5349/dc693 failed unconditional retirement receipt although retained identity is supported; transcript/reload/lease gate unreached; failed raw retained, no timing reuse."
},
"qualificationAsOfUtc": "2026-11-04T09:30:00Z",
"qualificationFacts": {
"onePassed": true,
"preludeDeltas": 1000,
"canonicalStopPrefixDeltas": 12,
"results": 0,
"heldDurationMs": 1010,
"heldSamples": 15,
"bothPidsAndBothGroupsAbsent": true,
"ownIntentAndAckAudits": true,
"retirementState": "cleared-identity-with-authoritative-retirement",
"leaseReleasedReloadAndStrictQuiet": true,
"manualSignals": false,
"noiseOrPerformancePassed": false
}
}
scripts/native-ingestion/h5d/README.md
+122
View File
@@ -0,0 +1,122 @@
# Frozen H5d application driver
This package retains the six exact H5d source overlays and six reviewed helpers.
`sources.json` records their hashes. Execute immutable driver **2df9897e1ce2ca90be29fa097d776c3766e217ad**,
tag `native-ingestion-h5d`, annotated object `47ef6d1a5d81c31d4602aad50666f67c58f4cdb4`.
Its measurement hash is `e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4`;
its lock is `54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8`.
The mixed delivery checkout has a different lock and is not an executed driver.
The [blocked Stop companion](../blocked-stop/README.md) uses a separate immutable
checkout inheriting H5c measurement `2265aabe…`, not H5d measurement `e1d8930c…`.
## Recorded qualification and limits
As of 2026-11-04 09:30 UTC, H5d SMALL passed: 48 deltas, three results and a
nine-delta Stop prefix with no result; all 40 sealed files and cleanup were checked.
FULL's owning Playwright test passed in 853,339 ms with strict physical quiet.
The frozen assessor retains the original loaded 120-second streaming miss:
41/512 foreground markers near 119.841 seconds. The original 900-second duration
gate passed. All 51,200 loaded deltas and 100 results were independently verified,
including 43 recovered `provider_transport_failed` failures, 402,584 ms through
verified completion, and all leases closed. These are qualification facts,
not accepted A/A noise or evidence of application improvement.
The existing-history workload is one foreground run with 1,000 emitted and
verified prelude deltas under nine fresh background runs. It does not create ten
history-bearing runs. Retain all source performance results separately from app
measurement. No application A/A campaign had passed at the recorded cutoff.
The lead's dated application report owns subsequent campaign progress.
The old offline reader `968fbbfc…` failed an unsupported `runIds[0]` assumption.
The new foreground reader `b1e74ffd…` correlates the unique prelude run through
actual durable/provider-clock nonce, marker, activation and authority identities.
Root and peer reviewed its old-red, permutation and nine-negative checks and its
readback of a COPY of the same owning PASS raw report. Original failure, first
seal and raw bytes remain preserved. This repairs a qualification postreader;
it changes no SUT or measurement source and cannot promote a failed owning test.
The old reader is not packaged as the current reader.
## Independent setup
Use a disposable Linux host with the recorded Node **v26.4.0**, pnpm **9.15.4**,
PostgreSQL, Chromium and native toolchain. Node 24 is the local Mac check toolchain.
Capture actual executable paths and exact outputs before release: `node --version`,
`node -p process.execPath`, `pnpm --version`, `rustc --version`, `cargo --version`,
and the actual PostgreSQL and Chromium executables with `--version`. Runtime
release pins must use those values. Never infer a version from this package.
```sh
TOOLS_CHECKOUT="$(git rev-parse --show-toplevel)"
BASE=/home/ubuntu/native-ingestion
DRIVER_ROOT="$BASE/harness-h5d"
HELPERS_ROOT="$BASE/helpers"
git -C "$TOOLS_CHECKOUT" fetch origin tag native-ingestion-h5d
test "$(git -C "$TOOLS_CHECKOUT" rev-parse 'native-ingestion-h5d^{commit}')" = 2df9897e1ce2ca90be29fa097d776c3766e217ad
git -C "$TOOLS_CHECKOUT" worktree add --detach "$DRIVER_ROOT" native-ingestion-h5d
pnpm --dir "$DRIVER_ROOT" install --frozen-lockfile --offline --ignore-scripts
mkdir -p "$HELPERS_ROOT"
cp "$TOOLS_CHECKOUT"/scripts/native-ingestion/h5d/*.mjs "$HELPERS_ROOT/"
cp "$TOOLS_CHECKOUT"/scripts/native-ingestion/h5d/*.py "$HELPERS_ROOT/"
node "$HELPERS_ROOT/native-ingestion-run-h5d-browser.mjs"
```
Use a fresh driver path and install its own locked tools. Verify copied bytes
against `sources.json`. Build every measured application independently with its
normal locked install and build. Recorded aligned builds are B0 `bac53ec3…`, B1
`07061a60…`, and unoptimized B2 `ca7cba17…`; qualified build receipt SHA is
`a1259ba760f3cbbe44817e333f9f620bcf3769fb0a6c16f9ce2f0f5b2ed60d94`.
Validate all four artifact roots, UI shadow absence and native inputs every arm.
`ca7cba17` is not a final B2 candidate; a new candidate requires actual immutable
source, normal build and qualified descriptor. No candidate is invented here.
The exact setup/readiness wrappers use recorded absolute paths and new output
namespaces: `h5d/setup` and `h5d/setup-full-b0`. Read them before use; they perform
live inventory/quiet checks. After an explicit quiet-window execution release,
the recorded functional commands are:
```sh
mkdir -p "$BASE/evidence/h5d/setup"
node "$HELPERS_ROOT/native-ingestion-h5d-linux-readiness.mjs"
python3 "$HELPERS_ROOT/native-ingestion-linux-h5d-qualification.py" --tree B0 --name small-b0
# In a separate released quiet window, with its distinct setup/output namespace:
mkdir -p "$BASE/evidence/h5d/setup-full-b0"
node "$HELPERS_ROOT/native-ingestion-h5d-full-linux-readiness.mjs"
python3 "$HELPERS_ROOT/native-ingestion-linux-h5d-qualification.py" --full --tree B0 --name full-b0
```
The qualification wrapper runs real production-built app/browser/database/provider
paths. Do not execute it during another timing window. Its 30-minute resource
watchdog does not enlarge the original 120/180/900-second gates or H5 observation
600/1200-second limits. Qualification flags and source caps remain unchanged.
On a COPY of a completed qualification output, run the current offline reader,
then frozen validator. Omit `--full` for SMALL. These read evidence, not the SUT:
```sh
python3 "$HELPERS_ROOT/native-ingestion-extract-h5d-qualification-foreground.py" "$QUALIFICATION_COPY" --revision=bac53ec3bfeaf6546e7aaeaee516487b30534654 --full
node "$HELPERS_ROOT/native-ingestion-validate-h5d-evidence.mjs" "$QUALIFICATION_COPY" "$DRIVER_ROOT"
```
## Fresh measurements
The runner's default invocation is a dry run. A byte-frozen concrete release
binds source/helper hashes, exact build/native/UI inventories, six actual runtime
outputs, output ownership, qualification and prior noise/series hashes.
```sh
node "$HELPERS_ROOT/native-ingestion-run-h5d-browser.mjs" --release="$RELEASE_JSON" --stage=browser-aa-b0 --output-root="$EVIDENCE_ROOT" --validate-release
```
Only the sole benchmark owner executes the released stage by replacing
`--validate-release` with `--execute`, using the same frozen release bytes. Run
`browser-aa-b0`, `browser-aa-b1`, `browser-b0-b1`, `browser-b1-b2`, `browser-b0-b2`
in order: five fresh alternating pairs each, 50 full arms. Both suitable A/A
artifacts and their exact absolute/relative noise must freeze before comparisons
or further optimization. No qualification arm is reused. Stop and preserve raw
on any owner, cleanup or suitability failure. No replacement sample or looser
budget follows from a failure. A final candidate must pass all original SLAs
and the separate default `browser-finalize-h4e` proof after both comparisons.
The main and companion 50-arm campaigns have separate workloads and noise.
Diagnostic profiles and counter instrumentation are separate; the unqualified
counter wrapper is intentionally absent from this package.
scripts/native-ingestion/h5d/native-ingestion-extract-h5d-qualification-foreground.py
+74
View File
@@ -0,0 +1,74 @@
import pathlib,json,base64,argparse,hashlib
def assert_history_prelude(case):
pre=case['preludeProof']
foreground=[v for v in case['durable'] if v['token']==case['nonce'] and v['markerCode']==case['markerCode']]
clocks=[v for v in case['providerClocks'] if v['nonce']==case['nonce']]
activation=[v for v in case['activations'] if v['index']==0 and v['token']==case['nonce']]
authority=[v for v in case['controllerAuthority']['rows'] if v['run_id']==pre['runId']]
assert len(foreground)==len(clocks)==len(activation)==len(authority)==1
assert pre['runId']==foreground[0]['runId']==clocks[0]['runId'] and pre['runId'] in case['runIds']
assert authority[0]['issue_id']==activation[0]['issueId']
assert pre['individualDeltas']==1000 and pre['uniqueSourceIdentities']==1000
assert pre['verifiedAtMs']<case['epochMs'] and pre['retainedGlobalHead']>=1000
p=argparse.ArgumentParser();p.add_argument('directory');p.add_argument('--revision',required=True);p.add_argument('--full',action='store_true');a=p.parse_args();d=pathlib.Path(a.directory);r=json.loads((d/'playwright.json').read_text());attachments=[]
def visit(value):
if isinstance(value,list):
for child in value:visit(child)
elif isinstance(value,dict):
attachments.extend(item for item in value.get('attachments',[])if item.get('name')=='native-ingestion-evidence')
for key,child in value.items():
if key!='attachments':visit(child)
visit(r);assert len(attachments)==1,len(attachments);item=attachments[0]
raw=base64.b64decode(item['body'])if item.get('body')else pathlib.Path(item['path']).read_bytes();report=json.loads(raw);(d/'native-ingestion-evidence.json').write_bytes(raw)
assert report.get('complete')is True and not report.get('primaryFailure'),report.get('primaryFailure')
assert report['sourceRevision']==report['compiledBuildRevision']==a.revision
assert report['protocolVersion']=='v5c-independent-agents-complete-settlement-original-sla'
assert report['observationMode']=='h5c-observe-original-sla-independent-agents-settlement' and report['observationComplete'] is True
assert report['measurementSha256']=='e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4'
assert report['transport']=='sandbox-remote-with-local-stop'
assert report['admissionProtocol']=='two-at-a-time-provider-and-clock-readiness-before-common-epoch'
assert report['setupActivationConcurrency']==2
assert report['harnessProvenance']['sourceRevision']=='2df9897e1ce2ca90be29fa097d776c3766e217ad'
assert report['qualificationOnly']is(not a.full)
assert report['serverArtifact']=='compiled' and report.get('diagnosticOnly')is False
assert report['cleanup']=={'errors':[],'complete':True}
expected=['paced','burst16','burst128','burst512','existing-history','operator-stop']if a.full else['qualification','operator-stop'];assert [c['name']for c in report['cases']]==expected
ids=report['remote']['agentIds'];setup=report['remote']['agentSetup']
assert len(ids)==(100 if a.full else 3) and len(set(ids))==len(ids)
assert setup['complete'] is True and setup['concurrency']==8 and setup['count']==len(ids)
assert len(setup['attempts'])==len(ids) and all(x['status']=='fulfilled'for x in setup['attempts'])
assert all(ids[x['index']]==x['agentId']for x in setup['attempts'])
rows=[]
for case in report['cases']:
stop=case['name']=='operator-stop';assert case['persistedAfterReload']is True
assert case['setupActivationConcurrency']==2 and case['participants']==case['runs']
assert sorted(x['index']for x in case['activations'])==list(range(case['runs']))
assert all(x['status']=='fulfilled'for x in case['activations'])
waves=case['activationWaves'];assert len(waves)==(case['runs']//2)
assert all(1<=len(w['indices'])<=2 and w['statuses']==['fulfilled']*len(w['indices'])for w in waves)
if not a.full and not stop:assert(case['runs'],case['events'],case['intervalMs'])==(3,16,25)
assert len(case['durable'])==case['runs']==len(case['runIds'])
if case['name']=='existing-history':
assert_history_prelude(case)
assert case['matchedVisibleClocks']==len(case['visible']) and len(case['visible'])>=(3 if stop else case['events'])
authority=case['controllerAuthority'];assert authority['kind']==('host-file-journal'if stop else'database-controller-checkpoint')
assert authority['observedReadOnly']=='on'and authority['observedIsolation']=='repeatable read'
assert len(authority['rows'])==case['runs']and authority['connectionClosedAtMs']<case['epochMs']
for proof in case['durable']:
assert proof['matchedSourceClocks']==proof['individualDeltas']
assert proof['uniqueSourceIdentities']==proof['individualDeltas']
if not stop:assert proof['individualDeltas']==case['events']
if stop:
s=case['localStopProof'];db=s['database']['rows'][0]
assert db['status']=='cancelled'and db['cancellation_dispatch_state']=='acknowledged'
assert all(s['processes'][k]is True for k in['runnerAbsent','groupAbsent','providerAbsent'])
assert s['lease']['releasedAt']and s['lease']['cleanupStatus']!='failed'
if not db['process_pid']and not db['process_group_id']:assert int(db['process_stopped_seq'])>int(db['process_identity_seq'])
else:
assert len(set(x['agent_id']for x in case['controllerAuthority']['rows']))==case['runs']
assert all(x['agent_id']in ids for x in case['controllerAuthority']['rows'])
assert len(case['remoteLeases'])==case['runs']
assert all(lease['releasedAt']and lease['cleanupStatus']=='success'for lease in case['remoteLeases'])
rows.append({'name':case['name'],'runs':case['runs'],'eventsPerRun':case['events'],'visibleMarkers':len(case['visible']),'individualDeltas':sum(v['individualDeltas']for v in case['durable']),'sourceClocksMatched':sum(v['matchedSourceClocks']for v in case['durable']),'runIds':case['runIds'],'transport':case['transport'],'setupWaves':len(case['activationWaves']),'setupMs':case['allReadyObservedAtMs']-case['setupReadinessStartedAtMs'],'drainThroughTerminalMs':case['drainThroughTerminalMs']})
summary={'complete':True,'qualificationOnly':True,'performanceAcceptanceEligible':False,'sourceRevision':a.revision,'harnessRevision':report['harnessProvenance']['sourceRevision'],'reportSha256':hashlib.sha256(raw).hexdigest(),'fullFixture':a.full,'cases':rows,'cleanup':report['cleanup']};(d/'independent-qualification-check.json').write_text(json.dumps(summary,indent=2)+'\n');print(json.dumps(summary))
scripts/native-ingestion/h5d/native-ingestion-h5d-full-linux-readiness.mjs
+28
View File
@@ -0,0 +1,28 @@
import assert from 'node:assert/strict';
import {readFile,writeFile,realpath} from 'node:fs/promises';
import {execFileSync} from 'node:child_process';
import {createHash} from 'node:crypto';
import path from 'node:path';
const base='/home/ubuntu/native-ingestion',helper=base+'/helpers/native-ingestion-run-h5d-browser.mjs',out=base+'/evidence/h5d/setup-full-b0';
const sha=b=>createHash('sha256').update(b).digest('hex');
assert.equal(sha(await readFile(helper)),'4afaa8ffcd764b498bada29c8916715b73d34462fd49152e44ab8437ffb330eb');
const h=await import(helper);
const quiet=await h.physicalQuietGuard({receiptFile:out+'/linux-quiet-readiness.json',phase:'pre-arm',armStage:'H5c-SETUP-ONLY'});
const receiptFile=base+'/aligned-4a83-builds-qualified.json',summaryFile=base+'/evidence/aligned-4a83/environment/artifact-inventory-summary.json';
const receipt=JSON.parse(await readFile(receiptFile,'utf8')),summary=JSON.parse(await readFile(summaryFile,'utf8'));
const results=[];
for(const [variant,directory] of [['B0','aligned-baseline'],['B1','aligned-cursors']]) {
const row=receipt.results.find(v=>v.variant===variant),i=summary.inventories[variant],native=receipt.nativeStageHashes.find(v=>v.variant===variant);
const descriptor={appRoot:base+'/'+directory,revision:row.revision,qualifiedReceiptFile:receiptFile,qualifiedReceiptSha256:sha(await readFile(receiptFile)),inventorySummaryFile:summaryFile,inventorySummarySha256:sha(await readFile(summaryFile)),manifestFile:path.join(path.dirname(summaryFile),i.manifest),manifestFileSha256:i.manifestSha256,aggregateSha256:i.aggregateSha256,binarySha256:native.binarySha256,rustSourceTree:receipt.rustSourceTree};
const q=await h.validateVariantBuild(variant,descriptor),live=await h.validateActualInventory(descriptor,q.manifest),ui=await h.assertNoPreferredUiShadow(descriptor.appRoot);
assert.equal(execFileSync('git',['rev-parse','HEAD'],{cwd:descriptor.appRoot,encoding:'utf8'}).trim(),descriptor.revision);
assert.equal(execFileSync('git',['status','--short'],{cwd:descriptor.appRoot,encoding:'utf8'}).trim(),'');
assert.equal(JSON.parse(await readFile(path.join(descriptor.appRoot,'server/dist/build-info.json'),'utf8')).commit,descriptor.revision);
const target=path.join(descriptor.appRoot,'packages/paperclip-runner/runner/target'),binary=path.join(target,'release/paperclip-runnerd');
assert.equal(await realpath(target),native.targetRealPath);assert.equal(sha(await readFile(binary)),native.binarySha256);
results.push({variant,descriptor,live,ui,nativeTargetRealPath:await realpath(target)});
}
const pathNode=await h.assertPathNode(process.env,process.execPath,base+'/aligned-baseline');
const data={setupOnly:true,noSutBrowserProviderOrEvents:true,performanceAcceptanceEligible:false,atMs:Date.now(),helperSha256:sha(await readFile(helper)),harnessRevision:'2df9897e1ce2ca90be29fa097d776c3766e217ad',quiet,path:pathNode,results};
await writeFile(out+'/linux-readiness.json',JSON.stringify(data,null,2)+'\n',{flag:'wx'});
console.log(JSON.stringify({complete:true,setupOnly:true,noSutBrowserProviderOrEvents:true,variants:results.map(v=>({variant:v.variant,fileCount:v.live.fileCount,aggregate:v.live.aggregateSha256,uiShadowAbsent:v.ui.preferredIndexAbsent})),quiet:quiet.passed,nodeVersion:process.version}));
scripts/native-ingestion/h5d/native-ingestion-h5d-linux-readiness.mjs
+28
View File
@@ -0,0 +1,28 @@
import assert from 'node:assert/strict';
import {readFile,writeFile,realpath} from 'node:fs/promises';
import {execFileSync} from 'node:child_process';
import {createHash} from 'node:crypto';
import path from 'node:path';
const base='/home/ubuntu/native-ingestion',helper=base+'/helpers/native-ingestion-run-h5d-browser.mjs',out=base+'/evidence/h5d/setup';
const sha=b=>createHash('sha256').update(b).digest('hex');
assert.equal(sha(await readFile(helper)),'4afaa8ffcd764b498bada29c8916715b73d34462fd49152e44ab8437ffb330eb');
const h=await import(helper);
const quiet=await h.physicalQuietGuard({receiptFile:out+'/linux-quiet-readiness.json',phase:'pre-arm',armStage:'H5c-SETUP-ONLY'});
const receiptFile=base+'/aligned-4a83-builds-qualified.json',summaryFile=base+'/evidence/aligned-4a83/environment/artifact-inventory-summary.json';
const receipt=JSON.parse(await readFile(receiptFile,'utf8')),summary=JSON.parse(await readFile(summaryFile,'utf8'));
const results=[];
for(const [variant,directory] of [['B0','aligned-baseline'],['B1','aligned-cursors']]) {
const row=receipt.results.find(v=>v.variant===variant),i=summary.inventories[variant],native=receipt.nativeStageHashes.find(v=>v.variant===variant);
const descriptor={appRoot:base+'/'+directory,revision:row.revision,qualifiedReceiptFile:receiptFile,qualifiedReceiptSha256:sha(await readFile(receiptFile)),inventorySummaryFile:summaryFile,inventorySummarySha256:sha(await readFile(summaryFile)),manifestFile:path.join(path.dirname(summaryFile),i.manifest),manifestFileSha256:i.manifestSha256,aggregateSha256:i.aggregateSha256,binarySha256:native.binarySha256,rustSourceTree:receipt.rustSourceTree};
const q=await h.validateVariantBuild(variant,descriptor),live=await h.validateActualInventory(descriptor,q.manifest),ui=await h.assertNoPreferredUiShadow(descriptor.appRoot);
assert.equal(execFileSync('git',['rev-parse','HEAD'],{cwd:descriptor.appRoot,encoding:'utf8'}).trim(),descriptor.revision);
assert.equal(execFileSync('git',['status','--short'],{cwd:descriptor.appRoot,encoding:'utf8'}).trim(),'');
assert.equal(JSON.parse(await readFile(path.join(descriptor.appRoot,'server/dist/build-info.json'),'utf8')).commit,descriptor.revision);
const target=path.join(descriptor.appRoot,'packages/paperclip-runner/runner/target'),binary=path.join(target,'release/paperclip-runnerd');
assert.equal(await realpath(target),native.targetRealPath);assert.equal(sha(await readFile(binary)),native.binarySha256);
results.push({variant,descriptor,live,ui,nativeTargetRealPath:await realpath(target)});
}
const pathNode=await h.assertPathNode(process.env,process.execPath,base+'/aligned-baseline');
const data={setupOnly:true,noSutBrowserProviderOrEvents:true,performanceAcceptanceEligible:false,atMs:Date.now(),helperSha256:sha(await readFile(helper)),harnessRevision:'2df9897e1ce2ca90be29fa097d776c3766e217ad',quiet,path:pathNode,results};
await writeFile(out+'/linux-readiness.json',JSON.stringify(data,null,2)+'\n',{flag:'wx'});
console.log(JSON.stringify({complete:true,setupOnly:true,noSutBrowserProviderOrEvents:true,variants:results.map(v=>({variant:v.variant,fileCount:v.live.fileCount,aggregate:v.live.aggregateSha256,uiShadowAbsent:v.ui.preferredIndexAbsent})),quiet:quiet.passed,nodeVersion:process.version}));
scripts/native-ingestion/h5d/native-ingestion-linux-h5d-qualification.py
+66
View File
@@ -0,0 +1,66 @@
import pathlib,subprocess,json,os,socket,datetime,hashlib,traceback,argparse
p=argparse.ArgumentParser();p.add_argument('--full',action='store_true');p.add_argument('--tree',choices=['B0','B1','B2'],default='B2');p.add_argument('--name',required=True);a=p.parse_args()
base=pathlib.Path('/home/ubuntu/native-ingestion');repo=base/'paperclip';h=base/'harness-h5d';out=base/'evidence/h5d/qualification'/a.name;out.mkdir(parents=True,exist_ok=False)
pins={'B0':('aligned-baseline','bac53ec3bfeaf6546e7aaeaee516487b30534654'),'B1':('aligned-cursors','07061a601cdc20603c8c7a46f9ddbad7ac5f5939'),'B2':('aligned-batching','ca7cba171cd5198ed569d75012edc858041cbc44')};app=base/pins[a.tree][0]
ref='2df9897e1ce2ca90be29fa097d776c3766e217ad';protocol='e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4';comparator='09daef50fad5eef4dbda4e63a191749d0d167d3ae9d38bfd22df8721e2aac4a3';lock='54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8'
r={'startedAt':datetime.datetime.now(datetime.timezone.utc).isoformat(),'qualificationOnly':True,'performanceAcceptanceEligible':False,'fullFixture':a.full,'sourceRevision':pins[a.tree][1],'harnessRevision':ref,'steps':[]}
def now():return datetime.datetime.now(datetime.timezone.utc).isoformat()
def save():(out/'owner-receipt.json').write_text(json.dumps(r,indent=2)+'\n')
def cmd(name,args,cwd,env=None,timeout=None):
row={'name':name,'argv':list(map(str,args)),'cwd':str(cwd),'startedAt':now()};r['steps'].append(row);save()
with(out/(name+'.log')).open('xb')as log:result=subprocess.run(args,cwd=cwd,env=env,stdout=log,stderr=subprocess.STDOUT,timeout=timeout)
row.update({'exit':result.returncode,'finishedAt':now()});save();assert result.returncode==0,(name,result.returncode)
try:
assert(base/'aligned-4a83-builds.exit').read_text().strip()=='0'
qualified=json.loads((base/'aligned-4a83-builds-qualified.json').read_text())
assert {row['variant']:row['revision'] for row in qualified['results']}=={v:pin for v,(directory,pin) in pins.items()}
rows=[]
for q in pathlib.Path('/proc').iterdir():
if not q.name.isdigit():continue
try:
exe=os.readlink(q/'exe')
if pathlib.Path(exe).name in['node','postgres','paperclip-runner','rustc','cargo','chrome','chromium','chrome-headless-shell']:rows.append({'pid':int(q.name),'executable':exe})
except OSError:pass
assert not rows,rows
for directory,pin in pins.values():
root=base/directory;assert subprocess.check_output(['git','rev-parse','HEAD'],cwd=root,text=True).strip()==pin
assert not subprocess.check_output(['git','status','--short'],cwd=root,text=True).strip()
assert json.loads((root/'server/dist/build-info.json').read_text())['commit']==pin
assert h.exists(), 'Stage the exact independent driver before qualification'
assert not(h/'node_modules').is_symlink();assert subprocess.check_output(['git','rev-parse','HEAD'],cwd=h,text=True).strip()==ref
assert not subprocess.check_output(['git','status','--short'],cwd=h,text=True).strip()
files=['native-event-ingestion.journey.ts','playwright-native-ingestion.config.ts','fixtures/in-feed-codex.mjs','fixtures/native-ingestion-bin/codex','fixtures/composer-stop-codex-loader.mjs','fixtures/native-ingestion-bootstrap.mjs','fixtures/native-ingestion-remote-server.mjs','fixtures/native-ingestion-protocol-qualification.mjs','fixtures/native-ingestion-h5.mjs']
assert hashlib.sha256(b''.join((h/'tests/e2e'/f).read_bytes()for f in files)).hexdigest()==protocol
assert hashlib.sha256((h/'scripts/compare-native-ingestion-browser.ts').read_bytes()).hexdigest()==comparator
assert hashlib.sha256((h/'pnpm-lock.yaml').read_bytes()).hexdigest()==lock
assert hashlib.sha256((h/'tests/e2e/fixtures/native-ingestion-h5.md').read_bytes()).hexdigest()=='c647cfb68b360882a3bf42771b35e37c52e0aba7bca01511f4ebfb9482bf45bc'
r['resourceWatchdogMillis']=1800000;r['resourceWatchdogScope']='Resource-only including startup/test/teardown; original SLA and H5 observation unchanged';r['protocolSha256']=protocol;r['comparatorSha256']=comparator;r['driverLockSha256']=lock
with socket.socket()as s:s.bind(('127.0.0.1',0));port=s.getsockname()[1]
env=dict(os.environ)
for key in ['DATABASE_URL','PAPERCLIP_RUNNER_STATE_DIR','PAPERCLIP_INGESTION_SERVER_COMMAND','PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_DIR','PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_FS','PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_INSPECTOR','PAPERCLIP_INGESTION_BROWSER_CAPTURE','PAPERCLIP_INGESTION_SMOKE','NODE_OPTIONS','PAPERCLIP_INGESTION_OBSERVATION_MODE']:env.pop(key,None)
binary=str(app/'packages/paperclip-runner/runner/target/release/paperclip-runnerd')
overrides={'PAPERCLIP_INGESTION_OBSERVATION_MODE':'h5c-observe-original-sla-independent-agents-settlement','PAPERCLIP_INGESTION_TRANSPORT':'sandbox-remote-with-local-stop','PAPERCLIP_INGESTION_APP_ROOT':str(app),'PAPERCLIP_INGESTION_EVIDENCE_DIR':str(out),'PAPERCLIP_E2E_PORT':str(port),'PAPERCLIP_RUNNER_BINARY':binary,'PAPERCLIP_RUNNER_REMOTE_BINARY_PATH':binary,'PAPERCLIP_INGESTION_VARIANT':a.name+'-FUNCTIONAL-QUALIFICATION-ONLY','PAPERCLIP_INGESTION_DRIVER_DEPENDENCIES':'Independent H5c frozen offline ignore-scripts driver install; application owns independent normal locked install/build'}
if not a.full:overrides['PAPERCLIP_INGESTION_SMOKE']='1'
overrides['PATH']=env['PATH'];env.update(overrides);r['environmentOverrides']=overrides;save();subprocess.run(['sync'],check=True)
node=pathlib.Path('/opt/native-ingestion/node-v26.4.0-linux-x64/bin/node')
helper=base/'helpers/native-ingestion-run-h5d-browser.mjs'
assert hashlib.sha256(helper.read_bytes()).hexdigest()=='4afaa8ffcd764b498bada29c8916715b73d34462fd49152e44ab8437ffb330eb'
path_guard="const {assertPathNode}=await import("+json.dumps(str(helper))+");console.log(JSON.stringify(await assertPathNode(process.env,process.execPath,process.argv[1])));"
path_argv=[str(node),'--input-type=module','-e',path_guard,str(app)]
path_result=subprocess.run(path_argv,cwd=app,env=env,check=True,capture_output=True,text=True)
r['nodePathPreflight']={'executable':str(node),'argv':path_argv[1:],'cwd':str(app),'pathNode':json.loads(path_result.stdout)}
(out/'path-node-preflight.json').write_text(json.dumps(r['nodePathPreflight'],indent=2)+'\n');save()
cmd('playwright',['timeout','--signal=INT','--kill-after=40s','30m',str(node),'node_modules/@playwright/test/cli.js','test','--config','tests/e2e/playwright-native-ingestion.config.ts'],h,env,1845)
r['playwrightPassed']=True;r['finishedAt']=now();save()
except BaseException as error:r.update({'passed':False,'error':str(error),'traceback':traceback.format_exc(),'finishedAt':now()});save();(out/'owner.exit').write_text('1\n');raise
finally:
helper=base/'helpers/native-ingestion-run-h5d-browser.mjs'
guard="import net from 'node:net'; import {physicalQuietGuard} from '"+str(helper)+"'; const s=net.createServer(); await new Promise((ok,bad)=>{s.once('error',bad);s.listen(Number(process.argv[2]),'127.0.0.1',ok)}); await new Promise(ok=>s.close(ok)); await physicalQuietGuard({receiptFile:process.argv[1],phase:'post-arm',armStage:'H5d-QUALIFICATION-ONLY'});"
try:
result=subprocess.run(['node','--input-type=module','-e',guard,str(out/'physical-quiet-after.json'),str(r['environmentOverrides']['PAPERCLIP_E2E_PORT'])],check=True,capture_output=True,text=True)
(out/'port-after.json').write_text(json.dumps({'complete':True,'port':r['environmentOverrides']['PAPERCLIP_E2E_PORT'],'checkedAt':now(),'nodeBindAndListenPassed':True})+'\n')
if r.get('playwrightPassed'):
r['passed']=True;r['cleanupComplete']=True;save();(out/'owner.exit').write_text('0\n')
except BaseException as cleanup_error:
r['externalQuietFailure']=str(cleanup_error);r['passed']=False;r['cleanupComplete']=False;save();(out/'owner.exit').write_text('1\n');raise
scripts/native-ingestion/h5d/native-ingestion-run-h5d-browser.mjs
+705
View File
@@ -0,0 +1,705 @@
import assert from 'node:assert/strict';
import { spawn, execFileSync } from 'node:child_process';
import { createHash } from 'node:crypto';
import { readFile, writeFile, mkdir, open, access, readdir, realpath, stat, lstat } from 'node:fs/promises';
import { createReadStream } from 'node:fs';
import net from 'node:net';
import path from 'node:path';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { gunzipSync } from 'node:zlib';
// Prospective H5 application orchestrator. No launch without --execute and a
// byte-frozen root release. Original ce33 helper SHA is provenance only:
// 2244e238642caafd488a7c86fb463376c8459ee0f06f8054555407a174698fa1.
export const MODE = 'h5c-observe-original-sla-independent-agents-settlement';
export const PROTOCOL = 'v5c-independent-agents-complete-settlement-original-sla';
export const WORKLOAD_SHA256 = '3eea2f24c0c0c38134492a0a17e1b9a398335c085dd248f03b14c14bb5a19835';
const H5_REVISION = '2df9897e1ce2ca90be29fa097d776c3766e217ad';
const OLD_NATIVE_TARGET = '/home/ubuntu/native-ingestion/aligned-baseline/packages/paperclip-runner/runner/target';
const H5_PROTOCOL_SHA256 = 'e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4';
const H5_COMPARATOR_SHA256 = '09daef50fad5eef4dbda4e63a191749d0d167d3ae9d38bfd22df8721e2aac4a3';
const H5_INVENTORY_HELPER_SHA256 = '76b897dd6b7a90e784e228d78d79633a6d7011fe3ea01737f3bef9c6113298e9';
const H5_PROTOCOL_DOCUMENT_SHA256 = 'c647cfb68b360882a3bf42771b35e37c52e0aba7bca01511f4ebfb9482bf45bc';
const INVENTORY_ROOTS = ['server/dist', 'ui/dist', 'packages/db/src', 'packages/shared/src'];
export const RESOURCE_WATCHDOG_MILLIS = 30 * 60_000;
export const STAGES = Object.freeze(['browser-aa-b0', 'browser-aa-b1', 'browser-b0-b1', 'browser-b1-b2', 'browser-b0-b2']);
export const FINALIZE_STAGE = 'browser-finalize-h4e';
const protocolFiles = ['native-event-ingestion.journey.ts', 'playwright-native-ingestion.config.ts', 'fixtures/in-feed-codex.mjs',
'fixtures/native-ingestion-bin/codex', 'fixtures/composer-stop-codex-loader.mjs', 'fixtures/native-ingestion-bootstrap.mjs',
'fixtures/native-ingestion-remote-server.mjs', 'fixtures/native-ingestion-protocol-qualification.mjs', 'fixtures/native-ingestion-h5.mjs'];
const BASELINES = Object.freeze({ B0: 'bac53ec3bfeaf6546e7aaeaee516487b30534654', B1: '07061a601cdc20603c8c7a46f9ddbad7ac5f5939' });
const OLD_B2 = 'ca7cba171cd5198ed569d75012edc858041cbc44';
const OLD_RECEIPT = 'a1259ba760f3cbbe44817e333f9f620bcf3769fb0a6c16f9ce2f0f5b2ed60d94';
const OLD_SUMMARY = 'd3766fc4206d569d10fd02224433f1e9a2caa6992a7c62216e80af368aa4e44d';
const OLD_BINARY = 'abb65bdc073991e428bd37f9809603ee7564543cca0ed0305dc231137c93070f';
const OLD_RUST = 'bbe29bba574ce8d5cfddf2f7461ae0eb5983df07';
const OLD_INVENTORIES = Object.freeze({
B0: { manifestFileSha256: 'a1c797a75dd3174f414fc5529acf56783e1a3dfe7b22f1e460a87b60f6bff01d', aggregateSha256: 'a50720be2ce53f9222187d63609c5936f94b6a00c60df2ca112261abd1bf921e' },
B1: { manifestFileSha256: 'df48e5c511a2f31e31180e05f18e2322ca4f384ea06f6f96a57de24d39849ef8', aggregateSha256: 'bca5b6fad62e70631c135a9ff377efeb587aba17ef0842c8fd82804ed99a81ab' },
});
const LOCK_SHA256 = '54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8';
const GATES_SHA256 = 'ded8bcbb8f05732348f6520b9c5e41b7c35bc098997eb263a9dcb174eede75fc';
const selfFile = fileURLToPath(import.meta.url);
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
const getJson = async file => JSON.parse(await readFile(file, 'utf8'));
const saveJson = async (file, value, fresh = false) => writeFile(file, JSON.stringify(value, null, 2) + '\n', fresh ? { flag: 'wx' } : undefined);
const digest = (value, name) => assert.ok(typeof value === 'string' && /^[a-f0-9]{64}$/.test(value), `Missing exact ${name} SHA-256`);
const revision = (value, name) => assert.ok(typeof value === 'string' && /^[a-f0-9]{40}$/.test(value), `Missing immutable full ${name} revision`);
const absolute = (value, name) => assert.ok(typeof value === 'string' && path.isAbsolute(value), `Missing absolute ${name} path`);
const option = name => process.argv.find(value => value.startsWith(`--${name}=`))?.slice(name.length + 3);
export function stagePlan(stage) {
assert.ok(STAGES.includes(stage) || stage === FINALIZE_STAGE, 'Select one registered H5 stage');
if (stage === FINALIZE_STAGE) return { stage, requiredVariants: ['B0', 'B1', 'B2'], priorStages: [...STAGES], fullArms: 0, finalizationOnly: true };
const [baseline, candidate] = stage === STAGES[0] ? ['B0', 'B0'] : stage === STAGES[1] ? ['B1', 'B1']
: stage === STAGES[2] ? ['B0', 'B1'] : stage === STAGES[3] ? ['B1', 'B2'] : ['B0', 'B2'];
const aa = baseline === candidate;
return { stage, baseline, candidate, aa, expectation: stage === STAGES[2] ? 'nonregression' : 'improvement',
candidatePolicy: aa ? null : stage === STAGES[2] ? 'cursor-nonregression' : 'final-b2',
requiredVariants: [...new Set([baseline, candidate])], priorStages: STAGES.slice(0, STAGES.indexOf(stage)),
pairs: 5, fullArms: 10, noiseStage: aa ? null : `browser-aa-${baseline.toLowerCase()}` };
}
async function hashedJson(file, expected, name, compressed = false) {
absolute(file, name); digest(expected, name);
const bytes = await readFile(file); assert.equal(hash(bytes), expected, `${name} bytes changed`);
return { bytes, value: JSON.parse((compressed ? gunzipSync(bytes, { maxOutputLength: 128 * 1024 * 1024 }) : bytes).toString('utf8')) };
}
export async function validateVariantBuild(variant, descriptor) {
assert.ok(descriptor && typeof descriptor === 'object', `Required ${variant} qualified build is missing`);
absolute(descriptor.appRoot, `${variant} appRoot`); revision(descriptor.revision, variant);
digest(descriptor.binarySha256, `${variant} binary`); revision(descriptor.rustSourceTree, `${variant} Rust source tree`);
const receipt = (await hashedJson(descriptor.qualifiedReceiptFile, descriptor.qualifiedReceiptSha256, `${variant} qualified receipt`)).value;
const summary = (await hashedJson(descriptor.inventorySummaryFile, descriptor.inventorySummarySha256, `${variant} inventory summary`)).value;
const manifest = (await hashedJson(descriptor.manifestFile, descriptor.manifestFileSha256, `${variant} inventory manifest`, true)).value;
digest(descriptor.aggregateSha256, `${variant} aggregate inventory`);
const rows = receipt.results?.filter(row => row.variant === variant);
assert.equal(rows?.length, 1, `${variant} qualified receipt must select this variant exactly once`);
assert.equal(rows[0].revision, descriptor.revision, `${variant} qualified receipt revision mismatch`);
assert.equal(rows[0].compiledBuildRevision, descriptor.revision, `${variant} qualified compiled revision mismatch`);
assert.equal(rows[0].sourceClean, true, `${variant} qualified source was not clean`);
const native = receipt.nativeStageHashes?.filter(row => row.variant === variant);
assert.equal(native?.length, 1, `${variant} qualified native stage is missing`);
assert.equal(native[0].binarySha256, descriptor.binarySha256, `${variant} native stage binary mismatch`);
assert.equal(receipt.binarySha256, descriptor.binarySha256, `${variant} receipt binary mismatch`);
assert.equal(receipt.rustSourceTree, descriptor.rustSourceTree, `${variant} Rust source identity mismatch`);
assert.equal(receipt.rustInputObjects?.runner, descriptor.rustSourceTree, `${variant} qualified Rust input objects are missing`);
absolute(native[0].targetRealPath, `${variant} qualified native target`);
for (const [file, object] of Object.entries(receipt.rustInputObjects)) {
assert.ok(file && !path.posix.isAbsolute(file) && path.posix.normalize(file) === file && !file.includes('\\') && !file.split('/').includes('..'), 'Qualified Rust input path is invalid');
revision(object, `${variant} Rust input ${file}`);
}
assert.equal(receipt.artifactInventorySummary, descriptor.qualifiedInventorySummaryReference ?? descriptor.inventorySummaryFile, `${variant} receipt points to another inventory summary`);
assert.equal(manifest.variant, variant); assert.equal(manifest.revision, descriptor.revision);
assert.equal(manifest.compiledBuildRevision, descriptor.revision); assert.equal(manifest.aggregateSha256, descriptor.aggregateSha256);
assert.deepEqual(manifest.roots, INVENTORY_ROOTS);
assert.ok(Array.isArray(manifest.files) && manifest.files.length === manifest.fileCount && manifest.fileCount > 0);
const row = summary.inventories?.[variant]; assert.ok(row, `${variant} inventory summary entry missing`);
assert.equal(row.manifest, path.basename(descriptor.manifestFile)); assert.equal(row.manifestSha256, descriptor.manifestFileSha256);
assert.equal(row.aggregateSha256, descriptor.aggregateSha256); assert.equal(row.fileCount, manifest.fileCount);
if (Object.hasOwn(BASELINES, variant)) {
assert.equal(descriptor.revision, BASELINES[variant], 'Original baseline reference must remain unchanged');
assert.equal(descriptor.qualifiedReceiptSha256, OLD_RECEIPT, 'B0/B1 must use the original, unmodified aligned4a83 qualified receipt');
assert.equal(descriptor.inventorySummarySha256, OLD_SUMMARY); assert.equal(descriptor.binarySha256, OLD_BINARY);
assert.equal(descriptor.rustSourceTree, OLD_RUST);
for (const [key, value] of Object.entries(OLD_INVENTORIES[variant])) assert.equal(descriptor[key], value, `Original ${variant} inventory changed`);
} else {
assert.equal(variant, 'B2');
assert.ok(![...Object.values(BASELINES), OLD_B2].includes(descriptor.revision), 'Final B2 needs its new actual immutable product revision');
assert.notEqual(descriptor.qualifiedReceiptSha256, OLD_RECEIPT, 'Final B2 needs its own new qualified receipt');
assert.notEqual(descriptor.binarySha256, OLD_BINARY, 'Final Rust B2 needs its newly qualified binary');
assert.notEqual(descriptor.rustSourceTree, OLD_RUST, 'Final Rust B2 needs its newly qualified Rust inputs');
assert.notEqual(native[0].targetRealPath, OLD_NATIVE_TARGET, 'Final Rust B2 must not reuse the original B0/B1 target');
}
return { receipt, summary, manifest };
}
// Match the existing inventory generator and H5 helper exactly: Unicode code
// point order, Python ensure_ascii JSON with sorted bytes/path/sha256 keys.
function pythonOrder(left, right) {
const a = [...left], b = [...right];
for (let index = 0; index < Math.min(a.length, b.length); index++) {
const difference = a[index].codePointAt(0) - b[index].codePointAt(0);
if (difference) return difference;
}
return a.length - b.length;
}
const pythonRow = row => JSON.stringify({ bytes: row.bytes, path: row.path, sha256: row.sha256 })
.replace(/[\u007f-\uffff]/g, character => `\\u${character.charCodeAt(0).toString(16).padStart(4, '0')}`);
function inventoryPath(value) {
assert.ok(typeof value === 'string' && value.length && !value.includes('\\') && !value.includes('\0')
&& !path.posix.isAbsolute(value) && path.posix.normalize(value) === value && !value.split('/').includes('..')
&& INVENTORY_ROOTS.some(root => value.startsWith(`${root}/`)), 'Inventory filename escapes registered roots');
return value;
}
export async function validateActualInventory(build, manifest) {
const startedAtMs = Date.now(), started = performance.now(), canonicalRoot = await realpath(build.appRoot);
assert.deepEqual(manifest.roots, INVENTORY_ROOTS);
const expected = new Map(), canonicalFiles = new Set();
for (const row of manifest.files) {
const name = inventoryPath(row.path);
assert.equal(expected.has(name), false, 'Duplicate inventory filename'); expected.set(name, row);
digest(row.sha256, 'inventory file'); assert.ok(Number.isSafeInteger(row.bytes) && row.bytes >= 0);
}
assert.equal(expected.size, manifest.fileCount, 'Inventory file count is incomplete');
const actualNames = [], canonicalDirectories = new Set();
const walk = async relative => {
const directory = path.join(build.appRoot, relative), canonical = await realpath(directory);
assert.ok(canonical.startsWith(`${canonicalRoot}${path.sep}`), 'Inventory directory resolves outside APP_ROOT');
assert.equal(canonicalDirectories.has(canonical), false, 'Inventory directory aliases another registered directory'); canonicalDirectories.add(canonical);
for (const entry of await readdir(directory, { withFileTypes: true })) {
const child = `${relative}/${entry.name}`;
if (entry.isDirectory()) await walk(child);
else {
const file = path.join(build.appRoot, child), metadata = await lstat(file);
if (metadata.isSymbolicLink()) assert.equal((await stat(file)).isDirectory(), false, 'Inventory directory symlink is not a frozen file');
assert.ok(metadata.isFile() || metadata.isSymbolicLink(), 'Inventory includes a nonregular entry');
inventoryPath(child); actualNames.push(child);
}
}
};
for (const root of INVENTORY_ROOTS) await walk(root);
actualNames.sort(pythonOrder);
assert.deepEqual(actualNames, [...expected.keys()].sort(pythonOrder), 'Actual APP_ROOT file set differs from frozen inventory');
const actualRows = new Array(actualNames.length); let next = 0;
await Promise.all(Array.from({ length: Math.min(8, actualNames.length) }, async () => {
for (;;) {
const index = next++; if (index >= actualNames.length) return;
const name = actualNames[index], file = path.join(build.appRoot, name), canonical = await realpath(file);
assert.ok(canonical.startsWith(`${canonicalRoot}${path.sep}`), 'Inventory file resolves outside APP_ROOT');
assert.equal(canonicalFiles.has(canonical), false, 'Inventory filenames alias the same canonical file'); canonicalFiles.add(canonical);
const before = await stat(file); assert.ok(before.isFile(), 'Inventory evidence must be a regular file');
const content = createHash('sha256'); for await (const bytes of createReadStream(file)) content.update(bytes);
const after = await stat(file);
assert.equal(after.size, before.size, 'Artifact changed during inventory hashing'); assert.equal(after.mtimeMs, before.mtimeMs, 'Artifact changed during inventory hashing');
const row = { bytes: after.size, path: name, sha256: content.digest('hex') }, frozen = expected.get(name);
assert.equal(row.bytes, frozen.bytes, `Actual artifact size changed: ${name}`); assert.equal(row.sha256, frozen.sha256, `Actual artifact bytes changed: ${name}`);
actualRows[index] = row;
}
}));
const aggregateSha256 = hash(Buffer.from(`${actualRows.map(pythonRow).join('\n')}\n`, 'utf8'));
assert.equal(aggregateSha256, manifest.aggregateSha256, 'Actual APP_ROOT aggregate differs from frozen inventory');
assert.equal(aggregateSha256, build.aggregateSha256, 'Actual APP_ROOT aggregate differs from qualified build');
return { appRoot: build.appRoot, canonicalAppRoot: canonicalRoot, fileCount: actualRows.length, aggregateSha256,
startedAtMs, completedAtMs: Date.now(), elapsedMs: performance.now() - started,
scope: 'Live registered file set, canonical containment/uniqueness, size and SHA checked before this arm; not an assertion of future immutability' };
}
function harnessFields(release) {
return Object.fromEntries(['harnessRevision', 'protocolSha256', 'comparatorSha256', 'inventoryHelperSha256', 'protocolDocumentSha256', 'lockSha256', 'gatesSha256', 'workloadSha256', 'observationMode', 'protocolVersion'].map(key => [key, release[key]]));
}
export async function loadRelease(releaseFile, stage, outputRoot) {
absolute(releaseFile, 'root release'); absolute(outputRoot, 'output root');
const releaseBytes = await readFile(releaseFile), release = JSON.parse(releaseBytes);
const plan = stagePlan(stage);
assert.equal(release.authorized, true, 'Root release is not authorized'); assert.equal(release.stage, stage);
assert.equal(release.outputRoot, outputRoot, 'Root release freezes a different exact output path');
assert.equal(release.orchestratorSha256, hash(await readFile(selfFile)), 'Orchestrator changed after release');
absolute(release.harnessRoot, 'H5 harness'); revision(release.harnessRevision, 'H5 harness');
assert.equal(release.harnessRevision, H5_REVISION, 'Use the actual independently frozen H5 full revision');
assert.equal(release.observationMode, MODE); assert.equal(release.protocolVersion, PROTOCOL);
assert.equal(release.workloadSha256, WORKLOAD_SHA256);
assert.equal(release.resourceWatchdogMillis, RESOURCE_WATCHDOG_MILLIS, 'Resource watchdog registration changed; this is not an SLA');
assert.equal(release.lockSha256, LOCK_SHA256); assert.equal(release.gatesSha256, GATES_SHA256);
for (const [key, expected] of [['protocolSha256', H5_PROTOCOL_SHA256], ['comparatorSha256', H5_COMPARATOR_SHA256], ['inventoryHelperSha256', H5_INVENTORY_HELPER_SHA256], ['protocolDocumentSha256', H5_PROTOCOL_DOCUMENT_SHA256]])
assert.equal(release[key], expected, `Reviewed H5 source pin changed: ${key}`);
for (const variant of plan.requiredVariants) await validateVariantBuild(variant, release.perVariantBuilds?.[variant]);
const context = { releaseFile, releaseBytesSha256: hash(releaseBytes), release, plan, out: outputRoot, commands: [], commandIndex: 0 };
await checkPriorStages(context);
if (plan.noiseStage) await checkNoise(context);
if (plan.finalizationOnly) await checkFinalizationInputs(context);
return context;
}
async function checkReleaseBytes(context) {
assert.equal(hash(await readFile(context.releaseFile)), context.releaseBytesSha256, 'Exact root release bytes changed');
assert.equal(hash(await readFile(selfFile)), context.release.orchestratorSha256, 'Orchestrator changed after release');
if (context.rootReleaseCopy) assert.equal(hash(await readFile(context.rootReleaseCopy)), context.releaseBytesSha256, 'Preserved root release bytes changed');
}
function checkComparisonIdentity(report, context) {
assert.equal(report.complete, true); assert.equal(report.observationMode, MODE); assert.equal(report.protocolVersion, PROTOCOL);
assert.equal(report.workloadSha256, WORKLOAD_SHA256); assert.equal(report.measurementSha256, context.release.protocolSha256);
assert.equal(report.comparatorSha256, context.release.comparatorSha256); assert.equal(report.gatesSha256, GATES_SHA256);
assert.equal(report.pairCount, 5);
}
export function assertAaNoiseSuitability(noise) {
assert.equal(noise.aaSuitability?.passed, true, 'Unsuitable or missing browser A/A cannot freeze noise');
assert.deepEqual(noise.aaSuitability.rule, { pairCount: 5, pairedFraction: 0.20, medianDriftFraction: 0.10,
contrastRepresentationFloorMs: 2, representationLimitedMedianMs: 20 }, 'Browser A/A suitability rule changed');
const keys = [];
for (const name of ['paced', 'burst16', 'burst128', 'burst512', 'existing-history', 'operator-stop']) {
keys.push(...['firstVisibleAfterEmissionMs', 'emissionToVisible.medianMs', 'drainThroughTerminalMs'].map(metric => `${name}:${metric}`));
if (name === 'paced') keys.push(`${name}:interChunkGaps.maxMs`);
if (!['burst16', 'operator-stop'].includes(name)) keys.push(`${name}:emissionToVisible.p95Ms`, `${name}:intendedToVisible.p95Ms`);
if (name === 'operator-stop') keys.push(`${name}:stopThroughTerminalMs`, `${name}:stopThroughAcknowledgementMs`);
else keys.push(`${name}:h5.independentDrain.elapsedFromEpochMs`);
}
const rows = noise.measurements;
assert.ok(Array.isArray(rows), 'Browser A/A measurements are missing');
// The frozen comparator adds both Stop p95 metrics only for a >=100-marker
// observed Stop prefix. Its owning raw validation remains authoritative.
if (rows.some(row => row.name === 'operator-stop' && ['emissionToVisible.p95Ms', 'intendedToVisible.p95Ms'].includes(row.metric)))
keys.push('operator-stop:emissionToVisible.p95Ms', 'operator-stop:intendedToVisible.p95Ms');
assert.deepEqual(rows.map(row => `${row.name}:${row.metric}`).sort(), keys.sort(), 'Browser A/A metric set changed or verified completion metric is missing');
assert.equal(noise.aaSuitability.metricCount, keys.length, 'Browser A/A metric count changed');
for (const row of rows) {
assert.equal(row.aaSuitability?.passed, true, 'Every browser A/A measurement must pass suitability');
assert.equal(row.aaSuitability.pairCount, 5); assert.equal(row.aaSuitability.sampleCount, 10);
assert.equal(row.samples?.length, 5, 'Every browser A/A metric needs five pairs');
for (const key of ['aaNoiseEnvelopeRelative', 'aaNoiseEnvelopeAbsoluteMs'])
assert.ok(Number.isFinite(row[key]) && row[key] >= 0, 'Browser A/A actual noise must be finite and nonnegative');
assert.equal(row.aaNoiseEnvelopeRelative, row.aaSuitability.observedNoiseRelative, 'Browser A/A relative noise changed');
assert.equal(row.aaNoiseEnvelopeAbsoluteMs, row.aaSuitability.observedNoiseAbsoluteMs, 'Browser A/A absolute noise changed');
}
}
async function checkPriorStages(context) {
for (const stage of context.plan.priorStages) {
const pin = context.release.priorStageReports?.[stage]; assert.ok(pin, `Stage order requires frozen ${stage} first`);
assert.equal(pin.file, path.join(context.out, `${stage}.json`), 'Prior stage report path changed');
const report = (await hashedJson(pin.file, pin.sha256, stage)).value;
checkComparisonIdentity(report, context);
const priorPlan = stagePlan(stage);
assert.equal(report.phase, priorPlan.aa ? 'aa' : 'compare');
if (priorPlan.aa) assertAaNoiseSuitability(report);
assert.equal(report.baselineRevision, BASELINES[priorPlan.baseline]);
const candidateRevision = BASELINES[priorPlan.candidate] ?? context.release.perVariantBuilds.B2?.revision;
assert.equal(report.candidateRevision, candidateRevision, 'Prior stage used a different measured candidate revision');
for (const [arm, variant] of [['baseline', priorPlan.baseline], ['candidate', priorPlan.candidate]]) {
const expected = context.release.perVariantBuilds?.[variant] ?? (BASELINES[variant] ? { ...OLD_INVENTORIES[variant], qualifiedReceiptSha256: OLD_RECEIPT, inventorySummarySha256: OLD_SUMMARY } : undefined);
assert.ok(expected, 'Prior candidate qualified build is unavailable');
for (const key of ['manifestFileSha256', 'aggregateSha256', 'qualifiedReceiptSha256', 'inventorySummarySha256'])
assert.equal(report.artifactInventories?.[arm]?.[key], expected[key], 'Prior stage changed a measured qualified inventory');
}
if (!priorPlan.aa) assert.equal(report.performancePassed, true, `Prior ${stage} performance failed`);
}
}
export async function checkNoise(context) {
const { release, plan, out } = context;
const expected = path.join(out, `${plan.noiseStage}.json`);
assert.equal(release.noiseReportFile, expected, 'Use the same exact frozen A/A noise path');
const noise = (await hashedJson(expected, release.noiseReportSha256, 'Reviewed A/A noise report')).value;
checkComparisonIdentity(noise, context); assert.equal(noise.phase, 'aa');
assert.equal(noise.baselineRevision, release.perVariantBuilds[plan.baseline].revision);
assert.equal(noise.candidateRevision, noise.baselineRevision);
assert.equal(noise.h5Assessments?.length, 5, 'Reviewed A/A must contain all five complete assessments');
assertAaNoiseSuitability(noise);
const build = release.perVariantBuilds[plan.baseline];
for (const arm of ['baseline', 'candidate']) for (const key of ['manifestFileSha256', 'aggregateSha256', 'qualifiedReceiptSha256', 'inventorySummarySha256'])
assert.equal(noise.artifactInventories?.[arm]?.[key], build[key], 'Frozen A/A baseline inventory differs from required build');
return noise;
}
const QUIET_SETTLE_MILLIS = 30_000, QUIET_POLL_MILLIS = 1_000, QUIET_PS_TIMEOUT_MILLIS = 2_000;
const QUIET_SCHEMA = 'paperclip.native-ingestion.physical-quiet.v1';
const QUIET_SCOPE = 'Dedicated Linux process-name guard for specified Node/PostgreSQL/Runner/browser/Rust-build names only; not a claim that every possible host activity is absent. Qualified H5 Stop proof separately covers provider/sleep control groups.';
const QUIET_ZOMBIE_POLICY = 'Strict: matched zombie/dead rows remain blockers until absent; no automatic killing or silent zombie exclusion.';
function processFamily(comm) {
if (/^(node|node-MainThread)$/.test(comm)) return 'node';
if (/^(postgres|postmaster|pg_ctl|initdb)$/.test(comm)) return 'postgresql';
if (/^(paperclip-runne|paperclip-runner.*|Runner)$/.test(comm)) return 'runner';
if (/^(chrome|headless|chromium)/i.test(comm)) return 'browser';
if (/^(rustc|cargo)$/.test(comm)) return 'rust-build';
return null;
}
export function classifyLinuxProcesses(text, orchestratorPid = process.pid) {
assert.ok(typeof text === 'string' && text.trim(), 'Linux ps returned no process rows');
assert.ok(Number.isSafeInteger(orchestratorPid) && orchestratorPid > 0, 'Invalid orchestrator PID');
const seen = new Set(), matched = []; let rowCount = 0;
for (const line of text.trim().split(/\r?\n/)) {
const fields = line.match(/^\s*(\d+)\s+(\d+)\s+(.+?)\s+([DIRSTtWXZPK])\s*$/);
assert.ok(fields, 'Malformed Linux ps PID/PPID/comm/state row');
const row = { pid: Number(fields[1]), ppid: Number(fields[2]), comm: fields[3].trim(), state: fields[4] };
assert.ok(Number.isSafeInteger(row.pid) && row.pid > 0 && Number.isSafeInteger(row.ppid) && row.ppid >= 0 && row.comm.length > 0 && row.comm.length <= 256 && !row.comm.includes('\0'), 'Malformed Linux ps process fields');
assert.equal(seen.has(row.pid), false, 'Duplicate Linux ps PID'); seen.add(row.pid); rowCount++;
const family = processFamily(row.comm);
if (family) matched.push({ ...row, family, allowed: row.pid === orchestratorPid && !['Z', 'X'].includes(row.state) });
}
assert.ok(seen.has(orchestratorPid), 'Linux ps omitted the current orchestrator PID');
return { rowCount, matched, blockers: matched.filter(row => !row.allowed) };
}
function linuxProcessRows(timeoutMillis) {
return execFileSync('/bin/ps', ['-eo', 'pid=,ppid=,comm=,state='], {
encoding: 'utf8', timeout: timeoutMillis, maxBuffer: 2 * 1024 * 1024, env: { LC_ALL: 'C' },
});
}
export async function physicalQuietGuard({ receiptFile, phase, armStage, primaryFailureAlreadyPresent = false,
platform = process.platform, orchestratorPid = process.pid, readPs = linuxProcessRows,
now = () => performance.now(), pause = milliseconds => new Promise(resolve => setTimeout(resolve, milliseconds)) }) {
absolute(receiptFile, 'physical quiet receipt');
const started = now(), receipt = { schema: QUIET_SCHEMA, scope: QUIET_SCOPE, phase, armStage, platform, orchestratorPid,
fields: ['pid', 'ppid', 'comm', 'state'], zombiePolicy: QUIET_ZOMBIE_POLICY,
resourceSettleLimitMillis: QUIET_SETTLE_MILLIS, pollIntervalMillis: QUIET_POLL_MILLIS, psTimeoutMillis: QUIET_PS_TIMEOUT_MILLIS,
primaryFailureAlreadyPresent, startedAtMs: Date.now(), passed: false, observations: [] };
let failure;
try {
assert.equal(platform, 'linux', 'Accepted H5 physical quiet guard requires Linux');
assert.ok(['pre-arm', 'post-arm'].includes(phase), 'Unknown physical quiet phase');
for (;;) {
const remaining = QUIET_SETTLE_MILLIS - (now() - started);
assert.ok(remaining > 0, 'Physical quiet settle deadline expired; stop for cleanup');
const observed = classifyLinuxProcesses(readPs(Math.max(1, Math.min(QUIET_PS_TIMEOUT_MILLIS, Math.floor(remaining)))), orchestratorPid);
receipt.observations.push({ atMs: Date.now(), elapsedMillis: now() - started, ...observed });
assert.ok(now() - started < QUIET_SETTLE_MILLIS, 'Physical quiet settle deadline expired; stop for cleanup');
if (observed.blockers.length === 0) { receipt.passed = true; break; }
const wait = Math.min(QUIET_POLL_MILLIS, QUIET_SETTLE_MILLIS - (now() - started));
assert.ok(wait > 0, 'Physical quiet settle deadline expired; stop for cleanup');
await pause(wait);
}
} catch (error) {
failure = error;
receipt.failure = { name: error instanceof Error ? error.name : 'UnknownError', message: error instanceof Error ? error.message : String(error) };
}
receipt.completedAtMs = Date.now(); receipt.elapsedMillis = now() - started;
const bytes = Buffer.from(JSON.stringify(receipt, null, 2) + '\n');
await writeFile(receiptFile, bytes, { flag: 'wx' });
const result = { file: receiptFile, bytes: bytes.length, sha256: hash(bytes), passed: receipt.passed };
if (failure) {
const error = new Error(`Physical quiet guard failed (${phase}); receipt=${receiptFile}; sha256=${result.sha256}`, { cause: failure });
Object.assign(error, { receiptFile, receiptSha256: result.sha256 }); throw error;
}
return result;
}
export async function assertNoPreferredUiShadow(appRoot, statPath = lstat) {
const preferredIndex = path.join(appRoot, 'server/ui-dist/index.html'), preferredDirectory = path.dirname(preferredIndex);
try { await statPath(preferredIndex); assert.fail('Preferred server/ui-dist/index.html exists (including a symlink); ui/dist inventory cannot attest the selected UI'); }
catch (error) { if (error.code !== 'ENOENT') throw error; }
try { assert.equal((await statPath(preferredDirectory)).isSymbolicLink(), false, 'Preferred server/ui-dist directory symlink prevents the absence guard'); }
catch (error) { if (error.code !== 'ENOENT') throw error; }
return { checkedAtMs: Date.now(), preferredIndex, preferredIndexAbsent: true, preferredDirectorySymlink: false,
selectedUiRoot: path.join(appRoot, 'ui/dist'),
assumption: 'Preferred server/ui-dist index absent at preflight; inventoried ui/dist is the compiled fallback. No routing/build changes or future immutability claim.' };
}
async function checkFrozen(context, selectedVariant) {
await checkReleaseBytes(context);
for (const key of ['DATABASE_URL', 'PAPERCLIP_RUNNER_STATE_DIR', 'PAPERCLIP_INGESTION_SERVER_COMMAND', 'PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_DIR',
'PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_FS', 'PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_INSPECTOR', 'PAPERCLIP_INGESTION_BROWSER_CAPTURE', 'NODE_OPTIONS'])
assert.ok(!process.env[key], `Acceptance rejects ambient ${key}`);
const { release, plan } = context, actualInventories = {};
for (const variant of plan.requiredVariants) {
const build = release.perVariantBuilds[variant], qualified = await validateVariantBuild(variant, build);
assert.equal(execFileSync('git', ['rev-parse', 'HEAD'], { cwd: build.appRoot, encoding: 'utf8' }).trim(), build.revision, `${variant} SUT revision changed`);
assert.equal(execFileSync('git', ['status', '--short'], { cwd: build.appRoot, encoding: 'utf8' }).trim(), '', `${variant} SUT is dirty`);
assert.equal((await getJson(path.join(build.appRoot, 'server/dist/build-info.json'))).commit, build.revision);
const binaryFile = path.join(build.appRoot, 'packages/paperclip-runner/runner/target/release/paperclip-runnerd');
const native = qualified.receipt.nativeStageHashes.find(row => row.variant === variant);
const actualTarget = await realpath(path.join(build.appRoot, 'packages/paperclip-runner/runner/target'));
assert.equal(actualTarget, native.targetRealPath, `${variant} actual native target differs from qualified receipt`);
assert.equal(await realpath(binaryFile), path.join(actualTarget, 'release/paperclip-runnerd'), `${variant} native binary resolves outside its exact qualified target`);
assert.equal(hash(await readFile(binaryFile)), build.binarySha256, `${variant} actual native binary changed`);
for (const [file, object] of Object.entries(qualified.receipt.rustInputObjects))
assert.equal(execFileSync('git', ['rev-parse', `${build.revision}:packages/paperclip-runner/${file}`], { cwd: build.appRoot, encoding: 'utf8' }).trim(), object, `${variant} qualified Rust input changed: ${file}`);
assert.equal(execFileSync('git', ['rev-parse', `${build.revision}:packages/paperclip-runner/runner`], { cwd: build.appRoot, encoding: 'utf8' }).trim(), build.rustSourceTree);
if (!selectedVariant || selectedVariant === variant) actualInventories[variant] = {
...await validateActualInventory(build, qualified.manifest), uiServing: await assertNoPreferredUiShadow(build.appRoot),
};
}
assert.equal(execFileSync('git', ['rev-parse', 'HEAD'], { cwd: release.harnessRoot, encoding: 'utf8' }).trim(), release.harnessRevision);
assert.equal(execFileSync('git', ['status', '--short'], { cwd: release.harnessRoot, encoding: 'utf8' }).trim(), '');
assert.equal(hash(Buffer.concat(await Promise.all(protocolFiles.map(file => readFile(path.join(release.harnessRoot, 'tests/e2e', file)))))), release.protocolSha256);
for (const [file, expected] of [['scripts/compare-native-ingestion-browser.ts', release.comparatorSha256], ['scripts/native-ingestion-h5-inventory.mjs', release.inventoryHelperSha256], ['tests/e2e/fixtures/native-ingestion-h5.md', release.protocolDocumentSha256], ['scripts/native-ingestion-performance-gates.mjs', release.gatesSha256], ['pnpm-lock.yaml', release.lockSha256]])
assert.equal(hash(await readFile(path.join(release.harnessRoot, file))), expected, `Frozen H5 support changed: ${file}`);
await checkPriorStages(context);
if (plan.noiseStage) await checkNoise(context);
return { checkedAtMs: Date.now(), releaseFileSha256: context.releaseBytesSha256, actualInventories };
}
async function execute(context, step) {
const index = ++context.commandIndex, startedAt = new Date().toISOString(), log = path.join(context.out, `${step.stage}.log`);
const command = { index, stage: step.stage, cwd: step.cwd, executable: process.execPath, args: step.args, environmentOverrides: step.env ?? {}, startedAt, log,
resourceWatchdogMillis: RESOURCE_WATCHDOG_MILLIS, resourceWatchdogScope: 'Resource-only cap including startup/test/teardown; original SLA and H5 observation limits remain driver-owned' };
context.commands.push(command);
const save = () => saveJson(path.join(context.out, `${context.plan.stage}-commands.json`), context.commands);
await save(); console.log(`START ${step.stage} ${startedAt}; log=${log}`);
const stream = (await open(log, 'wx')).createWriteStream();
const child = spawn(process.execPath, step.args, { cwd: step.cwd, env: step.launchEnvironment ?? { ...process.env, ...step.env }, stdio: ['ignore', 'pipe', 'pipe'] });
command.childPid = child.pid; await save();
const interrupt = () => child.kill('SIGINT'), terminate = () => child.kill('SIGTERM');
process.once('SIGINT', interrupt); process.once('SIGTERM', terminate);
let forcedStop;
const watchdog = setTimeout(() => {
command.watchdogExpired = true; child.kill('SIGTERM');
forcedStop = setTimeout(() => { command.forcedDriverKill = true; child.kill('SIGKILL'); }, 30_000);
}, RESOURCE_WATCHDOG_MILLIS);
child.stdout.pipe(stream, { end: false }); child.stderr.pipe(stream, { end: false });
let exit;
try { exit = await new Promise((resolve, reject) => { child.once('error', reject); child.once('close', (code, signal) => resolve({ code, signal })); }); }
finally { clearTimeout(watchdog); clearTimeout(forcedStop); process.off('SIGINT', interrupt); process.off('SIGTERM', terminate); await new Promise(resolve => stream.end(resolve)); }
Object.assign(command, { finishedAt: new Date().toISOString(), ...exit }); await save();
console.log(`END ${step.stage} code=${exit.code} signal=${exit.signal ?? 'none'}`);
assert.equal(command.watchdogExpired, undefined, 'Resource watchdog expired; stop for physical cleanup, never accept or advance');
assert.equal(exit.code, 0, `Stage ${step.stage} failed; retain raw files and perform physical cleanup before continuing`);
}
async function freePort() {
const server = net.createServer();
await new Promise((resolve, reject) => { server.once('error', reject); server.listen(0, '127.0.0.1', resolve); });
const port = server.address().port; await new Promise(resolve => server.close(resolve)); return port;
}
async function assertPortReleased(port) {
const server = net.createServer();
await new Promise((resolve, reject) => { server.once('error', reject); server.listen(port, '127.0.0.1', resolve); });
await new Promise(resolve => server.close(resolve));
}
async function extractEvidence(directory) {
const playwrightFile = path.join(directory, 'playwright.json'), report = await getJson(playwrightFile), attachments = [];
const visit = node => { if (Array.isArray(node)) for (const child of node) visit(child); else if (node && typeof node === 'object') {
if (Array.isArray(node.attachments)) attachments.push(...node.attachments.filter(value => value.name === 'native-ingestion-evidence'));
for (const [key, child] of Object.entries(node)) if (key !== 'attachments') visit(child);
} };
visit(report); assert.equal(attachments.length, 1, 'Exactly one native-ingestion evidence attachment is required');
const attachment = attachments[0];
const evidence = attachment.body ? JSON.parse(Buffer.from(attachment.body, 'base64').toString()) : await getJson(path.resolve(directory, attachment.path));
const file = path.join(directory, 'native-ingestion-evidence.json'); await saveJson(file, evidence, true);
return { file, evidence, playwrightFile };
}
async function armSidecar(directory, variant, build) {
const destination = path.join(directory, 'inventory'); await mkdir(destination);
const copies = {};
for (const [key, sha] of [['manifestFile', 'manifestFileSha256'], ['inventorySummaryFile', 'inventorySummarySha256'], ['qualifiedReceiptFile', 'qualifiedReceiptSha256']]) {
const bytes = await readFile(build[key]); assert.equal(hash(bytes), build[sha], 'Per-arm provenance changed while copying');
const file = path.join(destination, path.basename(build[key])); await writeFile(file, bytes, { flag: 'wx' }); copies[key] = path.relative(directory, file);
}
const sidecar = { schema: 'paperclip.native-ingestion.arm-inventory.v1', variant, sourceRevision: build.revision, compiledBuildRevision: build.revision,
playwrightReportSha256: hash(await readFile(path.join(directory, 'playwright.json'))), ...copies,
manifestFileSha256: build.manifestFileSha256, aggregateSha256: build.aggregateSha256,
qualifiedReceiptSha256: build.qualifiedReceiptSha256, inventorySummarySha256: build.inventorySummarySha256 };
const file = path.join(directory, 'arm-inventory.json'); await saveJson(file, sidecar, true); return file;
}
async function armArtifactFiles(result) {
const sidecar = await getJson(result.inventoryFile), base = path.dirname(result.inventoryFile);
return [result.file, result.playwrightFile, result.inventoryFile,
...['manifestFile', 'inventorySummaryFile', 'qualifiedReceiptFile'].map(key => path.resolve(base, sidecar[key])),
path.join(base, 'pre-arm-inventory-check.json'), path.join(base, 'physical-http-port-check.json'),
path.join(base, 'physical-quiet-pre.json'), path.join(base, 'physical-quiet-post.json')];
}
export async function sealArmArtifacts(out, result) {
const canonicalRoot = await realpath(out), files = await armArtifactFiles(result), seen = new Set(), seals = [];
for (const file of files) {
absolute(file, 'raw arm artifact'); const canonical = await realpath(file);
assert.ok(canonical.startsWith(`${canonicalRoot}${path.sep}`), 'Raw arm artifact resolves outside owned output');
assert.equal(seen.has(canonical), false, 'Duplicate raw arm artifact'); seen.add(canonical);
const bytes = await readFile(file); seals.push({ file, bytes: bytes.length, sha256: hash(bytes) });
}
return seals;
}
export async function verifyRawArtifactSeals(out, manifest) {
const canonicalRoot = await realpath(out);
for (const entry of manifest) for (const arm of ['baseline', 'candidate']) {
const seals = entry[`${arm}Artifacts`]; assert.ok(Array.isArray(seals) && seals.length === 10, 'Raw arm byte seals are incomplete');
const seen = new Set();
for (const seal of seals) {
absolute(seal.file, 'sealed arm artifact'); digest(seal.sha256, 'sealed arm artifact');
assert.equal(seen.has(seal.file), false, 'Duplicate sealed arm artifact'); seen.add(seal.file);
assert.ok((await realpath(seal.file)).startsWith(`${canonicalRoot}${path.sep}`), 'Sealed arm artifact resolves outside output');
const bytes = await readFile(seal.file);
assert.equal(bytes.length, seal.bytes, 'Raw arm artifact size changed after timing');
assert.equal(hash(bytes), seal.sha256, 'Raw arm artifact bytes changed after timing');
}
const expected = await armArtifactFiles({ file: entry[`${arm}File`], playwrightFile: entry[`${arm}PlaywrightFile`], inventoryFile: entry[`${arm}InventoryFile`] });
assert.deepEqual([...seen].sort(), expected.sort(), 'Raw seals do not cover this exact evidence/PW/sidecar/inventory set');
for (const [filename, phase] of [['physical-quiet-pre.json', 'pre-arm'], ['physical-quiet-post.json', 'post-arm']]) {
const receipt = await getJson(path.join(path.dirname(entry[`${arm}InventoryFile`]), filename));
assert.equal(receipt.schema, QUIET_SCHEMA); assert.equal(receipt.passed, true); assert.equal(receipt.platform, 'linux');
assert.equal(receipt.phase, phase); assert.equal(receipt.zombiePolicy, QUIET_ZOMBIE_POLICY);
}
}
}
function physicalGuardReferences(manifest) {
return manifest.flatMap((entry, pair) => ['baseline', 'candidate'].map(arm => ({ pair: pair + 1, arm,
receipts: entry[`${arm}Artifacts`].filter(seal => /^physical-quiet-(pre|post)\.json$/.test(path.basename(seal.file))) })));
}
export async function assertPathNode(environment, approvedExecutable, workingDirectory,
{ run = execFileSync, resolve = realpath, currentExecutable = process.execPath } = {}) {
absolute(workingDirectory, 'PATH Node working directory'); absolute(approvedExecutable, 'Approved Node executable');
assert.ok(typeof environment.PATH === 'string' && environment.PATH.length > 0, 'Actual launch PATH is missing');
const actual = JSON.parse(run('node', ['-p', 'JSON.stringify({executable:process.execPath,version:process.version,cwd:process.cwd()})'],
{ env: environment, cwd: workingDirectory, encoding: 'utf8', timeout: 10000 }));
assert.equal(actual.version, 'v26.4.0', 'PATH node version differs from approved VM runtime');
absolute(actual.executable, 'PATH-selected Node executable'); absolute(actual.cwd, 'PATH-selected Node cwd');
const canonicalExecutable = await resolve(actual.executable), approvedCanonicalExecutable = await resolve(approvedExecutable);
assert.equal(canonicalExecutable, approvedCanonicalExecutable, 'PATH node executable differs from approved runtime');
assert.equal(approvedCanonicalExecutable, await resolve(currentExecutable), 'Approved Node differs from orchestrator runtime');
const canonicalCwd = await resolve(actual.cwd), approvedCanonicalCwd = await resolve(workingDirectory);
assert.equal(canonicalCwd, approvedCanonicalCwd, 'PATH node did not run in the actual appRoot cwd');
return { ...actual, canonicalExecutable, approvedCanonicalExecutable, canonicalCwd, approvedCanonicalCwd,
PATH: environment.PATH, requestedCwd: workingDirectory };
}
async function browserRun(context, armStage, variant) {
const preflight = await checkFrozen(context, variant); // ALL receipts; selected live artifact inventory before EVERY arm.
const { release } = context, build = release.perVariantBuilds[variant], directory = path.join(context.out, armStage);
await mkdir(directory, { recursive: false });
await physicalQuietGuard({ receiptFile: path.join(directory, 'physical-quiet-pre.json'), phase: 'pre-arm', armStage });
const port = await freePort();
const environmentOverrides = { PAPERCLIP_INGESTION_OBSERVATION_MODE: MODE, PAPERCLIP_RUNNER_BINARY: path.join(build.appRoot, 'packages/paperclip-runner/runner/target/release/paperclip-runnerd'),
PAPERCLIP_RUNNER_REMOTE_BINARY_PATH: path.join(build.appRoot, 'packages/paperclip-runner/runner/target/release/paperclip-runnerd'),
PAPERCLIP_INGESTION_TRANSPORT: 'sandbox-remote-with-local-stop', PAPERCLIP_E2E_PORT: String(port), PAPERCLIP_INGESTION_SMOKE: '0',
PAPERCLIP_INGESTION_VARIANT: `${armStage}-${variant}`, PAPERCLIP_INGESTION_APP_ROOT: build.appRoot, PAPERCLIP_INGESTION_EVIDENCE_DIR: directory,
PAPERCLIP_INGESTION_DRIVER_DEPENDENCIES: 'Independent frozen H5 driver install; each measured variant owns its separately qualified application/native build', PATH: process.env.PATH };
const launchEnvironment = { ...process.env, ...environmentOverrides };
const pathNode = await assertPathNode(launchEnvironment, process.execPath, build.appRoot);
await saveJson(path.join(directory, 'pre-arm-inventory-check.json'), { ...preflight, pathNode }, true);
let primaryError, inventoryFile, extracted;
try {
await execute(context, { stage: armStage, cwd: release.harnessRoot,
args: ['node_modules/@playwright/test/cli.js', 'test', '--config', 'tests/e2e/playwright-native-ingestion.config.ts'],
env: environmentOverrides, launchEnvironment });
} catch (error) { primaryError = error; }
try {
inventoryFile = await armSidecar(directory, variant, build); extracted = await extractEvidence(directory);
if (!primaryError) {
const { evidence } = extracted;
assert.equal(evidence.complete, true); assert.equal(evidence.observationComplete, true); assert.equal(evidence.observationMode, MODE);
assert.equal(evidence.workloadSha256, WORKLOAD_SHA256); assert.equal(evidence.protocolVersion, PROTOCOL);
assert.equal(evidence.transport, 'sandbox-remote-with-local-stop'); assert.equal(evidence.cleanup?.complete, true); assert.deepEqual(evidence.cleanup.errors, []);
assert.equal(evidence.measurementSha256, release.protocolSha256); assert.equal(evidence.harnessProvenance?.sourceRevision, release.harnessRevision); assert.equal(evidence.harnessProvenance?.sourceStatus, '');
assert.equal(evidence.sourceRevision, build.revision); assert.equal(evidence.compiledBuildRevision, build.revision);
assert.equal(evidence.serverArtifact, 'compiled'); assert.equal(evidence.qualificationOnly, false); assert.equal(evidence.diagnosticOnly, false); assert.equal(evidence.performanceAcceptanceEligible, true);
assert.equal(evidence.browserRecording?.trace, 'off'); assert.equal(evidence.browserRecording?.video, 'off'); assert.equal(evidence.browserRecording?.screencast, false);
const loaded = evidence.cases.find(item => item.name === 'burst512'); assert.equal(loaded.runs, 100); assert.equal(loaded.participants, 100); assert.equal(loaded.durable.length, 100);
const { validateArmInventory } = await import(pathToFileURL(path.join(release.harnessRoot, 'scripts/native-ingestion-h5-inventory.mjs')).href);
await validateArmInventory(inventoryFile, { evidence, owningPlaywright: { reportSha256: hash(await readFile(extracted.playwrightFile)) } });
}
} catch (error) { primaryError ??= error; }
try { await assertPortReleased(port); await saveJson(path.join(directory, 'physical-http-port-check.json'), { released: true, port, scope: 'Owned HTTP listener only; PG/Runner/provider physical cleanup is separately required before the next release' }, true); }
catch (error) { primaryError ??= error; }
try { await physicalQuietGuard({ receiptFile: path.join(directory, 'physical-quiet-post.json'), phase: 'post-arm', armStage,
primaryFailureAlreadyPresent: Boolean(primaryError) }); }
catch (error) {
if (primaryError) console.error(`Secondary physical quiet failure; primary fixture failure is retained: ${error.message}`);
primaryError ??= error;
} // Failed guard receipt and SHA remain independent evidence; the fixture failure stays primary.
if (primaryError) throw primaryError;
const result = { file: extracted.file, playwrightFile: extracted.playwrightFile, inventoryFile };
return { ...result, artifacts: await sealArmArtifacts(context.out, result) };
}
function comparatorArgs(context, plan, pairsFile, output, originalProof) {
const args = ['scripts/compare-native-ingestion-browser.ts', `--observation-mode=${MODE}`, `--phase=${plan.aa ? 'aa' : 'compare'}`,
`--expect=${plan.expectation}`, `--pairs=${pairsFile}`, `--output=${output}`];
if (!plan.aa) args.push(`--candidate-policy=${plan.candidatePolicy}`, `--noise-report=${path.join(context.out, `${plan.noiseStage}.json`)}`,
...(originalProof ? [`--original-h4e-proof=${originalProof.proofFile}`, `--original-h4e-inventory=${originalProof.inventoryFile}`, '--require-acceptance'] : ['--require-performance']));
return args;
}
async function browserSeries(context) {
const { plan, release } = context, manifest = [], file = path.join(context.out, `${plan.stage}-pairs.json`), comparisonFile = path.join(context.out, `${plan.stage}.json`);
await access(comparisonFile).then(() => assert.fail('Prior comparison output already exists'), error => { if (error.code !== 'ENOENT') throw error; });
await saveJson(file, manifest, true);
for (let pair = 0; pair < 5; pair++) {
const entry = {}, arms = pair % 2 ? [['candidate', plan.candidate], ['baseline', plan.baseline]] : [['baseline', plan.baseline], ['candidate', plan.candidate]];
for (const [arm, variant] of arms) {
const result = await browserRun(context, `${plan.stage}-pair${pair + 1}-${arm}`, variant);
entry[`${arm}File`] = result.file; entry[`${arm}PlaywrightFile`] = result.playwrightFile; entry[`${arm}InventoryFile`] = result.inventoryFile; entry[`${arm}Artifacts`] = result.artifacts;
}
manifest.push(entry); await saveJson(file, manifest);
}
await checkFrozen(context); // Same exact release/noise path and bytes pre-comparator.
await verifyRawArtifactSeals(context.out, manifest);
const manifestSha256 = hash(await readFile(file));
await execute(context, { stage: `${plan.stage}-comparison`, cwd: release.harnessRoot, args: comparatorArgs(context, plan, file, comparisonFile) });
assert.equal(hash(await readFile(file)), manifestSha256, 'Raw paired manifest changed during comparison');
await verifyRawArtifactSeals(context.out, manifest);
const result = await getJson(comparisonFile); checkComparisonIdentity(result, context);
if (!plan.aa) assert.equal(result.performancePassed, true);
await saveJson(path.join(context.out, `${plan.stage}.completion.json`), { stage: plan.stage, complete: true, fullArms: 10, completedAtMs: Date.now(),
releaseFile: context.rootReleaseCopy, originalReleaseFile: context.releaseFile, releaseFileSha256: context.releaseBytesSha256, harnessPins: harnessFields(release),
perVariantBuilds: Object.fromEntries(plan.requiredVariants.map(variant => [variant, release.perVariantBuilds[variant]])),
manifestFile: file, manifestSha256, comparisonFile, comparisonSha256: hash(await readFile(comparisonFile)),
physicalGuards: physicalGuardReferences(manifest),
...(plan.noiseStage ? { noiseReportFile: release.noiseReportFile, noiseReportSha256: release.noiseReportSha256 } : {}),
performancePassed: result.performancePassed ?? null, acceptancePassed: result.acceptancePassed ?? null,
finalOriginalH4eConfirmationPending: plan.candidatePolicy === 'final-b2' }, true);
}
async function checkFinalizationInputs(context) {
const proof = context.release.originalH4eProof;
assert.ok(proof, 'One later unchanged H4e proof is required for both final comparisons');
for (const [key, sha] of [['proofFile', 'proofSha256'], ['inventoryFile', 'inventorySha256']]) await hashedJson(proof[key], proof[sha], `Original H4e ${key}`);
const proofReport = await getJson(proof.proofFile), proofAttempts = [];
const visit = value => {
if (Array.isArray(value)) for (const child of value) visit(child);
else if (value && typeof value === 'object') {
if (Array.isArray(value.results) && Object.hasOwn(value, 'expectedStatus')) proofAttempts.push(...value.results);
for (const [key, child] of Object.entries(value)) if (key !== 'results') visit(child);
}
};
visit(proofReport); assert.equal(proofAttempts.length, 1, 'One later unchanged original H4e attempt must confirm both stages');
const proofStartedAtMs = Date.parse(proofAttempts[0].startTime);
assert.ok(Number.isFinite(proofStartedAtMs), 'Original H4e proof has no actual attempt start time');
for (const stage of ['browser-b1-b2', 'browser-b0-b2']) {
const pin = context.release.completedSeries?.[stage]; assert.ok(pin, `Missing frozen completed ${stage}`);
assert.equal(pin.file, path.join(context.out, `${stage}.completion.json`));
const completion = (await hashedJson(pin.file, pin.sha256, `${stage} completion`)).value;
assert.equal(completion.stage, stage); assert.equal(completion.complete, true); assert.equal(completion.fullArms, 10); assert.equal(completion.performancePassed, true);
assert.ok(Number.isFinite(completion.completedAtMs) && completion.completedAtMs < proofStartedAtMs, 'Original H4e proof must run after BOTH timing stages complete');
assert.deepEqual(completion.harnessPins, harnessFields(context.release));
const plan = stagePlan(stage);
for (const variant of plan.requiredVariants) assert.deepEqual(completion.perVariantBuilds[variant], context.release.perVariantBuilds[variant], 'Finalization changed a measured qualified build');
assert.equal(completion.manifestFile, path.join(context.out, `${stage}-pairs.json`));
assert.equal(completion.comparisonFile, path.join(context.out, `${stage}.json`));
assert.equal(hash(await readFile(completion.manifestFile)), completion.manifestSha256, 'Raw timing manifest changed before finalization');
const manifest = await getJson(completion.manifestFile); assert.equal(manifest.length, 5);
await verifyRawArtifactSeals(context.out, manifest);
assert.deepEqual(completion.physicalGuards, physicalGuardReferences(manifest), 'Completion physical-guard receipts differ from sealed arm bytes');
assert.equal(hash(await readFile(completion.comparisonFile)), completion.comparisonSha256, 'Preliminary performance assessment changed');
assert.equal(completion.noiseReportFile, path.join(context.out, `${plan.noiseStage}.json`), 'Finalization rebound the frozen noise path');
assert.equal(hash(await readFile(completion.noiseReportFile)), completion.noiseReportSha256, 'Frozen noise bytes changed before finalization');
const originalRelease = await hashedJson(completion.releaseFile, completion.releaseFileSha256, `${stage} original release`);
assert.equal(originalRelease.value.noiseReportFile, completion.noiseReportFile); assert.equal(originalRelease.value.noiseReportSha256, completion.noiseReportSha256);
}
}
async function finalizeSeries(context) {
await checkFrozen(context); await checkFinalizationInputs(context);
for (const stage of ['browser-b1-b2', 'browser-b0-b2']) {
await checkFrozen(context); await checkFinalizationInputs(context);
const pin = context.release.completedSeries[stage], completion = (await hashedJson(pin.file, pin.sha256, `${stage} completion`)).value;
const output = path.join(context.out, `${stage}.final.json`);
await access(output).then(() => assert.fail('Final assessment output already exists'), error => { if (error.code !== 'ENOENT') throw error; });
await execute(context, { stage: `${stage}-original-h4e-finalization`, cwd: context.release.harnessRoot,
args: comparatorArgs(context, stagePlan(stage), completion.manifestFile, output, context.release.originalH4eProof) });
await checkFinalizationInputs(context);
assert.equal((await getJson(output)).acceptancePassed, true);
}
}
async function main() {
const stage = option('stage');
if (!process.argv.includes('--execute') && !process.argv.includes('--validate-release')) {
console.log(JSON.stringify({ draft: true, observationMode: MODE, protocolVersion: PROTOCOL, workloadSha256: WORKLOAD_SHA256,
stages: STAGES.map(stagePlan), fullArms: 50, finalization: stagePlan(FINALIZE_STAGE),
baselines: BASELINES, originalReceiptSha256: OLD_RECEIPT, resourceWatchdogMillis: RESOURCE_WATCHDOG_MILLIS,
frozenH5: { revision: H5_REVISION, protocolSha256: H5_PROTOCOL_SHA256, comparatorSha256: H5_COMPARATOR_SHA256, inventoryHelperSha256: H5_INVENTORY_HELPER_SHA256, protocolDocumentSha256: H5_PROTOCOL_DOCUMENT_SHA256 },
pendingPins: 'Future B2 has no descriptor until its actual new source and separately qualified native/application build are available. No H5 VM execution is released by this file.',
note: 'Plan only; no SUT/build/VM/network or arm starts. Root and independent peer review plus qualification release required.' }, null, 2)); return;
}
const out = option('output-root') ?? '/home/ubuntu/native-ingestion/evidence/h5/browser';
const context = await loadRelease(option('release'), stage, out);
if (process.argv.includes('--validate-release') && !process.argv.includes('--execute')) {
console.log(JSON.stringify({ valid: true, stage, requiredVariants: context.plan.requiredVariants,
validationScope: 'Static immutable release/receipt/manifest/noise/stage-order bindings only; no live SUT/driver/build attestation and no arm execution', releaseFileSha256: context.releaseBytesSha256 })); return;
}
await checkFrozen(context); await mkdir(out, { recursive: true });
// Keep the exact release bytes as a distinct artifact, so later stages never
// require an owner to retain an overwriteable release-input filename.
context.rootReleaseCopy = path.join(out, `${stage}-root-release.json`);
const releaseBytes = await readFile(context.releaseFile);
assert.equal(hash(releaseBytes), context.releaseBytesSha256, 'Root release changed before preservation');
await writeFile(context.rootReleaseCopy, releaseBytes, { flag: 'wx' });
await saveJson(path.join(out, `${stage}-release-record.json`), { ...context.release, releaseFile: context.releaseFile,
releaseFileSha256: context.releaseBytesSha256, recordedAt: new Date().toISOString() }, true);
if (context.plan.finalizationOnly) await finalizeSeries(context); else await browserSeries(context);
}
if (process.argv[1] && path.resolve(process.argv[1]) === selfFile) await main();
scripts/native-ingestion/h5d/native-ingestion-validate-h5d-evidence.mjs
+77
View File
@@ -0,0 +1,77 @@
import assert from 'node:assert/strict';
import {readFile,writeFile} from 'node:fs/promises';
import {createHash} from 'node:crypto';
import {execFileSync} from 'node:child_process';
import path from 'node:path';
import {pathToFileURL} from 'node:url';
const directory=process.argv[2],harness=process.argv[3]??'/home/ubuntu/native-ingestion/harness-h5d';
assert.ok(path.isAbsolute(directory)&&path.isAbsolute(harness));
const hash=bytes=>createHash('sha256').update(bytes).digest('hex');
const helper=path.join(harness,'tests/e2e/fixtures/native-ingestion-h5.mjs');
assert.equal(hash(await readFile(helper)),'7d0629849fe776faa51a2ca1d987429be1a529df38efa8d92466bff6a8dabff2');
assert.equal(execFileSync('git',['rev-parse','HEAD'],{cwd:harness,encoding:'utf8'}).trim(),'2df9897e1ce2ca90be29fa097d776c3766e217ad');
const {assertParticipantSetup,assertParticipantTopology,assertSettlementObservation,assertRecordedFailures,assessH5Arm,H5_REGISTRATION,H5_WORKLOADS,H5_LIMITS}=await import(pathToFileURL(helper).href);
const evidenceBytes=await readFile(path.join(directory,'native-ingestion-evidence.json')),evidence=JSON.parse(evidenceBytes);
const pwBytes=await readFile(path.join(directory,'playwright.json')),pw=JSON.parse(pwBytes),tests=[];
assert.deepEqual(pw.errors??[],[]);
function visit(value){if(Array.isArray(value))for(const child of value)visit(child);else if(value&&typeof value==='object'){
if(Array.isArray(value.results)&&Object.hasOwn(value,'expectedStatus'))tests.push(value);
for(const [key,child]of Object.entries(value))if(key!=='results')visit(child);
}}
visit(pw);assert.equal(tests.length,1);const test=tests[0];assert.equal(test.expectedStatus,'passed');assert.equal(test.results.length,1);
const result=test.results[0];assert.equal(result.status,'passed');assert.equal(result.retry,0);assert.deepEqual(result.errors,[]);assert.ok(!result.error);
const attachments=result.attachments.filter(value=>value.name==='native-ingestion-evidence');assert.equal(attachments.length,1);
const attached=attachments[0].body?Buffer.from(attachments[0].body,'base64'):await readFile(path.resolve(directory,attachments[0].path));
assert.equal(hash(attached),hash(evidenceBytes),'Owning Playwright attempt must attach these exact evidence bytes');
const owner={reportFile:path.join(directory,'playwright.json'),reportSha256:hash(pwBytes),status:result.status,retry:result.retry,errors:result.errors,startedAt:result.startTime,durationMs:result.duration,timeoutMs:test.timeout};
assert.equal(evidence.complete,true);assert.equal(evidence.observationComplete,true);assert.equal(evidence.protocolVersion,'v5c-independent-agents-complete-settlement-original-sla');
assert.equal(evidence.measurementSha256,'e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4');
assert.equal(evidence.harnessProvenance.sourceRevision,'2df9897e1ce2ca90be29fa097d776c3766e217ad');assert.equal(evidence.harnessProvenance.sourceStatus,'');
assert.equal(evidence.diagnosticOnly,false);assert.equal(evidence.browserRecording.trace,'off');assert.equal(evidence.browserRecording.video,'off');assert.equal(evidence.browserRecording.screencast,false);
const outer=JSON.parse(await readFile(path.join(directory,'owner-receipt.json'),'utf8'));
assert.equal(typeof outer.fullFixture,'boolean');assert.equal(typeof evidence.qualificationOnly,'boolean');
assert.equal(outer.fullFixture,!evidence.qualificationOnly,'Owning command FULL/SMOKE mode must match its evidence');
assert.equal(outer.sourceRevision,evidence.sourceRevision);assert.equal(outer.sourceRevision,evidence.compiledBuildRevision);assert.equal(evidence.serverArtifact,'compiled');
assert.equal(outer.harnessRevision,evidence.harnessProvenance.sourceRevision);
const pathPreflight=JSON.parse(await readFile(path.join(directory,'path-node-preflight.json'),'utf8'));
assert.deepEqual(pathPreflight,outer.nodePathPreflight);
assert.equal(pathPreflight.executable,'/opt/native-ingestion/node-v26.4.0-linux-x64/bin/node');
assert.equal(pathPreflight.cwd,outer.environmentOverrides.PAPERCLIP_INGESTION_APP_ROOT);
assert.equal(pathPreflight.pathNode.version,'v26.4.0');
for(const key of ['executable','cwd','canonicalCwd','approvedCanonicalCwd','canonicalExecutable','approvedCanonicalExecutable'])
assert.ok(typeof pathPreflight.pathNode[key]==='string'&&path.isAbsolute(pathPreflight.pathNode[key]),'Missing absolute PATH Node proof field');
assert.ok(typeof pathPreflight.pathNode.PATH==='string'&&pathPreflight.pathNode.PATH.length>0,'Missing recorded launch PATH');
assert.equal(pathPreflight.pathNode.requestedCwd,pathPreflight.cwd);
assert.equal(pathPreflight.pathNode.canonicalCwd,pathPreflight.pathNode.approvedCanonicalCwd);
assert.equal(pathPreflight.pathNode.canonicalExecutable,pathPreflight.pathNode.approvedCanonicalExecutable);
assert.equal(pathPreflight.pathNode.PATH,outer.environmentOverrides.PATH);
assert.equal(owner.timeoutMs,H5_LIMITS.fullInvocationMs);
const expectedCases=outer.fullFixture?H5_WORKLOADS:[
{name:'qualification',runs:3,events:16,intervalMs:25,prelude:0,stop:false},
{name:'operator-stop',runs:1,events:512,intervalMs:100,prelude:0,stop:true},
];
const matrix=item=>Object.fromEntries(['name','runs','events','intervalMs','prelude','stop'].map(key=>[key,item[key]]));
assert.deepEqual(evidence.cases.map(matrix),expectedCases.map(matrix),'Qualification requires the complete exact case matrix');
assertParticipantSetup(evidence);const cases=[];
for(const item of evidence.cases){
assertParticipantTopology(evidence,item);
const observed=assertSettlementObservation(item.h5.terminalObservation,item);
const recorded=assertRecordedFailures(item.recordedFailures,item,evidence.companyId);
cases.push({name:item.name,runs:item.runs,observedSettlement:observed,recordedFailures:recorded,
observedTransitions:item.h5.terminalObservation.transitions,recordedFailureProof:item.recordedFailures,
recordedRetryRate:recorded.uniqueEverFailedRuns/item.runs});
}
let assessment;
if(!evidence.qualificationOnly){assessment=assessH5Arm(evidence,owner);assert.ok(assessment.misses.every(value=>H5_REGISTRATION.knownBaselineMisses.includes(value)));}
assert.equal((await readFile(path.join(directory,'owner.exit'),'utf8')).trim(),'0');
assert.equal(outer.passed,true);assert.equal(outer.cleanupComplete,true);
const quiet=JSON.parse(await readFile(path.join(directory,'physical-quiet-after.json'),'utf8'));assert.equal(quiet.passed,true);assert.equal(quiet.platform,'linux');
const port=JSON.parse(await readFile(path.join(directory,'port-after.json'),'utf8'));assert.equal(port.nodeBindAndListenPassed,true);
const startup=await readFile(path.join(directory,'playwright.log'),'utf8');assert.match(startup,/Heartbeat\s+enabled\s*\(/,'Actual global scheduler must be enabled');
const summary={complete:true,qualificationOnly:true,performanceAcceptanceEligible:false,
scope:'Read-only post-run qualification validation with exact frozen H5c helpers; a full observation may retain original-SLA misses and is not an accepted A/A sample',
evidenceSha256:hash(evidenceBytes),owningPlaywright:owner,helperSha256:hash(await readFile(helper)),
actualGlobalSchedulerConfiguredEnabled:true,assessment,cases};
await writeFile(path.join(directory,'independent-settlement-validation.json'),JSON.stringify(summary,null,2)+'\n',{flag:'wx'});
console.log(JSON.stringify({complete:true,qualificationOnly:true,performanceAcceptanceEligible:false,assessment,
cases:cases.map(value=>({name:value.name,runs:value.runs,recordedRetryRate:value.recordedRetryRate,recordedErrorClasses:value.recordedFailures.recordedErrorClasses}))}));
scripts/native-ingestion/h5d/sources.json
+165
View File
@@ -0,0 +1,165 @@
{
"schema": "paperclip.native-ingestion.h5d-helper-sources.v1",
"scope": "Exact frozen sources and reviewed helper snapshots; packaging makes no performance claim.",
"repository": "https://github.com/paperclipai/paperclip.git",
"deliveryLockSha256AtPackaging": "7540075e96f9a93924a1e54775c17346c5760dd5d02902d83fb5228f918996ba",
"deliveryCheckoutWasExecutedDriver": false,
"mixedDeliveryTreeCannotReplaceEitherFrozenDriver": true,
"recordedVmRuntime": {
"node": "v26.4.0",
"pnpm": "9.15.4",
"otherRuntimeOutputs": "Exact absolute executable, argument and output pins from sealed runtime receipts; never guessed."
},
"rawB2Role": "ca7cba171cd5198ed569d75012edc858041cbc44 is unoptimized B2 provenance, not a final candidate.",
"frozenDriver": {
"commit": "2df9897e1ce2ca90be29fa097d776c3766e217ad",
"tree": "d0604f6fb899715f7ed623494fab24d64445e12b",
"baseRevision": "e431ec63d41fe90c6fd68bd2fc2bb969feeca917",
"annotatedTag": "native-ingestion-h5d",
"tagObject": "47ef6d1a5d81c31d4602aad50666f67c58f4cdb4",
"measurementSha256": "e1d8930c47b4d3a287aa0f3baf09fbd3aa04ac30b9313b707616dea7a3fec5d4",
"driverLockSha256": "54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8",
"comparatorSha256": "09daef50fad5eef4dbda4e63a191749d0d167d3ae9d38bfd22df8721e2aac4a3",
"preservedWorkloadSha256": "3eea2f24c0c0c38134492a0a17e1b9a398335c085dd248f03b14c14bb5a19835"
},
"sourceFreezeReceiptSha256": "d1b7a8d5a18bada9bf12aefb2ee98dbf3e5daef8e8a9adadab65c991004d091f",
"sourceOverlay": [
{
"file": "scripts/compare-native-ingestion-browser.ts",
"sha256": "09daef50fad5eef4dbda4e63a191749d0d167d3ae9d38bfd22df8721e2aac4a3",
"byteIdenticalToFrozenDriver": true
},
{
"file": "scripts/native-ingestion-performance-gates.test.mjs",
"sha256": "0724e6ed30ad0ad79851dbdd475e1f9e94c2028572b7b70b0c91af620227532e",
"byteIdenticalToFrozenDriver": true
},
{
"file": "scripts/native-ingestion-h5.test.mjs",
"sha256": "47fe63eff16017db43f146e3e88991801ee2f97efc5e757754b2076e843a4fd5",
"byteIdenticalToFrozenDriver": true
},
{
"file": "tests/e2e/fixtures/native-ingestion-h5.md",
"sha256": "c647cfb68b360882a3bf42771b35e37c52e0aba7bca01511f4ebfb9482bf45bc",
"byteIdenticalToFrozenDriver": true
},
{
"file": "tests/e2e/fixtures/native-ingestion-h5.mjs",
"sha256": "7d0629849fe776faa51a2ca1d987429be1a529df38efa8d92466bff6a8dabff2",
"byteIdenticalToFrozenDriver": true
},
{
"file": "tests/e2e/native-event-ingestion.journey.ts",
"sha256": "de0e8613e85aa9dc76cbd87b0f8418f25807266c60fa79fdf87ceec5b819711a",
"byteIdenticalToFrozenDriver": true
}
],
"helpers": [
{
"file": "native-ingestion-run-h5d-browser.mjs",
"originalTaskFile": "/private/tmp/native-ingestion-run-h5d-browser.mjs",
"bytes": 58316,
"sha256": "4afaa8ffcd764b498bada29c8916715b73d34462fd49152e44ab8437ffb330eb",
"packagingChanges": []
},
{
"file": "native-ingestion-linux-h5d-qualification.py",
"originalTaskFile": "/private/tmp/native-ingestion-linux-h5d-qualification.py",
"bytes": 7779,
"sha256": "9df879140eda2953e1794bf9d68e1e09457fd33d8ca397731b8bba255c8f8c6d",
"packagingChanges": []
},
{
"file": "native-ingestion-h5d-linux-readiness.mjs",
"originalTaskFile": "/private/tmp/native-ingestion-h5d-linux-readiness.mjs",
"bytes": 3278,
"sha256": "9a5b4ae1a543d937f4811e41fc814b8528df2adb8602f931dc7b4166f0c8e9c7",
"packagingChanges": []
},
{
"file": "native-ingestion-h5d-full-linux-readiness.mjs",
"originalTaskFile": "/private/tmp/native-ingestion-h5d-full-linux-readiness.mjs",
"bytes": 3286,
"sha256": "7ffae7a9aa278837be3e40e2e4ca2d77c3c76cfd881b492b76dd861fc80e7040",
"packagingChanges": []
},
{
"file": "native-ingestion-validate-h5d-evidence.mjs",
"originalTaskFile": "/private/tmp/native-ingestion-validate-h5d-evidence.mjs",
"bytes": 7590,
"sha256": "a733ea5af01fa25c2fba1af83110bbaec1a2de5a003b4c4e137b8654dec93c11",
"packagingChanges": []
},
{
"file": "native-ingestion-extract-h5d-qualification-foreground.py",
"originalTaskFile": "/private/tmp/native-ingestion-extract-h5d-qualification-foreground.py",
"bytes": 6339,
"sha256": "b1e74ffd6696d28ce0b5a060f98310c4f1963e40d860486cc1962898e8cfdcc1",
"packagingChanges": []
}
],
"sourceQualifiedAppRevisions": {
"B0": "bac53ec3bfeaf6546e7aaeaee516487b30534654",
"B1": "07061a601cdc20603c8c7a46f9ddbad7ac5f5939",
"B2": "ca7cba171cd5198ed569d75012edc858041cbc44"
},
"qualifiedBuildReceiptSha256": "a1259ba760f3cbbe44817e333f9f620bcf3769fb0a6c16f9ce2f0f5b2ed60d94",
"postreaderRepair": {
"reviewedAtUtc": "2026-11-04T09:17:11.678514+00:00",
"oldReaderSha256": "968fbbfce1942e96495f299b69c77ac904db72facf112664f54be6fc9b4e0cc1",
"newReaderSha256": "b1e74ffd6696d28ce0b5a060f98310c4f1963e40d860486cc1962898e8cfdcc1",
"diffSha256": "60314f2893acd3b34bda9f3864f323a225fd6cdcda4c389ab73f3cd1e4aef628",
"rawReportSha256": "d12ad9128d6a30bc335a69761d6004b2b3618fc275a07a84ef5434cfdb00b1e1",
"owningPlaywrightPassed": true,
"oldInvalidArrayPositionAssumptionRed": true,
"actualPreludeRunIndex": 9,
"checks": [
"actual immutable foreground run correlation",
"run-array permutation stays valid",
"reject wrong-prelude-member",
"reject wrong-provider-clock",
"reject wrong-foreground-token",
"reject wrong-authority-issue",
"reject duplicate-foreground-proof",
"reject missing-prelude-event",
"reject nonunique-prelude",
"reject prelude-not-before-epoch",
"reject short-retained-history"
],
"correctedExtractorExit": 0,
"rawOwnerSourceAndMeasurementUnchanged": true,
"originalSealUntouched": true,
"scope": "Offline postreader defect repair only; qualification remains ineligible for noise/performance. No owning failed trial regraded; no runtime/test rerun."
},
"qualificationAsOfUtc": "2026-11-04T09:30:00Z",
"qualificationFacts": {
"small": {
"passed": true,
"deltas": 48,
"results": 3,
"stopDeltas": 9,
"stopResults": 0,
"sealedFiles": 40
},
"full": {
"ownerPassed": true,
"ownerDurationMs": 853339,
"original900BudgetPassed": true,
"originalSlaMisses": [
"burst512:streaming120"
],
"loadedDeltasNearOriginalDeadline": 41,
"foregroundExpected": 512,
"loadedDurableDeltas": 51200,
"loadedResults": 100,
"recoveredProviderTransportFailures": 43,
"verifiedCompletionDurationMs": 402584,
"allLeasesClosed": true,
"cleanupQuietPassed": true
},
"performanceNoiseOrGainPassed": false,
"historyTopology": "One 1000-delta foreground history under nine fresh background runs; not ten history-bearing runs."
},
"excluded": "Unqualified counter wrapper and resource instrumentation are not packaged."
}
server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts
+6 -9
View File
@@ -265,16 +265,13 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => {
const [delivered] = await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.id, wake.id));
expect(delivered.runId).toBeTruthy();
// The receipt can commit before execution enters liveRunExecutions.
// Observe actual adapter admission before awaiting its drain.
await vi.waitFor(async () => {
const current = await heartbeat.getRun(delivered.runId!);
expect(countExecuteCallsForRun(delivered.runId!), JSON.stringify({
status: current?.status, errorCode: current?.errorCode,
})).toBe(1);
}, { timeout: 5_000 });
await heartbeat.waitForRunExecutionDrain(delivered.runId!);
// Join the tracked dispatch even when live registration has not started.
await heartbeat.drainActiveRunExecutions();
expect(countExecuteCallsForRun(delivered.runId!)).toBe(1);
expect((await heartbeat.getRun(delivered.runId!))?.contextSnapshot).toMatchObject({ wakeCommentIds: [comment.id] });
expect(await heartbeat.getRun(delivered.runId!)).toMatchObject({
status: "succeeded",
contextSnapshot: { wakeCommentIds: [comment.id] },
});
await heartbeat.resumeQueuedRuns();
expect(countExecuteCallsForRun(delivered.runId!)).toBe(1);
expect((await db.select().from(issueComments).where(eq(issueComments.id, comment.id)))[0]?.body).toBe(comment.body);
tests/e2e/fixtures/native-ingestion-blocked-stop-authority.ts
+138
View File
@@ -0,0 +1,138 @@
// Bounded verbatim copy of the H5c authority/process proof, e431ec63; frozen measured source stays unchanged.
import { createHash } from "node:crypto";
import { createRequire } from "node:module";
import { lstat, readFile, readdir, realpath } from "node:fs/promises";
import { hostname } from "node:os";
import path from "node:path";
import { expect } from "@playwright/test";
export async function runtimeAuthority(input: {
appRoot: string; companyId: string; environmentId: string; issueIds: string[]; transport: "local" | "sandbox-remote";
afterStop?: boolean; recordedFailureRunIds?: string[];
}) {
const configuration = process.env.PAPERCLIP_E2E_SERVER_CONFIG;
if (!configuration) throw new Error("Missing owned server configuration");
const config = JSON.parse(await readFile(configuration, "utf8"));
expect(config.database.mode).toBe("embedded-postgres");
const dataDirectory = await realpath(config.database.embeddedPostgresDataDir);
const ownedHome = await realpath(path.resolve(path.dirname(configuration), "../.."));
expect(dataDirectory.startsWith(`${ownedHome}${path.sep}`)).toBe(true);
const owner = (await readFile(path.join(dataDirectory, "postmaster.pid"), "utf8")).split(/\r?\n/);
expect(await realpath(owner[1])).toBe(dataDirectory);
const port = Number(owner[3]);
expect(Number.isSafeInteger(port) && port > 0 && port <= 65535).toBe(true);
const require = createRequire(path.join(input.appRoot, "packages/db/package.json"));
const postgres = require("postgres");
const sql = postgres({ host: "127.0.0.1", port, database: "paperclip", username: "paperclip", password: "paperclip",
max: 1, connect_timeout: 10, connection: { application_name: "native_ingestion_remote_authority_proof" } });
let proof;
try {
proof = await sql.begin("isolation level repeatable read read only", async (tx: any) => {
const settings = await tx`select current_setting('transaction_read_only') as read_only,
current_setting('transaction_isolation') as isolation`;
expect(settings[0].read_only).toBe("on");
expect(settings[0].isolation).toBe("repeatable read");
if (input.recordedFailureRunIds) {
const queryStartedAtMs = Date.now();
const rows = await tx`select r.id as run_id, r.company_id, r.agent_id, r.runtime_mode, r.status, r.native_phase,
r.error_code as current_error_code, r.error is not null as current_has_error,
jsonb_typeof(r.result_json->'recoveredExecutionFailure') as recovered_type,
r.result_json->'recoveredExecutionFailure'->>'schema' as recovered_schema,
r.result_json->'recoveredExecutionFailure'->>'errorCode' as recovered_error_code,
r.result_json->'recoveredExecutionFailure'->>'observedAt' as recovered_observed_at,
f.attempt as finalization_attempt
from heartbeat_runs r left join native_run_finalizations f on f.run_id = r.id and f.company_id = r.company_id
where r.company_id = ${input.companyId} and r.id = any(${input.recordedFailureRunIds}::uuid[])`;
return { kind: "recorded-run-failure-codes", queryStartedAtMs, checkedAtMs: Date.now(),
transaction: "READ ONLY REPEATABLE READ", observedReadOnly: settings[0].read_only,
observedIsolation: settings[0].isolation, rows };
}
const rows = await tx`select r.id as run_id, r.company_id, r.agent_id, r.native_issue_id as issue_id, r.runtime_mode,
r.native_session_id, r.runner_instance_id, r.status, r.process_pid, r.process_group_id, r.process_started_at,
r.runner_profile_json->'nativeExecutionInput'->'binding'->>'executionWorkspaceId' as admitted_execution_workspace_id,
r.result_json->'nativeCancellation'->>'dispatchState' as cancellation_dispatch_state,
r.result_json->'nativeCancellation' as native_cancellation,
i.seq as process_identity_seq, i.payload as process_identity, i.source_event_id as process_identity_source_event_id,
s.seq as process_stopped_seq, s.payload as process_stopped, s.source_event_id as process_stopped_source_event_id,
p.seq as process_lifecycle_seq, p.event_type as process_lifecycle_event_type, p.source_event_id as process_lifecycle_source_event_id,
l.id as lease_id, l.environment_id, l.provider, l.provider_lease_id,
l.status as lease_status, l.cleanup_status as lease_cleanup_status, l.released_at as lease_released_at,
c.part as checkpoint_part, octet_length(c.material::text) as checkpoint_header_bytes
from heartbeat_runs r
join environment_leases l on l.heartbeat_run_id = r.id and l.company_id = r.company_id
left join native_runner_checkpoints c on c.run_id = r.id and c.company_id = r.company_id
and c.issue_id = r.native_issue_id and c.part = 'header'
left join lateral (select seq, payload, source_event_id from heartbeat_run_events e where e.run_id = r.id
and e.company_id = r.company_id and e.event_type = 'native.process_identity_recorded'
and e.source_event_id is null order by seq desc limit 1) i on true
left join lateral (select seq, payload, source_event_id from heartbeat_run_events e where e.run_id = r.id
and e.company_id = r.company_id and e.event_type = 'native.local_process_stopped'
and e.source_event_id is null order by seq desc limit 1) s on true
left join lateral (select seq, event_type, source_event_id from heartbeat_run_events e where e.run_id = r.id
and e.company_id = r.company_id and e.event_type in ('native.process_start_requested',
'native.process_identity_recorded', 'native.local_process_stopped')
and e.source_event_id is null order by seq desc limit 1) p on true
where r.company_id = ${input.companyId} and r.native_issue_id = any(${input.issueIds}::uuid[])`;
expect(rows).toHaveLength(input.issueIds.length);
expect(new Set(rows.map((row: any) => row.issue_id)).size).toBe(input.issueIds.length);
for (const row of rows) {
expect(row.company_id).toBe(input.companyId);
expect(row.runtime_mode).toBe("native");
expect(row.environment_id).toBe(input.environmentId);
if (input.transport === "sandbox-remote") {
expect(row.provider).toBe("fake-plugin");
expect(row.provider_lease_id).toMatch(/^fake-plugin:\/\//);
expect(row.checkpoint_part).toBe("header");
expect(Number(row.checkpoint_header_bytes)).toBeGreaterThan(0);
} else {
expect(row.provider).toBe("local"); expect(row.provider_lease_id).toBeNull();
expect(row.checkpoint_part, "Local raw transport keeps its existing host-file journal authority").toBeNull();
expect(row.process_identity?.targetKind).toBe("local"); expect(row.process_identity?.hostname).toBe(hostname());
expect(row.process_identity_source_event_id).toBeNull();
expect(BigInt(String(row.process_identity_seq))).toBeGreaterThan(0n);
expect(row.admitted_execution_workspace_id).toMatch(/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i);
if (row.process_started_at) expect(row.process_identity.processStartedAt).toBe(new Date(row.process_started_at).toISOString());
}
}
return { kind: input.transport === "local" ? "host-file-journal" : "database-controller-checkpoint",
checkedBeforeMeasuredEpoch: !input.afterStop, checkedAtMs: Date.now(), transaction: "READ ONLY REPEATABLE READ",
observedReadOnly: settings[0].read_only, observedIsolation: settings[0].isolation, rows };
});
} finally { await sql.end({ timeout: 5 }); }
const connectionClosedAtMs = Date.now();
let localJournal;
if (input.transport === "local" && !input.afterStop && !input.recordedFailureRunIds) {
expect(input.issueIds).toHaveLength(1);
const base = await realpath(path.join(path.dirname(configuration), "runtime/paperclip-runner/durable-sessions"));
expect(base.startsWith(`${ownedHome}${path.sep}`)).toBe(true);
const directories = (await readdir(base, { withFileTypes: true })).filter(entry => entry.isDirectory() && /^[a-f0-9]{64}$/.test(entry.name));
expect(directories.length, "Bound the owned fixture journal metadata search").toBeLessThanOrEqual(256);
for (const directory of directories) {
const file = path.join(base, directory.name, "control-plane/control-plane-state.json");
const metadata = await lstat(file).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return null; throw error;
});
if (!metadata) continue;
expect(metadata.isFile() && !metadata.isSymbolicLink() && metadata.size <= 16 * 1024 * 1024).toBe(true);
expect((await realpath(file)).startsWith(`${base}${path.sep}`)).toBe(true);
const bytes = await readFile(file), state = JSON.parse(bytes.toString("utf8"));
if (state.identity?.runId !== proof!.rows[0].run_id) continue;
expect(localJournal, "Exactly one owned local journal must match the actual run").toBeUndefined();
expect(state.schema).toBe("paperclip.runner.durable.control-plane-state.v1");
expect(state.checkpointEventPolicy).not.toBe("control-v1");
expect(state.identity.runnerInstanceId).toBe(proof!.rows[0].runner_instance_id);
expect(state.identity.normalizedSessionId).toBe(proof!.rows[0].native_session_id);
expect(state.identity.environmentLeaseId).toBe(proof!.rows[0].admitted_execution_workspace_id);
localJournal = { file, sha256: createHash("sha256").update(bytes).digest("hex"), bytes: bytes.length,
schema: state.schema, identity: Object.fromEntries(["runnerInstanceId", "environmentLeaseId", "runId", "normalizedSessionId", "turnId", "itemId"]
.map(key => [key, state.identity[key]])), checkpointEventPolicy: state.checkpointEventPolicy, checkedAtMs: Date.now() };
}
expect(localJournal, "Missing actual owned local raw journal").toBeTruthy();
}
return { ...proof, connectionClosedAtMs, localJournal };
}
export const processAbsent = (pid: number) => {
expect(Number.isSafeInteger(pid) && Math.abs(pid) > 1).toBe(true);
try { process.kill(pid, 0); return false; }
catch (error) { if ((error as NodeJS.ErrnoException).code === "ESRCH") return true; throw error; }
};
tests/e2e/fixtures/native-ingestion-blocked-stop-clock.ts
+127
View File
@@ -0,0 +1,127 @@
// Gate and two-frame observer copied from frozen H5c e431ec63, narrowed to one owned participant.
import { createServer, type ServerResponse } from "node:http";
import { expect, type Page } from "@playwright/test";
export type SeenMarker = { markerCode: string; index: number; visibleMs: number };
export type Emission = { nonce: string; markerCode: string; index: number; intendedMs: number; emittedMs: number; receivedAtMs: number };
type ClockStream = { nonce: string; markerCode: string; opened: boolean; hello: boolean; ended: boolean; aborted: boolean;
partialBytes: number; records: Map<number, Emission>; error?: string; expectedLauncher: "host-local" | "staged-remote";
launcher?: string; providerPid?: number; providerReadyAtMs?: number; clockHelloAtMs?: number; readinessObservedAtMs?: number };
type ClockCase = { events: number; intervalMs: number; epochMs?: number; streams: Map<string, ClockStream>; errors: string[] };
export async function openClock(page: Page) {
const pending = new Map<string, Map<string, ServerResponse>>();
const clocks = new Map<string, ClockCase>();
let markerCounter = 0;
const nextMarkerCode = () => {
expect(markerCounter).toBeLessThan(36 ** 2);
return (markerCounter++).toString(36).padStart(2, "0");
};
const gate = createServer((req, res) => {
const url = new URL(req.url!, "http://127.0.0.1");
const route = url.pathname.match(/^\/native-ingestion-gate\/([a-f0-9]{32})(\/clock)?$/);
const key = route?.[1], participant = url.searchParams.get("participant");
const state = key ? clocks.get(key) : undefined;
const stream = participant ? state?.streams.get(participant) : undefined;
if (!key || !participant || !state || !stream) { res.writeHead(404).end(); return; }
if (!route?.[2] && req.method === "GET") {
if (pending.get(key)!.has(participant)) { res.writeHead(409).end(); return; }
stream.providerReadyAtMs = Date.now();
pending.get(key)!.set(participant, res);
res.on("close", () => pending.get(key)?.delete(participant));
return;
}
if (!route?.[2] || req.method !== "POST" || stream.opened) { res.writeHead(409).end(); return; }
stream.opened = true;
let buffered = "";
req.setEncoding("utf8");
const fail = (message: string) => {
stream.error = message; state.errors.push(`${participant}: ${message}`);
if (!res.writableEnded) res.writeHead(400).end();
};
const line = (text: string) => {
const value = JSON.parse(text);
if (value.nonce !== participant || value.markerCode !== stream.markerCode) throw new Error("Clock identity mismatch");
if (!stream.hello) {
if (value.type !== "hello" || value.events !== state.events || value.intervalMs !== state.intervalMs
|| value.launcher !== stream.expectedLauncher || !Number.isSafeInteger(value.providerPid) || value.providerPid <= 1
|| Object.keys(value).sort().join(",") !== "events,intervalMs,launcher,markerCode,nonce,providerPid,type") throw new Error("Clock hello mismatch");
stream.launcher = value.launcher; stream.providerPid = value.providerPid;
stream.hello = true; stream.clockHelloAtMs = Date.now(); return;
}
if (value.type !== "emission" || !Number.isSafeInteger(value.index) || value.index < 0 || value.index >= state.events
|| !Number.isSafeInteger(value.intendedMs) || !Number.isSafeInteger(value.emittedMs) || value.emittedMs <= 0
|| value.intendedMs !== state.epochMs! + value.index * state.intervalMs) throw new Error("Invalid original emission clock");
if (stream.records.has(value.index)) throw new Error("Duplicate or conflicting original emission clock");
stream.records.set(value.index, { nonce: participant, markerCode: stream.markerCode, index: value.index,
intendedMs: value.intendedMs, emittedMs: value.emittedMs, receivedAtMs: Date.now() });
};
req.on("data", (chunk: string) => {
if (stream.error) return;
try {
buffered += chunk;
for (;;) {
const end = buffered.indexOf("\n"); if (end < 0) break;
const text = buffered.slice(0, end); buffered = buffered.slice(end + 1);
if (text.length > 1024) throw new Error("Clock record exceeds bounded metadata size");
if (text) line(text);
}
if (buffered.length > 1024) throw new Error("Clock record exceeds bounded metadata size");
} catch (error) { fail(String(error)); }
});
req.on("end", () => {
stream.ended = true; stream.partialBytes = Buffer.byteLength(buffered);
if (buffered.length) fail("Clock stream ended with a partial record");
if (!stream.error && !res.writableEnded) res.writeHead(200).end();
});
req.on("aborted", () => { stream.aborted = true; stream.partialBytes = Buffer.byteLength(buffered); });
req.on("error", error => { if (!stream.aborted) fail(String(error)); });
});
await new Promise<void>(resolve => gate.listen(0, "127.0.0.1", resolve));
const address = gate.address();
if (!address || typeof address === "string") throw new Error("Local ingestion gate failed");
const base = `http://127.0.0.1:${address.port}/native-ingestion-gate/`;
await page.addInitScript(() => {
const seen = new Set<string>(), visible: SeenMarker[] = [];
Object.assign(window, { nativeIngestionVisible: visible });
let scheduled = false, dirty = false;
const snapshot = () => {
const text = document.body?.innerText ?? "";
const pattern = /\[([a-z0-9]{2})([a-z0-9]{1,3})\]/g;
const entries: Omit<SeenMarker, "visibleMs">[] = [];
for (const match of text.matchAll(pattern)) {
const index = Number.parseInt(match[2], 36);
const key = `${match[1]}:${index}`;
if (seen.has(key)) continue;
entries.push({ markerCode: match[1], index });
}
return entries;
};
const schedule = () => {
dirty = true;
if (scheduled) return;
scheduled = true;
requestAnimationFrame(() => {
dirty = false;
const entries = snapshot();
requestAnimationFrame(() => {
// Only stamp the previous frame's snapshot. A token inserted
// between frames has not had this snapshot's paint opportunity.
const visibleMs = Date.now();
for (const entry of entries) {
const key = `${entry.markerCode}:${entry.index}`;
if (seen.has(key)) continue;
seen.add(key); visible.push({ ...entry, visibleMs });
}
scheduled = false;
if (dirty) schedule();
});
});
};
const install = () => {
new MutationObserver(schedule).observe(document.body, { subtree: true, childList: true, characterData: true });
schedule();
};
if (document.body) install(); else addEventListener("DOMContentLoaded", install, { once: true });
});
return { pending, clocks, nextMarkerCode, base, gate };
}
tests/e2e/fixtures/native-ingestion-blocked-stop-db.ts
+43
View File
@@ -0,0 +1,43 @@
import { createRequire } from "node:module";
import { readFile, realpath } from "node:fs/promises";
import path from "node:path";
import { expect } from "@playwright/test";
export async function ownedDatabase(appRoot: string, name: string) {
const configuration = process.env.PAPERCLIP_E2E_SERVER_CONFIG;
if (!configuration) throw new Error("Missing disposable application configuration");
const config = JSON.parse(await readFile(configuration, "utf8"));
expect(config.database.mode).toBe("embedded-postgres");
const home = await realpath(path.resolve(path.dirname(configuration), "../.."));
const directory = await realpath(config.database.embeddedPostgresDataDir);
expect(directory.startsWith(`${home}${path.sep}`)).toBe(true);
const owner = (await readFile(path.join(directory, "postmaster.pid"), "utf8")).split(/\r?\n/);
expect(await realpath(owner[1])).toBe(directory);
const port = Number(owner[3]);
expect(Number.isSafeInteger(port) && port > 0 && port <= 65535).toBe(true);
const postgres = createRequire(path.join(appRoot, "packages/db/package.json"))("postgres");
return postgres({ host: "127.0.0.1", port, database: "paperclip", username: "paperclip", password: "paperclip",
max: 1, connect_timeout: 10, connection: { application_name: name } });
}
export async function cancellationSnapshot(sql: any, companyId: string, runId: string) {
const startedAtMs = Date.now();
const rows = await sql`select id as run_id, company_id, native_issue_id as issue_id, status,
result_json->'nativeCancellation' as intent,
result_json->'executionCancellation' as shared_execution_cancellation,
result_json->'cancellation' as run_cancellation,
(select jsonb_build_object('id', d.id, 'companyId', d.company_id, 'runId', d.run_id, 'issueId', d.issue_id,
'reasonCode', d.reason_code) from status_decisions d where d.company_id = r.company_id and d.run_id = r.id
and d.issue_id = r.native_issue_id and d.id::text = r.result_json->'nativeCancellation'->>'decisionId') as shared_decision,
(select coalesce(jsonb_agg(jsonb_build_object('id', a.id, 'action', a.action, 'details', a.details)), '[]'::jsonb)
from activity_log a where a.company_id = r.company_id and a.run_id = r.id
and a.action in ('native.cancellation_intent_recorded', 'native.cancellation_dispatch_acknowledged')) as audits
from heartbeat_runs r where r.company_id = ${companyId} and r.id = ${runId}`;
expect(rows).toHaveLength(1);
return { ...rows[0], queryStartedAtMs: startedAtMs, observedAtMs: Date.now() };
}
export async function blockers(sql: any, holderPid: number) {
return sql`select pid, application_name, state, wait_event_type, wait_event, query, query_start,
pg_blocking_pids(pid) as blocking_pids from pg_stat_activity
where datname = current_database() and pid <> pg_backend_pid()
and ${holderPid} = any(pg_blocking_pids(pid))`;
}
tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.mjs
+394
View File
@@ -0,0 +1,394 @@
import assert from 'node:assert/strict';
import { createHash } from 'node:crypto';
export const PROTOCOL = 'blocked-stop-history-v1';
export const WORKLOAD = Object.freeze({ prelude: 1000, events: 256, intervalMs: 100, visibleBeforeLock: 3,
heldAfterActualRequestMs: 1000, pollIntervalMs: 50, readinessMs: 300000, streamMs: 120000,
terminalMs: 180000, acknowledgementAfterSettlementMs: 30000, leaseAfterSettlementMs: 30000, invocationMs: 900000 });
export const WORKLOAD_SHA256 = createHash('sha256').update(JSON.stringify(WORKLOAD)).digest('hex');
export const METRICS = ['actionToAckMs', 'releaseAckToAbsenceMs', 'releaseAckToAckMs'];
export function isAppendLockQuery(query) {
// Shared B0/B1/B2 projection prefix, unlike cancellation's full-row SELECT.
const sql = String(query).replace(/\s+/g, ' ').trim();
return /^select "company_id", "agent_id"(?:,| from)/i.test(sql)
&& / from "heartbeat_runs" where "heartbeat_runs"\."id" = \$\d+.* for update/i.test(sql);
}
export function assertHeldSample(sample, holderPid, expectedProcesses) {
assert.equal(sample.httpPending, true, 'Stop HTTP must remain pending while its durable barrier is locked');
assert.equal(sample.intent, null, 'No committed intent before releasing the run lock');
assert.deepEqual(sample.auditIds, [], 'No committed cancellation audit before releasing the run lock');
assert.equal(sample.processesAlive, true, 'No physical provider Stop before the durable intent barrier');
assert(expectedProcesses?.runner && expectedProcesses.provider, 'Missing original owned process births');
assert.deepEqual(sample.processes?.runner, expectedProcesses.runner);
assert.deepEqual(sample.processes?.provider, expectedProcesses.provider);
assert.equal(sample.processes?.group, expectedProcesses.group);
assert.equal(sample.processes?.providerGroup, expectedProcesses.providerGroup);
assert.equal(sample.processes?.runnerGroupAlive, true);
assert.equal(sample.processes?.providerGroupAlive, true);
assert.equal(sample.clockOpen, true, 'The unchanged provider fixture closes its clock on turn/interrupt');
assert(sample.blockers.some(row => row.blocking_pids.includes(holderPid)), 'Missing independent PostgreSQL blocker');
}
// Public cancellation supports active/scheduled runs and native retryable failures.
// A receipt is not physical Stop proof; the independent process/lease oracle remains required.
export function cleanupCancellationEligible(run) {
return ['queued', 'running', 'scheduled_retry'].includes(run.status)
|| (run.status === 'failed' && run.runtimeMode === 'native' && run.nativePhase === 'retryable_failure');
}
export async function cleanupOwnedRunBeforeClockClose({ companyId, runId, readRun, cancelRun, closeClock, record }) {
const snapshot = run => {
assert.equal(run.id, runId); assert.equal(run.companyId, companyId);
return { id: run.id, companyId: run.companyId, status: run.status, runtimeMode: run.runtimeMode,
nativePhase: run.nativePhase ?? null, errorCode: run.errorCode ?? null,
nativeCancellation: run.resultJson?.nativeCancellation ?? null };
};
try {
if (readRun) {
const before = await readRun(); record.before = snapshot(before);
record.cancel = { attempted: cleanupCancellationEligible(before) };
if (record.cancel.attempted) {
const response = await cancelRun();
record.cancel.httpStatus = response.status; record.cancel.response = snapshot(response.run);
}
record.after = snapshot(await readRun());
}
} catch (error) { record.error = String(error); throw error; }
finally {
record.clockCloseStartedAtMs = Date.now();
try { await closeClock(); record.clockClosedAtMs = Date.now(); }
catch (error) { record.clockCloseError = String(error); throw error; }
}
}
export function assertIntentAudit(row) {
const intent = row.intent;
assert.equal(intent?.schema, 'paperclip.native-cancellation.v1');
assert.equal(intent.companyId, row.company_id); assert.equal(intent.runId, row.run_id);
assert.equal(intent.issueId, row.issue_id);
assert.equal(intent.scope, 'run'); assert.equal(intent.reasonCode, 'cancellation_run_only');
if (intent.decisionId != null) {
assert.equal(row.shared_decision?.id, intent.decisionId);
assert.equal(row.shared_decision.companyId, row.company_id); assert.equal(row.shared_decision.runId, row.run_id);
assert.equal(row.shared_decision.issueId, row.issue_id);
}
assert(row.audits.some(a => a.id === intent.intentAuditId && a.action === 'native.cancellation_intent_recorded'
&& a.details?.intentId === intent.intentId), 'Intent must match its committed own audit');
if (intent.dispatchState === 'acknowledged') {
assert.equal(intent.dispatched, true);
assert(row.audits.some(a => a.id === intent.acknowledgementAuditId
&& a.action === 'native.cancellation_dispatch_acknowledged' && a.details?.intentId === intent.intentId
&& a.details.intentAuditId === intent.intentAuditId && a.details.dispatched === true
&& (a.details.decisionId ?? null) === (intent.decisionId ?? null)), 'ACK must match its own committed audit and decision');
}
}
export function metrics(evidence) {
assert.equal(evidence.protocol, PROTOCOL); assert.equal(evidence.workloadSha256, WORKLOAD_SHA256);
assert.equal(evidence.complete, true); assert.equal(evidence.cleanup?.complete, true);
const t = evidence.timing;
for (const k of ['actionAtMs', 'requestAtMs', 'releaseRequestedAtMs', 'releaseCommitAckAtMs', 'absenceObservedAtMs', 'ackObservedAtMs'])
assert(Number.isSafeInteger(t[k]) && t[k] > 0, `Missing ${k}`);
assert(t.actionAtMs <= t.requestAtMs); assert(t.requestAtMs + WORKLOAD.heldAfterActualRequestMs <= t.releaseRequestedAtMs);
assert(t.releaseRequestedAtMs <= t.releaseCommitAckAtMs); assert(t.releaseCommitAckAtMs <= t.absenceObservedAtMs);
assert(t.releaseCommitAckAtMs <= t.ackObservedAtMs);
return { actionToAckMs: t.ackObservedAtMs - t.actionAtMs,
releaseAckToAbsenceMs: t.absenceObservedAtMs - t.releaseCommitAckAtMs,
releaseAckToAckMs: t.ackObservedAtMs - t.releaseCommitAckAtMs,
releaseRequestToAbsenceMs: t.absenceObservedAtMs - t.releaseRequestedAtMs,
releaseRequestToAckMs: t.ackObservedAtMs - t.releaseRequestedAtMs };
}
export function registration() {
const stages = [['b0-aa','b0','b0'], ['b1-aa','b1','b1'], ['b0-b1','b0','b1'], ['b1-b2','b1','b2'], ['b0-b2','b0','b2']];
return stages.flatMap(([stage,a,b]) => Array.from({ length: 5 }, (_, pair) => (pair % 2 ? [b,a] : [a,b])
.map((variant, index) => ({ stage, pair: pair + 1, index, variant, role: pair % 2 ? (index ? 'baseline' : 'candidate') : (index ? 'candidate' : 'baseline') })))).flat();
}
const median = values => {
assert(values.length > 0);
const sorted = [...values].sort((a,b) => a-b), middle = Math.floor(sorted.length / 2);
return sorted.length % 2 ? sorted[middle] : (sorted[middle - 1] + sorted[middle]) / 2;
};
const spread = values => ({ minimumMs: Math.min(...values), maximumMs: Math.max(...values),
spreadMs: Math.max(...values) - Math.min(...values) });
const hashReport = evidence => createHash('sha256').update(JSON.stringify(evidence)).digest('hex');
export const PRECISION_RULE = Object.freeze({ schema: 'blocked-stop-precision-v1',
pollIntervalMs: WORKLOAD.pollIntervalMs, floorMs: 2 * WORKLOAD.pollIntervalMs, pooledMedianFraction: 0.10,
pairedRule: 'maximum absolute difference <= budget', driftRule: 'absolute first-five vs last-five chronological median drift <= budget',
bracketRule: 'each observer bracket <= corresponding metric budget', unsuitableRule: 'retain all ten arms; no noise freeze or replacement',
scope: 'Measurement repeatability; not a product Stop deadline' });
export const PRECISION_RULE_SHA256 = createHash('sha256').update(JSON.stringify(PRECISION_RULE)).digest('hex');
export function observerBrackets(evidence) {
const t = evidence.timing, origin = t.releaseCommitAckAtMs;
const samples = evidence.observerSamples;
assert(Array.isArray(samples), 'Missing raw observer query/check samples');
const result = {};
for (const [kind, timestamp, negativeTimestamp] of [
['ack-query', 'ackObservedAtMs', 'lastAckNotObservedAtMs'],
['process-check', 'absenceObservedAtMs', 'lastProcessesPresentAtMs']]) {
const rows = samples.filter(row => row.kind === kind);
assert(rows.length > 0, `Missing ${kind} observer samples`);
let priorEnd = origin;
for (const [index,row] of rows.entries()) {
assert(Number.isSafeInteger(row.startedAtMs) && Number.isSafeInteger(row.completedAtMs));
assert(row.startedAtMs >= priorEnd && row.completedAtMs >= row.startedAtMs);
assert.equal(row.durationMs, row.completedAtMs - row.startedAtMs);
assert.equal(row.value, index === rows.length - 1, 'Retain every negative sample through the first successful observation');
priorEnd = row.completedAtMs;
}
const success = rows.at(-1), negative = rows.at(-2);
assert.equal(t[timestamp], success.completedAtMs);
if (negative) assert.equal(t[negativeTimestamp], negative.completedAtMs);
else assert(t[negativeTimestamp] === undefined, 'A negative timestamp needs its raw query/check sample');
// A negative query can read its snapshot at any point during execution;
// use its start as the conservative lower boundary and retain both ends.
const lowerAtMs = negative?.startedAtMs ?? origin;
result[kind] = { lowerAtMs, upperAtMs: success.completedAtMs,
widthMs: success.completedAtMs - lowerAtMs, lowerKind: negative ? 'last-negative-query-or-check-start' : 'release-COMMIT-ACK',
successfulStartedAtMs: success.startedAtMs, successfulDurationMs: success.durationMs,
...(negative ? { negativeStartedAtMs: negative.startedAtMs, negativeCompletedAtMs: negative.completedAtMs,
negativeDurationMs: negative.durationMs } : {}), sampleCount: rows.length };
}
return { actionToAckMs: result['ack-query'], releaseAckToAckMs: result['ack-query'],
releaseAckToAbsenceMs: result['process-check'] };
}
function validatedPairs(pairs) {
assert.equal(pairs.length, 5, 'Require five fresh pairs');
const validated = pairs.map(pair => {
assert.equal(pair.length, 2);
return pair.map(evidence => ({ evidence, values: assertFunctional(evidence), brackets: observerBrackets(evidence) }));
});
const chronological = validated.flat().sort((a,b) => a.evidence.startedAtMs - b.evidence.startedAtMs);
assert.equal(new Set(chronological.map(arm => arm.evidence.runId)).size, 10, 'Ten fresh owned runs are required');
assert.equal(new Set(chronological.map(arm => arm.evidence.companyId)).size, 10, 'Ten fresh disposable invocations are required');
for (const [index,arm] of chronological.entries()) {
assert(Number.isSafeInteger(arm.evidence.startedAtMs) && Number.isSafeInteger(arm.evidence.finishedAtMs));
assert(arm.evidence.finishedAtMs >= arm.evidence.startedAtMs);
if (index) assert(chronological[index-1].evidence.finishedAtMs < arm.evidence.startedAtMs,
'Arms must not overlap');
}
validated.forEach((pair,index) => {
assert.equal(pair[0].evidence.startedAtMs, chronological[2*index + index%2].evidence.startedAtMs,
'Pairs must alternate baseline-first then candidate-first in chronological order');
assert.equal(pair[1].evidence.startedAtMs, chronological[2*index + (index%2 ? 0 : 1)].evidence.startedAtMs);
});
return validated;
}
export function noiseEnvelope(pairs) {
const validated = validatedPairs(pairs), chronological = validated.flat().sort((a,b) => a.evidence.startedAtMs - b.evidence.startedAtMs);
for (const [index,arm] of chronological.entries()) {
assert(Number.isSafeInteger(arm.evidence.startedAtMs) && Number.isSafeInteger(arm.evidence.finishedAtMs));
assert(arm.evidence.finishedAtMs >= arm.evidence.startedAtMs);
if (index) assert(chronological[index-1].evidence.finishedAtMs < arm.evidence.startedAtMs,
'A/A requires ten distinct non-overlapping chronological arms');
}
const variants = new Set(chronological.map(arm => String(arm.evidence.variant).toLowerCase()));
assert.equal(variants.size, 1, 'A/A calibration must use one baseline variant');
const baselineVariant = [...variants][0]; assert(['b0','b1'].includes(baselineVariant));
const result = { schema: 'paperclip.native-ingestion.blocked-stop-noise.v1', protocol: PROTOCOL, baselineVariant,
workloadSha256: WORKLOAD_SHA256, precisionRule: PRECISION_RULE, precisionRuleSha256: PRECISION_RULE_SHA256,
sampleCounts: { pairs: 5, arms: 10 }, suitable: false, noiseFrozen: false, metrics: {},
arms: chronological.map(arm => ({ reportSha256: hashReport(arm.evidence), startedAtMs: arm.evidence.startedAtMs,
finishedAtMs: arm.evidence.finishedAtMs, values: arm.values, observerBrackets: arm.brackets })),
rawArms: chronological.map(arm => arm.evidence), pairBindings: validated.map(pair => pair.map(arm => hashReport(arm.evidence))),
sourceDigestScope: 'SHA256 of JSON.stringify(parsed raw arm); owning raw-byte seals are separately required' };
for (const metric of METRICS) {
const values = chronological.map(arm => arm.values[metric]), pooledMedianMs = median(values);
const baselineValues = validated.map(pair => pair[0].values[metric]), candidateValues = validated.map(pair => pair[1].values[metric]);
const baselineMedianMs = median(baselineValues), candidateMedianMs = median(candidateValues);
const differencesMs = validated.map(pair => pair[1].values[metric] - pair[0].values[metric]);
const maximumAbsolutePairedDifferenceMs = Math.max(...differencesMs.map(Math.abs));
const firstFiveMedianMs = median(values.slice(0,5)), lastFiveMedianMs = median(values.slice(5));
const absoluteChronologicalMedianDriftMs = Math.abs(lastFiveMedianMs - firstFiveMedianMs);
const precisionBudgetMs = Math.max(PRECISION_RULE.floorMs, PRECISION_RULE.pooledMedianFraction * pooledMedianMs);
const observerBracketWidthsMs = chronological.map(arm => arm.brackets[metric].widthMs);
const floorDominated = PRECISION_RULE.pooledMedianFraction * pooledMedianMs <= PRECISION_RULE.floorMs;
result.metrics[metric] = { sampleCount: 10, pairCount: 5, pooledMedianMs, baselineMedianMs, candidateMedianMs,
absoluteMedianDifferenceMs: Math.abs(candidateMedianMs - baselineMedianMs),
medianDifferencePercent: floorDominated ? null : 100 * (candidateMedianMs - baselineMedianMs) / baselineMedianMs,
...spread(values), pairedDifferencesMs: differencesMs, maximumAbsolutePairedDifferenceMs,
firstFiveMedianMs, lastFiveMedianMs, absoluteChronologicalMedianDriftMs, precisionBudgetMs,
observerBracketWidthsMs, maximumObserverBracketMs: Math.max(...observerBracketWidthsMs),
suitable: maximumAbsolutePairedDifferenceMs <= precisionBudgetMs && absoluteChronologicalMedianDriftMs <= precisionBudgetMs
&& observerBracketWidthsMs.every(width => width <= precisionBudgetMs),
noiseMs: null, floorDominated, resolutionLabel: floorDominated ? '100 ms resolution floor' : 'Observer brackets and budget reported',
subfloorOrP99ClaimsAllowed: false, percentageSpeedClaimsAllowed: !floorDominated };
}
result.suitable = METRICS.every(metric => result.metrics[metric].suitable);
result.noiseFrozen = result.suitable;
if (result.suitable) for (const metric of METRICS) result.metrics[metric].noiseMs = result.metrics[metric].maximumAbsolutePairedDifferenceMs;
return result;
}
export function assertNoiseEnvelope(noise) {
assert.equal(noise?.schema, 'paperclip.native-ingestion.blocked-stop-noise.v1');
assert.equal(noise.protocol, PROTOCOL); assert.equal(noise.workloadSha256, WORKLOAD_SHA256);
assert.equal(noise.precisionRuleSha256, PRECISION_RULE_SHA256); assert.deepEqual(noise.precisionRule, PRECISION_RULE);
assert.deepEqual(noise.sampleCounts, { pairs: 5, arms: 10 });
assert.equal(noise.suitable, true, 'Unsuitable A/A cannot freeze a noise envelope'); assert.equal(noise.noiseFrozen, true);
assert.equal(noise.rawArms?.length, 10);
// Recompute suitability and all numeric limits from retained raw arms. A
// caller cannot inflate a budget or flip a failed suitability flag.
const byDigest = new Map(noise.rawArms.map(arm => [hashReport(arm), arm]));
assert.equal(noise.pairBindings?.length, 5);
const logical = noise.pairBindings.map(pair => {
assert.equal(pair.length, 2);
return pair.map(digest => { const raw = byDigest.get(digest); assert(raw, 'Frozen arm digest mismatch'); return raw; });
});
const recomputed = noiseEnvelope(logical);
assert.equal(recomputed.suitable, true);
assert.deepEqual(noise.metrics, recomputed.metrics, 'Frozen noise or precision limit changed');
assert.equal(noise.baselineVariant, recomputed.baselineVariant);
assert.deepEqual(noise.arms, recomputed.arms, 'Frozen chronological arms changed');
return noise;
}
export function comparePairs(pairs, noise) {
const validated = validatedPairs(pairs); assertNoiseEnvelope(noise);
for (const [baseline] of validated) assert.equal(String(baseline.evidence.variant).toLowerCase(), noise.baselineVariant,
'Comparisons must use their own baseline calibration');
return Object.fromEntries(METRICS.map(metric => {
const calibrated = noise.metrics[metric], deltas = validated.map(([baseline,candidate]) => candidate.values[metric] - baseline.values[metric]);
const baselineValues = validated.map(pair => pair[0].values[metric]), candidateValues = validated.map(pair => pair[1].values[metric]);
const baselineMedianMs = median(baselineValues), candidateMedianMs = median(candidateValues);
const observerBracketWidthsMs = validated.map(pair => pair.map(arm => arm.brackets[metric].widthMs));
const floorDominated = calibrated.floorDominated || PRECISION_RULE.pooledMedianFraction * Math.min(baselineMedianMs,candidateMedianMs) <= PRECISION_RULE.floorMs;
return [metric, { sampleCounts: { pairs: 5, arms: 10 }, baselineMedianMs, candidateMedianMs, deltaMedianMs: median(deltas),
medianDifferencePercent: floorDominated ? null : 100 * (candidateMedianMs - baselineMedianMs) / baselineMedianMs,
deltasMs: deltas, noiseMs: calibrated.noiseMs, precisionBudgetMs: calibrated.precisionBudgetMs,
baselineSpread: spread(baselineValues), candidateSpread: spread(candidateValues), ...spread(deltas),
observerBracketWidthsMs, floorDominated, resolutionLabel: floorDominated ? '100 ms resolution floor' : 'Observer brackets and budget reported',
subfloorOrP99ClaimsAllowed: false, percentageSpeedClaimsAllowed: !floorDominated,
pass: deltas.every(delta => delta <= calibrated.noiseMs) && median(deltas) <= calibrated.noiseMs
&& observerBracketWidthsMs.flat().every(width => width <= calibrated.precisionBudgetMs) }];
}));
}
function assertOptionalShared(row, companyId, runId) {
assert.equal(row.company_id, companyId); assert.equal(row.run_id, runId);
const timestamp = value => typeof value === 'string' && /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z$/.test(value) && Number.isFinite(Date.parse(value));
const execution = row.shared_execution_cancellation;
if (execution != null) {
assert(execution && typeof execution === 'object' && !Array.isArray(execution));
assert(['requested','acknowledged'].includes(execution.state));
if (execution.state === 'requested') assert(timestamp(execution.requestedAt));
if (execution.state === 'acknowledged') assert(timestamp(execution.acknowledgedAt));
if (execution.requestedAt != null) assert(timestamp(execution.requestedAt));
if (execution.acknowledgedAt != null) assert(timestamp(execution.acknowledgedAt));
if (execution.requestedAt != null && execution.acknowledgedAt != null)
assert(Date.parse(execution.requestedAt) <= Date.parse(execution.acknowledgedAt));
}
const cancellation = row.run_cancellation;
if (cancellation != null) {
assert.equal(cancellation.source, 'operator'); assert.equal(cancellation.expected, true);
assert.equal(cancellation.initiator?.type, 'user');
assert(typeof cancellation.reason === 'string' && cancellation.reason.length > 0);
assert(timestamp(cancellation.recordedAt));
}
}
// Mirrors acknowledgedNativeStopExecutionHasStopped's two local identity states;
// independent PID/group absence is mandatory in either state, never inferred from ACK.
export function assertAuthoritativeRetirement(e) {
const before = e.authorityBefore?.rows?.[0], after = e.authorityAfter?.rows?.[0];
for (const row of [before, after]) {
assert.equal(row?.company_id, e.companyId); assert.equal(row?.run_id, e.runId);
assert.equal(row.issue_id, e.issueId); assert.equal(row.runtime_mode, 'native');
}
assert.equal(after.status, 'cancelled'); assert.equal(after.cancellation_dispatch_state, 'acknowledged');
assertIntentAudit(e.ack); assert.equal(e.ack.intent.dispatchState, 'acknowledged');
assert.equal(e.ack.company_id, e.companyId); assert.equal(e.ack.run_id, e.runId); assert.equal(e.ack.issue_id, e.issueId);
assert.deepEqual(after.native_cancellation, e.ack.intent, 'Snapshot must contain the same audited own ACK');
assert.deepEqual([e.processesAfter?.runnerAbsent, e.processesAfter?.groupAbsent,
e.processesAfter?.providerAbsent, e.processesAfter?.providerGroupAbsent], [true,true,true,true]);
assert.equal(e.processesAfter.group, e.processesBefore.group);
assert.equal(e.processesAfter.providerGroup, e.processesBefore.providerGroup);
for (const process of [e.processesBefore.runner, e.processesBefore.provider]) {
assert(Number.isSafeInteger(process.pid) && process.pid > 1 && process.birth);
assert(Number.isSafeInteger(process.group) && process.group > 1);
}
assert.equal(e.processesBefore.runner.group, e.processesBefore.group);
assert.equal(e.processesBefore.group, before.process_group_id);
assert.equal(e.processesBefore.runner.pid, before.process_pid);
assert.equal(e.processesBefore.provider.group, e.processesBefore.providerGroup);
const birth = value => {
if (value === null) return null;
assert(value instanceof Date || typeof value === 'string');
assert(Number.isFinite(new Date(value).getTime())); return new Date(value).toISOString();
};
const sequence = value => { assert(/^[1-9]\d*$/.test(String(value))); return BigInt(value); };
const originalBirth = birth(before.process_started_at), identitySeq = sequence(before.process_identity_seq);
assert(originalBirth !== null, 'Missing captured authoritative original process birth');
assert.equal(before.process_identity_source_event_id, null);
assert.equal(before.process_identity?.targetKind, 'local'); assert(before.process_identity.hostname);
assert.equal(before.process_identity.processPid, before.process_pid);
assert.equal(before.process_identity.processGroupId, before.process_group_id);
assert.equal(before.process_identity.processStartedAt, originalBirth);
assert.equal(after.process_identity_source_event_id, null);
assert.equal(sequence(after.process_identity_seq), identitySeq, 'A later identity is a different launch');
assert.deepEqual(after.process_identity, before.process_identity);
assert.equal(e.lease?.heartbeatRunId, e.runId); assert.equal(e.lease.companyId, e.companyId);
assert.equal(e.lease.id, before.lease_id); assert(before.lease_id);
assert.equal(after.lease_id, before.lease_id); assert.equal(after.environment_id, before.environment_id);
assert.equal(e.lease.provider, 'local'); assert.equal(e.lease.providerLeaseId, null);
assert.equal(after.provider, 'local'); assert.equal(after.provider_lease_id, null);
for (const lease of [e.lease, { releasedAt: after.lease_released_at, status: after.lease_status, cleanupStatus: after.lease_cleanup_status }]) {
assert(lease.releasedAt && ['released','expired','failed'].includes(lease.status)); assert.notEqual(lease.cleanupStatus, 'failed');
}
const cleared = after.process_pid === null && after.process_group_id === null;
if (cleared) assert.equal(after.process_started_at, null);
else {
assert.equal(after.process_pid, before.process_pid); assert.equal(after.process_group_id, before.process_group_id);
assert.equal(birth(after.process_started_at), originalBirth);
}
const noReceipt = [after.process_stopped_seq, after.process_stopped, after.process_stopped_source_event_id].every(value => value === null);
if (noReceipt) assert.equal(cleared, false, 'Cleared identity requires authoritative retirement evidence');
else {
assert(sequence(after.process_stopped_seq) > sequence(after.process_identity_seq));
assert.equal(after.process_stopped_source_event_id, null);
assert.deepEqual(after.process_stopped, { processPid: before.process_pid, processGroupId: before.process_group_id,
processStartedAt: originalBirth });
}
assert.equal(after.process_lifecycle_source_event_id, null);
assert.equal(after.process_lifecycle_event_type, noReceipt ? 'native.process_identity_recorded' : 'native.local_process_stopped');
assert.equal(sequence(after.process_lifecycle_seq), noReceipt ? identitySeq : sequence(after.process_stopped_seq),
'A later process start or identity invalidates prior retirement evidence');
return cleared ? 'cleared-identity-with-authoritative-retirement' : 'retained-identity-after-physical-exit';
}
export function assertFunctional(e) {
metrics(e); observerBrackets(e);
assert.equal(e.prelude?.individualDeltas, WORKLOAD.prelude); assert.equal(e.prelude?.uniqueIdentities, WORKLOAD.prelude);
assert(e.prelude.verifiedBeforeEpochAtMs < e.epochMs);
assert(e.visibleBeforeLock?.length >= WORKLOAD.visibleBeforeLock);
assert.equal(e.lock?.id, e.runId); assert.equal(e.lock.company_id, e.companyId);
assert(e.appendBlockedBeforeStop?.observedAtMs < e.timing.requestAtMs);
assert(e.appendBlockedBeforeStop.rows.some(row => isAppendLockQuery(row.query) && row.blocking_pids.includes(e.lock.holder_pid)));
assert(e.heldSamples?.length >= 2);
for (const sample of e.heldSamples) {
assertHeldSample(sample, e.lock.holder_pid, e.processesBefore); assertOptionalShared(sample, e.companyId, e.runId);
assert(sample.queryStartedAtMs >= e.timing.requestAtMs && sample.observedAtMs <= e.timing.releaseRequestedAtMs);
}
assert(e.heldSamples.at(-1).emittedCount > e.heldSamples[0].emittedCount, 'Provider must still emit through the stall');
assertIntentAudit(e.ack); assertOptionalShared(e.ack, e.companyId, e.runId);
assert.equal(e.ack.intent.dispatchState, 'acknowledged');
if (e.sharedSettlement != null) {
assert.equal(e.sharedSettlement.runId, e.runId); assert.equal(e.sharedSettlement.companyId, e.companyId);
assert.equal(e.sharedSettlement.status, 'cancelled');
assertOptionalShared({ company_id: e.sharedSettlement.companyId, run_id: e.sharedSettlement.runId,
shared_execution_cancellation: e.sharedSettlement.executionCancellation, run_cancellation: e.sharedSettlement.cancellation }, e.companyId, e.runId);
if (e.sharedSettlement.nativeCancellation != null) assert.equal(e.sharedSettlement.nativeCancellation.intentId, e.ack.intent.intentId);
}
assert.equal(e.ack.company_id, e.companyId); assert.equal(e.ack.run_id, e.runId); assert.equal(e.ack.issue_id, e.issueId);
assert.equal(e.stopHttpStatus, 200);
assert(e.timing.httpSettledAtMs >= e.timing.releaseRequestedAtMs);
assert.equal(e.authoritativeRetirementState, assertAuthoritativeRetirement(e));
const transcript = e.transcript;
assert.equal(transcript?.persistedAfterReload, true); assert.equal(transcript?.ordered, true);
assert.equal(transcript.resultAcceptedEvents, 0); assert.equal(transcript.preludeCount, WORKLOAD.prelude);
assert.equal(transcript.individualDeltas, transcript.deltas.length);
assert.equal(transcript.uniqueSourceIdentities, transcript.deltas.length);
const identities = new Set();
for (const [index,event] of transcript.deltas.entries()) {
assert.equal(event.runId, e.runId); assert.equal(event.eventType, 'item.delta');
assert(event.sourceInstanceId && event.sourceEventId && Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0);
identities.add(`${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`);
assert.equal(event.payload.text, `[${e.clock.markerCode}${index.toString(36)}] `);
assert(e.clock.records.some(record => record.index === index && record.nonce === e.clock.nonce));
}
assert.equal(identities.size, transcript.deltas.length);
for (const marker of transcript.seen) assert(marker.index >= 0 && marker.index < transcript.deltas.length);
assert(e.clock.ended || e.clock.aborted); assert(!e.clock.error);
assert.equal(e.measurement?.actionToAckIncludesForcedWait, true);
assert.equal(e.measurement?.recording?.trace, 'off'); assert.equal(e.measurement?.recording?.video, 'off');
return metrics(e);
}
tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.test.mjs
+333
View File
@@ -0,0 +1,333 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { PROTOCOL, WORKLOAD as W, WORKLOAD_SHA256, METRICS, isAppendLockQuery, assertHeldSample,
assertIntentAudit, metrics, registration, noiseEnvelope, comparePairs, assertFunctional, assertAuthoritativeRetirement, observerBrackets, assertNoiseEnvelope, PRECISION_RULE, cleanupCancellationEligible, cleanupOwnedRunBeforeClockClose } from './native-ingestion-blocked-stop-protocol.mjs';
const evidence = (shift = 0) => ({ protocol: PROTOCOL, workloadSha256: WORKLOAD_SHA256, complete: true, cleanup: { complete: true },
timing: { actionAtMs: 1000, requestAtMs: 1005, releaseRequestedAtMs: 2005, releaseCommitAckAtMs: 2007,
absenceObservedAtMs: 2030 + shift, ackObservedAtMs: 2040 + shift } });
const processes = () => ({ runner: { pid: 10, group: 10, birth: 'runner-birth' }, provider: { pid: 11, group: 11, birth: 'provider-birth' }, group: 10, providerGroup: 11 });
const sample = () => ({ processes: { ...processes(), runnerGroupAlive: true, providerGroupAlive: true }, httpPending: true, intent: null, auditIds: [], processesAlive: true, clockOpen: true, blockers: [{ blocking_pids: [88] }] });
const intentRow = () => ({ company_id: 'c', run_id: 'r', issue_id: 'i', intent: {
schema: 'paperclip.native-cancellation.v1', companyId: 'c', runId: 'r', issueId: 'i',
intentId: 'own', intentAuditId: 'a', dispatchState: 'acknowledged', acknowledgementAuditId: 'b',
scope: 'run', reasonCode: 'cancellation_run_only', dispatched: true, decisionId: null },
audits: [{ id: 'a', action: 'native.cancellation_intent_recorded', details: { intentId: 'own' } },
{ id: 'b', action: 'native.cancellation_dispatch_acknowledged', details: { intentId: 'own', intentAuditId: 'a', dispatched: true, decisionId: null } }] });
test('geometry fits the unchanged 4,000 character bound, includes existing emitted history', () => {
const chars = '.'.repeat(W.prelude - 1) + ` PRELUDE_${'a'.repeat(32)}_READY `
+ Array.from({ length: W.events }, (_, index) => `[00${index.toString(36)}] `).join('');
assert(chars.length < 4000); assert.equal(W.prelude, 1000); assert.equal(W.events, 256); assert.equal(W.intervalMs, 100);
});
test('B0 and candidate append SQL match while cancellation full-row and unrelated SQL do not', () => {
for (const extra of ['', ', "next_event_seq", "event_cursor_projection_seq"']) assert(isAppendLockQuery(
`select "company_id", "agent_id"${extra} from "heartbeat_runs" where "heartbeat_runs"."id" = $1 limit $2 for update`));
assert(!isAppendLockQuery('select "id", "company_id", "agent_id" from "heartbeat_runs" where "heartbeat_runs"."id" = $1 for update'));
assert(!isAppendLockQuery('select "company_id", "agent_id" from "heartbeat_runs" where "heartbeat_runs"."id" = $1'));
});
test('held sample requires every barrier and physical liveness proof', () => {
assertHeldSample(sample(), 88, processes());
for (const patch of [{ httpPending: false }, { intent: {} }, { auditIds: ['a'] }, { processesAlive: false }, { clockOpen: false }, { blockers: [] }])
assert.throws(() => assertHeldSample({ ...sample(), ...patch }, 88, processes()));
assert.throws(() => assertHeldSample(sample(), 89, processes()));
});
test('ACK correlates both committed audits to the same owned intent', () => {
assertIntentAudit(intentRow());
for (const mutate of [row => { row.intent.runId = 'foreign'; }, row => { row.audits.pop(); },
row => { row.audits[1].details.intentId = 'foreign'; }, row => { row.intent.intentAuditId = 'missing'; }]) {
const row = intentRow(); mutate(row); assert.throws(() => assertIntentAudit(row));
}
});
test('full action latency retains forced wait; release metrics use COMMIT ACK and also retain request-origin bounds', () => {
assert.deepEqual(metrics(evidence()), { actionToAckMs: 1040, releaseAckToAbsenceMs: 23, releaseAckToAckMs: 33,
releaseRequestToAbsenceMs: 25, releaseRequestToAckMs: 35 });
for (const key of ['actionAtMs','requestAtMs','releaseRequestedAtMs','releaseCommitAckAtMs','absenceObservedAtMs','ackObservedAtMs']) {
const e = evidence(); delete e.timing[key]; assert.throws(() => metrics(e));
}
const e = evidence(); e.timing.releaseRequestedAtMs--; assert.throws(() => metrics(e));
e.timing.releaseRequestedAtMs++; e.complete = false; assert.throws(() => metrics(e));
});
test('registered companion is 50 separate alternating arms with fresh B0 and B1 A/A', () => {
const arms = registration(); assert.equal(arms.length, 50);
for (const stage of ['b0-aa','b1-aa','b0-b1','b1-b2','b0-b2']) {
const rows = arms.filter(a => a.stage === stage); assert.equal(rows.length, 10);
for (let pair = 1; pair <= 5; pair++) assert.equal(rows.filter(a => a.pair === pair).length, 2);
assert.deepEqual(rows.slice(0, 4).map(a => a.role), ['baseline','candidate','candidate','baseline']);
}
});
const functional = () => {
const e = evidence(), identity = { company_id: 'c', run_id: 'r', issue_id: 'i', runtime_mode: 'native', process_pid: 10, process_group_id: 10,
process_started_at: '2026-11-04T00:00:00.000Z', process_identity_seq: '1', process_identity_source_event_id: null,
process_identity: { targetKind: 'local', hostname: 'fixture-host', processPid: 10, processGroupId: 10, processStartedAt: '2026-11-04T00:00:00.000Z' },
lease_id: 'lease', environment_id: 'environment', provider: 'local', provider_lease_id: null };
return { ...e, variant: 'b0', startedAtMs: 100, finishedAtMs: 3000, companyId: 'c', runId: 'r', issueId: 'i', epochMs: 900,
prelude: { individualDeltas: 1000, uniqueIdentities: 1000, verifiedBeforeEpochAtMs: 850 },
visibleBeforeLock: [0,1,2], lock: { id: 'r', company_id: 'c', holder_pid: 88 },
appendBlockedBeforeStop: { observedAtMs: 1004, rows: [{ query: 'select "company_id", "agent_id" from "heartbeat_runs" where "heartbeat_runs"."id" = $1 for update', blocking_pids: [88] }] },
heldSamples: [{ ...sample(), company_id: 'c', run_id: 'r', queryStartedAtMs: 1010, observedAtMs: 1015, emittedCount: 3 },
{ ...sample(), company_id: 'c', run_id: 'r', queryStartedAtMs: 1995, observedAtMs: 2004, emittedCount: 13 }], ack: intentRow(), stopHttpStatus: 200,
timing: { ...e.timing, httpSettledAtMs: 2042, lastAckNotObservedAtMs: 2036, lastProcessesPresentAtMs: 2026 },
observerSamples: [{ kind: 'ack-query', startedAtMs: 2035, completedAtMs: 2036, durationMs: 1, value: false },
{ kind: 'ack-query', startedAtMs: 2039, completedAtMs: 2040, durationMs: 1, value: true },
{ kind: 'process-check', startedAtMs: 2025, completedAtMs: 2026, durationMs: 1, value: false },
{ kind: 'process-check', startedAtMs: 2029, completedAtMs: 2030, durationMs: 1, value: true }], processesBefore: processes(), processesAfter: { runnerAbsent: true, groupAbsent: true, providerAbsent: true, providerGroupAbsent: true, group: 10, providerGroup: 11 },
authorityBefore: { rows: [{ ...identity }] },
authorityAfter: { rows: [{ ...structuredClone(identity), status: 'cancelled', cancellation_dispatch_state: 'acknowledged', native_cancellation: intentRow().intent,
process_pid: null, process_group_id: null, process_started_at: null, process_stopped_seq: '20',
process_stopped_source_event_id: null, process_stopped: { processPid: 10, processGroupId: 10, processStartedAt: identity.process_started_at },
process_lifecycle_seq: '20', process_lifecycle_event_type: 'native.local_process_stopped', process_lifecycle_source_event_id: null,
lease_status: 'released', lease_cleanup_status: 'success', lease_released_at: 'time' }] },
authoritativeRetirementState: 'cleared-identity-with-authoritative-retirement',
lease: { id: 'lease', heartbeatRunId: 'r', companyId: 'c', provider: 'local', providerLeaseId: null, releasedAt: 'time', status: 'released', cleanupStatus: 'success' },
transcript: { individualDeltas: 3, uniqueSourceIdentities: 3, persistedAfterReload: true, ordered: true, resultAcceptedEvents: 0,
preludeCount: 1000, seen: [{ index: 0 },{ index: 1 },{ index: 2 }], deltas: [0,1,2].map(index => ({ runId: 'r', eventType: 'item.delta',
sourceInstanceId: 'source', sourceEventId: `id${index}`, sourceSeq: index+1, payload: { text: `[00${index}] ` } })) },
clock: { markerCode: '00', nonce: 'nonce', ended: true, records: [0,1,2].map(index => ({ index, nonce: 'nonce' })) },
measurement: { actionToAckIncludesForcedWait: true, recording: { trace: 'off', video: 'off' } } };
};
test('functional oracle accepts complete correlated proof and rejects plausible missing or wrong outcomes', () => {
assertFunctional(functional());
const mutations = [e => { delete e.ack; }, e => { e.ack.intent.intentId = 'other'; },
e => { e.appendBlockedBeforeStop.rows[0].blocking_pids = []; }, e => { e.prelude.individualDeltas = 999; },
e => { e.authorityAfter.rows[0].process_stopped.processPid = 99; }, e => { e.processesAfter.providerAbsent = false; },
e => { e.heldSamples[1].emittedCount = 3; }, e => { e.transcript.deltas[1].payload.text = '[000] '; },
e => { e.transcript.deltas[1].sourceEventId = 'id0'; e.transcript.deltas[1].sourceSeq = 1; },
e => { e.transcript.persistedAfterReload = false; }, e => { e.measurement.recording.trace = 'on'; }];
for (const mutate of mutations) { const e = functional(); mutate(e); assert.throws(() => assertFunctional(e)); }
});
const retained = (e = functional(), receipt = false) => {
const before = e.authorityBefore.rows[0], after = e.authorityAfter.rows[0];
Object.assign(after, { process_pid: before.process_pid, process_group_id: before.process_group_id, process_started_at: before.process_started_at });
if (!receipt) Object.assign(after, { process_stopped_seq: null, process_stopped: null, process_stopped_source_event_id: null,
process_lifecycle_seq: before.process_identity_seq, process_lifecycle_event_type: 'native.process_identity_recorded' });
e.authoritativeRetirementState = 'retained-identity-after-physical-exit'; return e;
};
test('retained original identity with NULL retirement receipt passes only with independent exit, own ACK and released lease', () => {
const e = retained();
assert.equal(assertAuthoritativeRetirement(e), 'retained-identity-after-physical-exit'); assertFunctional(e);
assert.equal(e.authorityAfter.rows[0].process_stopped, null);
});
test('cleared identity requires bound server retirement; a bound optional receipt also validates for retained identity', () => {
const cleared = functional(); assert.equal(assertAuthoritativeRetirement(cleared), 'cleared-identity-with-authoritative-retirement');
assertFunctional(cleared); assertFunctional(retained(functional(), true));
});
test('both states require valid captured authoritative original birth, including before JSON serialization', () => {
for (const factory of [functional, retained]) {
const valid = factory(), before = valid.authorityBefore.rows[0], after = valid.authorityAfter.rows[0];
before.process_started_at = new Date(before.process_started_at);
if (after.process_started_at !== null) after.process_started_at = new Date(after.process_started_at);
assertFunctional(valid); assertFunctional(JSON.parse(JSON.stringify(valid)));
for (const mutate of [e => {
const original = e.authorityBefore.rows[0], current = e.authorityAfter.rows[0];
original.process_started_at = null; original.process_identity.processStartedAt = null;
current.process_identity.processStartedAt = null; current.process_started_at = null;
if (current.process_stopped) current.process_stopped.processStartedAt = null;
}, e => { delete e.authorityBefore.rows[0].process_started_at; },
e => { e.authorityBefore.rows[0].process_started_at = 'not-a-birth'; e.authorityBefore.rows[0].process_identity.processStartedAt = 'not-a-birth'; }]) {
const invalid = factory(); mutate(invalid);
assert.throws(() => assertFunctional(invalid));
}
}
});
test('cleared identity rejects absent, unbound, partial or later-invalidated retirement evidence', () => {
const mutations = [
row => { row.process_stopped_seq = null; row.process_stopped = null; },
row => { row.process_stopped_seq = null; }, row => { row.process_stopped_source_event_id = 'provider-event'; },
row => { row.process_stopped.processPid = 99; }, row => { row.process_stopped.processGroupId = 99; },
row => { row.process_stopped.processStartedAt = '2026-11-04T00:00:01.000Z'; }, row => { delete row.process_stopped.processStartedAt; },
row => { row.process_stopped_seq = '1'; }, row => { row.process_started_at = '2026-11-04T00:00:00.000Z'; },
row => { row.process_lifecycle_seq = '21'; row.process_lifecycle_event_type = 'native.process_start_requested'; },
row => { row.process_identity_seq = '21'; }, row => { row.process_lifecycle_seq = '21'; },
row => { row.process_lifecycle_source_event_id = 'provider-event'; }];
for (const mutate of mutations) { const e = functional(); mutate(e.authorityAfter.rows[0]); assert.throws(() => assertAuthoritativeRetirement(e)); }
});
test('retained identity rejects changed PID, group, birth, server identity or later launch and verifies optional receipts', () => {
const mutations = [row => { row.process_pid = null; }, row => { row.process_group_id = null; },
row => { row.process_pid = 99; }, row => { row.process_group_id = 99; },
row => { row.process_started_at = '2026-11-04T00:00:01.000Z'; }, row => { row.process_started_at = null; },
row => { row.process_identity.processStartedAt = 'other'; }, row => { row.process_identity_seq = '2'; },
row => { row.process_lifecycle_seq = '2'; row.process_lifecycle_event_type = 'native.process_start_requested'; },
row => { row.process_identity_source_event_id = 'provider-event'; }, row => { delete row.process_stopped; }];
for (const mutate of mutations) { const e = retained(); mutate(e.authorityAfter.rows[0]); assert.throws(() => assertAuthoritativeRetirement(e)); }
const optional = retained(functional(), true); optional.authorityAfter.rows[0].process_stopped.processPid = 99;
assert.throws(() => assertAuthoritativeRetirement(optional));
});
test('neither retirement state bypasses physical process/group absence, own audited ACK or exact lease cleanup', () => {
const mutations = [e => { delete e.processesAfter.providerGroupAbsent; }, e => { e.processesAfter.groupAbsent = false; },
e => { e.processesAfter.runnerAbsent = false; }, e => { e.processesAfter.providerAbsent = false; },
e => { e.authorityAfter.rows[0].native_cancellation.intentId = 'other'; }, e => { e.ack.audits.pop(); },
e => { e.authorityAfter.rows[0].status = 'failed'; }, e => { e.lease.id = 'other'; },
e => { e.authorityAfter.rows[0].lease_id = 'other'; }, e => { e.lease.releasedAt = null; },
e => { e.authorityAfter.rows[0].lease_released_at = null; }, e => { e.authorityAfter.rows[0].lease_cleanup_status = 'failed'; },
e => { e.authorityAfter.rows[0].provider = 'remote'; }];
for (const factory of [functional, retained]) for (const mutate of mutations) {
const e = factory(); mutate(e); assert.throws(() => assertAuthoritativeRetirement(e));
}
const forged = retained(); forged.authoritativeRetirementState = 'cleared-identity-with-authoritative-retirement';
assert.throws(() => assertFunctional(forged));
});
test('shared decision is correlated when present and absent run-only decisions stay valid', () => {
const row = intentRow(); assertIntentAudit(row);
row.intent.decisionId = 'decision'; row.audits[1].details.decisionId = 'decision';
row.shared_decision = { id: 'decision', companyId: 'c', runId: 'r', issueId: 'i' }; assertIntentAudit(row);
row.shared_decision.runId = 'foreign'; assert.throws(() => assertIntentAudit(row));
});
const arm = (index, shift = 0, variant = 'b0') => {
const offset = index * 10000;
const evidence = JSON.parse(JSON.stringify(functional()), (key,value) => {
if (value === 'r') return `r${index}`; if (value === 'c') return `c${index}`; if (value === 'i') return `i${index}`;
return typeof value === 'number' && (key.endsWith('AtMs') || key === 'epochMs') ? value + offset : value;
});
evidence.variant = variant;
for (const key of ['ackObservedAtMs','absenceObservedAtMs','lastAckNotObservedAtMs','lastProcessesPresentAtMs','httpSettledAtMs'])
evidence.timing[key] += shift;
for (const sample of evidence.observerSamples) { sample.startedAtMs += shift; sample.completedAtMs += shift; }
return evidence;
};
const pairs = (shift = () => 0) => Array.from({ length: 5 }, (_, pair) => [
arm(2*pair + (pair % 2), shift(pair,0)), arm(2*pair + (pair % 2 ? 0 : 1), shift(pair,1))]);
test('noise artifact freezes actual suitable A/A limits with counts, spread, chronological drift and resolution labels', () => {
const noise = noiseEnvelope(pairs((pair,side) => side ? pair+1 : 0));
assert.equal(noise.suitable, true); assert.equal(noise.noiseFrozen, true); assertNoiseEnvelope(noise);
assert.deepEqual(noise.sampleCounts, { pairs: 5, arms: 10 }); assert.equal(noise.rawArms.length, 10);
for (const metric of METRICS) {
const row = noise.metrics[metric]; assert.equal(row.noiseMs, 5); assert.equal(row.sampleCount, 10);
assert.equal(row.precisionBudgetMs, Math.max(2*W.pollIntervalMs, 0.10*row.pooledMedianMs));
assert(row.maximumObserverBracketMs <= row.precisionBudgetMs);
}
assert.equal(noise.metrics.releaseAckToAckMs.resolutionLabel, '100 ms resolution floor');
assert.equal(noise.metrics.releaseAckToAckMs.medianDifferencePercent, null);
assert.equal(noise.metrics.releaseAckToAckMs.percentageSpeedClaimsAllowed, false);
});
test('every calibration and comparison arm must pass the complete functional oracle', () => {
const noise = noiseEnvelope(pairs());
const mutations = [e => { delete e.lock; }, e => { delete e.ack; }, e => { delete e.processesAfter; },
e => { delete e.lease; }, e => { delete e.transcript; }, e => Object.assign(e, evidence()),
e => { e.transcript.persistedAfterReload = false; }];
for (const mutate of mutations) {
const invalid = pairs(); mutate(invalid[4][1]);
assert.throws(() => noiseEnvelope(invalid)); assert.throws(() => comparePairs(invalid, noise));
}
const forged = pairs(); forged[0][0] = evidence();
assert.throws(() => noiseEnvelope(forged)); assert.throws(() => comparePairs(forged, noise));
});
test('paired jitter and chronological drift outside prospective budget retain all arms and cannot freeze noise', () => {
for (const data of [pairs((pair,side) => side ? 150 : 0), pairs(pair => pair * 60)]) {
const noise = noiseEnvelope(data);
assert.equal(noise.suitable, false); assert.equal(noise.noiseFrozen, false); assert.equal(noise.rawArms.length, 10);
assert(METRICS.every(metric => noise.metrics[metric].noiseMs === null));
assert.throws(() => assertNoiseEnvelope(noise)); assert.throws(() => comparePairs(pairs(), noise));
}
const drift = noiseEnvelope(pairs(pair => pair * 60));
assert.equal(drift.metrics.releaseAckToAckMs.maximumAbsolutePairedDifferenceMs, 0);
assert(drift.metrics.releaseAckToAckMs.absoluteChronologicalMedianDriftMs > drift.metrics.releaseAckToAckMs.precisionBudgetMs);
});
test('raw observer brackets retain actual query/check durations and reject missing or forged timing', () => {
const e = functional(), brackets = observerBrackets(e);
assert.equal(brackets.releaseAckToAckMs.widthMs, 5);
assert.equal(brackets.releaseAckToAckMs.negativeDurationMs, 1);
assert.equal(brackets.releaseAckToAbsenceMs.successfulDurationMs, 1);
e.observerSamples[0].durationMs = 0; assert.throws(() => observerBrackets(e));
delete e.observerSamples; assert.throws(() => assertFunctional(e));
});
test('a wide observer bracket invalidates A/A suitability and comparison despite stable latency medians', () => {
const data = pairs(() => 120), e = data[0][0];
e.observerSamples = e.observerSamples.filter(sample => sample.kind !== 'ack-query' || sample.value);
delete e.timing.lastAckNotObservedAtMs;
const unsuitable = noiseEnvelope(data);
assert.equal(unsuitable.metrics.releaseAckToAckMs.maximumAbsolutePairedDifferenceMs, 0);
assert.equal(unsuitable.suitable, false); assert.throws(() => assertNoiseEnvelope(unsuitable));
const noise = noiseEnvelope(pairs(() => 120));
const result = comparePairs(data, noise);
assert.equal(result.releaseAckToAckMs.pass, false);
assert.equal(result.releaseAckToAckMs.noiseMs, 0);
});
test('comparison uses unchanged frozen noise and budget; one paired regression or inflated artifact fails', () => {
const calibration = pairs((pair,side) => side ? pair+1 : 0), noise = noiseEnvelope(calibration);
assert(Object.values(comparePairs(calibration, noise)).every(row => row.pass));
const regression = pairs((pair,side) => side ? 6 : 0);
assert(Object.values(comparePairs(regression, noise)).every(row => !row.pass));
const forged = structuredClone(noise); forged.metrics.releaseAckToAckMs.noiseMs = 999;
assert.throws(() => comparePairs(pairs(), forged));
const inflated = structuredClone(noise); inflated.metrics.releaseAckToAckMs.precisionBudgetMs = 999;
assert.throws(() => comparePairs(pairs(), inflated));
assert.throws(() => noiseEnvelope(pairs().slice(1)));
});
test('optional shared cancellation accepts absent/requested B0 metadata and validates present state, time, attribution and run scope', () => {
const when = '2026-11-04T10:00:00.000Z';
const e = functional(); assertFunctional(e);
e.ack.shared_execution_cancellation = { state: 'requested', requestedAt: when };
e.ack.run_cancellation = { source: 'operator', expected: true, initiator: { type: 'user' }, reason: 'Operator Stop', recordedAt: when };
assertFunctional(e);
e.ack.shared_execution_cancellation = { state: 'acknowledged', acknowledgedAt: when }; assertFunctional(e);
for (const patch of [{ state: 'requested' }, { state: 'acknowledged', acknowledgedAt: 'invalid' },
{ state: 'other', requestedAt: when }]) {
const bad = structuredClone(e); bad.ack.shared_execution_cancellation = patch; assert.throws(() => assertFunctional(bad));
}
const wrong = structuredClone(e); wrong.ack.run_cancellation.source = 'provider'; assert.throws(() => assertFunctional(wrong));
const foreign = structuredClone(e); foreign.sharedSettlement = { companyId: 'c', runId: 'foreign', status: 'cancelled' };
assert.throws(() => assertFunctional(foreign));
});
test('calibration and comparison reject reused runs, overlapping arms and nonalternating chronology', () => {
const noise = noiseEnvelope(pairs());
for (const corrupt of [data => { data[4][1] = data[0][0]; },
data => { data[0][1].startedAtMs = data[0][0].finishedAtMs; },
data => { data[1].reverse(); }]) {
const invalid = pairs(); corrupt(invalid);
assert.throws(() => noiseEnvelope(invalid)); assert.throws(() => comparePairs(invalid, noise));
}
});
test('distinct provider group is valid, same-group topology is also valid, and both exact owned groups must be proved', () => {
const distinct = functional(); assert.notEqual(distinct.processesBefore.group, distinct.processesBefore.providerGroup); assertFunctional(distinct);
const shared = functional(); shared.processesBefore.provider.group = 10; shared.processesBefore.providerGroup = 10;
shared.processesAfter.providerGroup = 10;
for (const held of shared.heldSamples) { held.processes.provider.group = 10; held.processes.providerGroup = 10; }
assertFunctional(shared);
for (const mutate of [e => { delete e.processesBefore.providerGroup; }, e => { e.processesBefore.providerGroup = 99; },
e => { e.processesBefore.provider.group = 0; }, e => { delete e.processesAfter.providerGroupAbsent; },
e => { e.processesAfter.providerGroupAbsent = false; }, e => { e.processesAfter.providerGroup = 99; },
e => { e.authorityBefore.rows[0].process_group_id = 99; }, e => { e.processesBefore.runner.group = 99; },
e => { e.heldSamples[1].processes.provider.birth = 'different-birth'; }, e => { e.heldSamples[1].processes.runner.pid = 99; },
e => { e.heldSamples[1].processes.providerGroupAlive = false; }, e => { e.heldSamples[1].processes.runnerGroupAlive = false; },
e => { delete e.heldSamples[1].processes; }]) {
const e = functional(); mutate(e); assert.throws(() => assertFunctional(e));
}
});
test('cleanup eligibility follows supported active/scheduled and native retryable phases only', () => {
for (const status of ['queued','running','scheduled_retry']) assert(cleanupCancellationEligible({ status }));
assert(cleanupCancellationEligible({ status:'failed', runtimeMode:'native', nativePhase:'retryable_failure' }));
for (const run of [{ status:'failed', runtimeMode:'legacy', nativePhase:'retryable_failure' },
{ status:'failed', runtimeMode:'native', nativePhase:'terminal_failure' }, { status:'failed', runtimeMode:'native' },
{ status:'succeeded' }, { status:'cancelled' }, { status:'timed_out' }]) assert.equal(cleanupCancellationEligible(run), false);
});
test('owned cleanup records public cancellation before closing its clock, including retryable failure', async () => {
for (const before of [{ status:'running', runtimeMode:'native' }, { status:'failed', runtimeMode:'native', nativePhase:'retryable_failure' }]) {
const order=[],record={}; let open=true,run={ id:'r',companyId:'c',...before };
await cleanupOwnedRunBeforeClockClose({ companyId:'c',runId:'r',record,
readRun:async()=>{ assert(open);order.push('read');return run; },
cancelRun:async()=>{ assert(open);order.push('cancel');run={...run,status:'cancelled',resultJson:{nativeCancellation:{dispatchState:'acknowledged'}}};return {status:200,run}; },
closeClock:async()=>{ order.push('close');open=false; } });
assert.deepEqual(order,['read','cancel','read','close']); assert.equal(record.before.status,before.status);
assert.equal(record.cancel.httpStatus,200); assert.equal(record.after.status,'cancelled'); assert.equal(record.after.nativeCancellation.dispatchState,'acknowledged');
assert.equal('providerAbsent' in record,false,'Cancellation response is not physical Stop evidence');
}
});
test('cleanup does not cancel terminal/foreign runs and closes clock after a failed cancellation attempt', async () => {
for (const mode of ['terminal','foreign','cancel-error']) {
const order=[],record={};let cancelled=false;
const action=()=>cleanupOwnedRunBeforeClockClose({companyId:'c',runId:'r',record,
readRun:async()=>{order.push('read');return {id:'r',companyId:mode==='foreign'?'other':'c',status:mode==='terminal'?'succeeded':'running'};},
cancelRun:async()=>{cancelled=true;order.push('cancel');throw Error('controlled cancellation error');},
closeClock:async()=>{order.push('close');} });
if(mode==='terminal'){await action();assert.equal(record.cancel.attempted,false);assert.equal(cancelled,false);}
else {await assert.rejects(action());assert(record.error);assert.equal(cancelled,mode==='cancel-error');}
assert.equal(order.at(-1),'close');assert(record.clockClosedAtMs);
}
});
tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs
+22
View File
@@ -0,0 +1,22 @@
// Read-only validation of an independently extracted raw attachment; no app/DB/process execution.
import { readFile, writeFile } from 'node:fs/promises';
import { createHash } from 'node:crypto';
import { pathToFileURL } from 'node:url';
import { assertFunctional, PROTOCOL, WORKLOAD_SHA256, registration } from './native-ingestion-blocked-stop-protocol.mjs';
export async function validateFile(file) {
const bytes = await readFile(file), evidence = JSON.parse(bytes.toString('utf8'));
const metrics = assertFunctional(evidence);
return { protocol: PROTOCOL, workloadSha256: WORKLOAD_SHA256, sourceFile: file,
sourceSha256: createHash('sha256').update(bytes).digest('hex'), sourceRevision: evidence.sourceRevision,
runId: evidence.runId, metrics, functionalPass: true,
qualification: 'Functional gate only; owning Playwright, frozen preflight, cleanup and paired noise/comparison campaign remain separately required' };
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
if (process.argv[2] === '--registration') process.stdout.write(`${JSON.stringify(registration(), null, 2)}\n`);
else {
if (!process.argv[2]) throw new Error('Pass extracted raw companion evidence JSON, or --registration');
const result = await validateFile(process.argv[2]);
if (process.argv[3]) await writeFile(process.argv[3], `${JSON.stringify(result, null, 2)}\n`, { flag: 'wx' });
else process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
}
}
tests/e2e/fixtures/native-ingestion-blocked-stop.md
+257
View File
@@ -0,0 +1,257 @@
# Existing-history blocked Stop companion (prospective instrumentation v3)
Status: v3 local source and pure oracle calibration only. No v3 application, browser,
PostgreSQL, VM or quantitative arm has run. No runtime qualification or performance
claim follows from the pure tests. Execution needs lead release after independent
source review. This is separate from the immutable H5c 50-arm campaign.
## Preserved v1/v2 qualification failures and prospective corrections
The one v1 qualification at `200396d5` failed before the emission epoch because
the probe assumed the provider shared the Runner process group. Actual Runner
PID/group was 2284214; provider PID/group was 2284218. Separate groups are allowed.
No held-append, Stop timing or calibration data was collected. The failed raw
qualification remains retained and cannot supply a v2 timing arm or noise sample.
Normal API cleanup returned 200 and the port was released, but strict physical
quiet failed: both exact owned processes remained sleeping after the app closed.
The old finally block closed the clock before cancelling the run and did not
attempt cancellation for failed native retryable phases. Those observations do not establish a product
cause or successful physical termination from the API response.
This v2 source records independent provider PID birth and process group, proves
both groups alive during the hold and absent after Stop, and keeps the server
termination receipt correlated to the Runner group. Every held sample retains
both original process births. Shared-group topology remains valid when observed.
Cleanup releases its owned lock, attempts supported public cancellation while the
clock is still available, records before/HTTP response/after status and native
cancellation metadata, and then closes the clock. No manual process kill belongs
to a passed functional oracle; exact owned cleanup after a failed qualification
is separate negative evidence. New source review and runtime qualification are
required. Protocol name, workload, precision rule and all deadlines are unchanged.
The one v2 qualification at `6e5349cd` reached acknowledged native cancellation,
cancelled settlement and independently observed Runner/provider PID and group
absence. It then failed the probe's unconditional requirement for an immediate
`native.local_process_stopped` event: the row still retained original PID/group
2286265, its latest server identity sequence was 19, and the stop receipt was NULL.
Transcript, reload and the later lease gate were not reached. That requirement
exceeded the supported product contract. The failed trial and
its observed timings remain negative evidence, cannot supply a v3 timing arm or
noise sample, and are not reclassified as a qualified measurement.
V3 reads final authority after the existing lease-release poll in one existing
READ ONLY REPEATABLE READ snapshot, including the native cancellation intent,
lease cleanup and latest server-authored process lifecycle fence. It adds no
poll, wait budget or timing shift: ACK and physical-absence observations retain
their original boundaries. `assertAuthoritativeRetirement` records either
`retained-identity-after-physical-exit` or
`cleared-identity-with-authoritative-retirement`. Both require cancelled state,
the same own audited ACK, the original owned local lease released without failed
cleanup, and independently observed absence of both exact PIDs and both groups.
Retained mutable PID/group/birth and latest server identity must exactly match
the captured original identity. An absent retirement receipt is valid only in
that retained state. If a receipt is present, it must still bind to the original
PID/group/birth. Cleared mutable identity requires that bound server-authored
retirement receipt, later than the latest identity, with no later process start
or identity event. Partial, changed or relaunched identity is rejected.
The authoritative product basis is
`server/src/services/acknowledged-native-stop.ts`,
`acknowledgedNativeStopExecutionHasStopped`: after local lease cleanup, retained
PID/group use the host's ESRCH observations; cleared fields require
`hasNativeLocalProcessStop`. In `native-local-process-stop.ts`,
`recordNativeLocalProcessStop` records the original PID/group/birth before identity
clearing, while `hasNativeLocalProcessStop` rejects a later server start/identity.
`releaseSettledNativeRunProcess` separately gates retirement on coordinator,
provider and lease state. Native cancellation ACK persistence in
`native-session-executor.ts` does not itself retire process identity. The event is
authoritative identity-clearing evidence, not a universal immediate Stop receipt
or a substitute for physical process observation. This is a probe correction;
it includes no product change or attribution of a product defect.
## Registration and ownership
Explicit config: `tests/e2e/playwright-native-ingestion-blocked-stop.config.ts`.
Explicit source: `tests/e2e/native-event-ingestion-blocked-stop.journey.ts`.
Protocol: `blocked-stop-history-v1`; workload digest comes from the exact exported
`WORKLOAD` JSON in `native-ingestion-blocked-stop-protocol.mjs`.
The config inherits the unchanged H5c production-built app bootstrap, local
Codex command fixture, normal local environment, and shipped UI. It selects only
the companion source and fixes a 900-second test budget. There is one owned
control agent and one task created and assigned through visible normal controls.
Provider execution is a deterministic existing Codex leaf fixture, not a live
model or a real Cloud provider. This companion grants no Cloud Stop qualification.
No resolver override, HTTP/API change, persisted outcome seeding or app/product
code change is included. Authority/gate/paint code is a bounded copy from H5c
`e431ec63d41fe90c6fd68bd2fc2bb969feeca917`; the original nine measured files stay
unchanged. The new files have their own source identity and review.
## Fixed workload and functional gate
All variants emit 1,000 actual prelude deltas, independently read back through
normal event APIs before the common gate releases. They then emit 256 compact
markers at 100 ms, fitting the unchanged 4,000 character Rust text bound together
with the prelude. Readiness is bounded at 300 seconds, measured streaming at
120 seconds, terminal settlement at 180 seconds; original later 30-second ACK
and lease observations remain after settlement. Setup stays inside the owning
900-second test budget. Server bootstrap retains its existing separate limit.
After at least three visible markers and before Stop, a separately owned
PostgreSQL connection acquires `FOR UPDATE` on the exact company/run row. A
second connection reads `pg_stat_activity` and `pg_blocking_pids` and must see
an actual append query waiting on that holder. Its SELECT projection prefix is
checked against the B0/B1/B2 source signatures, distinguishing cancellation's
full-row SELECT. Raw SQL metadata is retained privately; no product hook is used.
The ordinary visible Stop button must cause the actual matching public POST.
For at least 1,000 ms after that observed request, repeated independent snapshots
must show pending HTTP, no committed native cancellation intent or audit, the
same runner/provider PID births alive in their exact group, and an open fixture
clock. The fixture closes that clock on `turn/interrupt`. Provider emissions must
continue across the hold. The hold is not subtracted from action-to-ACK latency.
A late sample/query/timer can overrun the minimum; actual boundaries stay visible.
Release records both COMMIT request and COMMIT response acknowledgement. Neither
is claimed to be the exact PostgreSQL commit instant. Independent observations
then require the cancellation intent and its own matching committed audit,
native acknowledged dispatch and its own matching acknowledgement audit,
normal cancelled settlement, successful Stop HTTP, independently absent
runner/group/provider/provider-group, and the exact owned local lease released
without failed cleanup. The coherent final snapshot must meet the retained or
cleared identity contract above; cleared fields require the bound server-authored
`native.local_process_stopped` retirement receipt. Transcript checks require complete compact payloads, unique source
identities, ordered canonical prefix, original emission clocks, zero accepted
results, and persistence after reload. The raw transport recovery policy stays
unchanged: raw fixture emissions after the canonical prefix are not represented
as universally database-durable.
The native receipt, shared `executionCancellation` metadata (including absence)
and public cancelled run are recorded separately. The native dispatch audit must
match the same intent and intent audit, `dispatched: true`, and nullable decision
ID. A non-null decision is independently correlated through the owned
`status_decisions` company/issue/run row. Run-only Stop may have no decision; a
missing optional decision or shared field is not silently invented. A shared ACK field is not synthesized from
row `recordedAt` or a provider clock. Source/runtime review must confirm which
shared receipt contract the serving revision exposes before releasing the probe.
No universal 30-second click-to-ACK gate is introduced.
Every actual ACK query and process check through its first success retains its
start, completion, duration and boolean observation. An uncertainty bracket runs
from the last negative query/check start (conservative for snapshot-read timing)
or release-COMMIT-ACK when no negative was seen, through first successful
completion. Both negative and successful operation durations stay explicit.
Timing includes action-to-observed-native-ACK, release-COMMIT-ACK-to-independently
observed absence, and release-COMMIT-ACK-to-observed-native-ACK. Both release
request-origin values also remain. Last unsuccessful and successful-query/check
bounds are retained with a 50-ms poll interval; query duration and scheduling add
uncertainty. Screenshots and reload happen after those timing observations.
Any failure, timeout, censored observation, query/clock fault or cleanup error
makes the arm ineligible. Partial raw Playwright results and the final attachment
remain even after failure. Cleanup first releases/rolls back only the owned lock,
closes both connections, settles only its owned run while its clock is available,
then retires the clock and restores owned experimental settings. Supported public
phases are queued/running/scheduled_retry and failed native retryable_failure;
other terminal phases are not presumed cancelable or physically stopped. This
cleanup subset follows `server/src/services/heartbeat/recovery.ts` active statuses
and `native-cancellation-request.ts` retryable eligibility, reached through the
normal board POST in `routes/agents.ts`. The server rechecks durable coordinator
eligibility and owns cancellation barriers; the probe does not grant authority.
The external operator must independently prove the
server, database, runner, provider and browser are gone before the next arm.
## Separate quantitative registration (50 short arms)
Generate the explicit registration without executing an app:
```sh
node tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs --registration
```
Order is B0 A/A five fresh alternating pairs, B1 A/A five fresh alternating pairs,
then B0→B1, B1→B2 and B0→B2 five alternating pairs each. Each pair is two fresh
owned invocations with matching data/load/payload/resource/cache conditions.
There are no repeats hidden inside an invocation. Keep the original main 50 arms
and their noise files untouched.
Before any companion data, freeze this exact suitability rule independently per
metric. The precision budget is `max(2 * 50 ms, 0.10 * pooled ten-arm A/A metric
median)`. Both the maximum absolute paired difference and the absolute median
drift between the first five and last five chronological arms must fit that
budget. Every raw ACK/absence observer bracket must also fit the corresponding
metric budget; action-to-ACK uses the ACK observer bracket. This is a
measurement repeatability rule, not a product Stop deadline.
Only suitable calibration freezes its observed maximum absolute paired
difference as noise. B0 comparisons use B0 A/A; B1→B2 uses B1 A/A. Every paired
candidate difference and their median must remain within that own frozen noise,
and all comparison observer brackets must fit the frozen baseline precision
budget. No unblocked Stop noise is reused. An unsuitable ten-arm calibration
retains every raw arm and produces no usable noise envelope: no silent
replacement, budget increase or inflated envelope follows from noisy data.
`noiseEnvelope(pairs)` takes five logical `[baseline, candidate]` raw-report
pairs whose chronological order alternates baseline-first then candidate-first.
It validates the full functional oracle for every arm, requires ten distinct
owned runs/companies with nonoverlapping arm times, and returns the structured
`paperclip.native-ingestion.blocked-stop-noise.v1` artifact. This contains
sample counts, all raw arms and report digests, logical pair bindings, actual
absolute/percentage median differences, spread, chronological drift, budgets,
query/check uncertainty brackets, suitability, and nullable noise values.
Percentage fields are null when the 100-ms precision floor determines the
budget. Those metrics are labeled **100 ms resolution floor** and cannot support
subfloor, p99 or percentage speed claims. Ten observed arms do not establish p99.
`assertNoiseEnvelope(artifact)` revalidates all retained functional proofs and
recomputes suitability and numeric limits; unsuitable or inflated artifacts
throw. `comparePairs(pairs, artifact)` validates every comparison arm's complete
functional proof and returns per-metric `.pass` results using only that
baseline's unchanged frozen noise and budget. It rejects reused runs,
nonalternating/overlapping arms, missing lock/ACK/process/lease/transcript proof,
and forged `complete` flags. Preserve all failures and censored arms outside
these successful artifacts as well. A single functional probe is not
quantitative nonregression evidence.
## Reproduction and execution gates
Credential-free local checks, with Node 24:
```sh
node --test tests/e2e/fixtures/native-ingestion-blocked-stop-protocol.test.mjs
node --check tests/e2e/native-event-ingestion-blocked-stop.journey.ts
node --check tests/e2e/playwright-native-ingestion-blocked-stop.config.ts
node tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs --registration
```
Before runtime, the lead must review/freeze the companion source, workload digest,
registration, validator and companion-only release descriptor. Keep the reviewed
H5c private runner `6b6376b01de6cdc15ca05343a0bdd5c00424f1d6c546b9b4d61cd76716b253c4`
unchanged. A companion orchestrator should import its checked exported
`validateVariantBuild`, `validateActualInventory`, `assertNoPreferredUiShadow`,
`physicalQuietGuard`, `sealArmArtifacts` and `verifyRawArtifactSeals`; it must not
invoke or alter the original stage plan. Bind each arm to exact source/build/UI,
qualified receipt, full inventory manifest, Node/dependency/native pins, unchanged
frozen fixture hashes, companion hashes, recording-off flags and a clean source.
Use original B0 `bac53ec3`, B1 `07061a60`, unoptimized B2 `ca7cba17` only where
explicitly released; a future optimized B2 needs its own qualified descriptor.
The existing build receipt `a1259ba7…`, full inventory `d3766fc4…` and native binary
`abb65bdc…` are provenance, not substitutes for current artifact validation.
Only the performance owner may stage/execute on the sole benchmark VM, after
lead release. First do discovery, narrow static typecheck, and one smallest
functional qualification with raw Playwright, server/banner, PostgreSQL blocker,
process/clock and external cleanup evidence. Validate its extracted attachment:
```sh
node tests/e2e/fixtures/native-ingestion-blocked-stop-validate.mjs RAW.json RECEIPT.json
```
Do not start the 50-arm registration until the source and runtime probe qualify.
Capture quiet before/after-arm receipts, owning Playwright exit/status/duration,
raw API/SQL/process evidence, immutable report seals, and all cleanup outcomes.
The source validator is an additional functional oracle; it cannot establish the
owning Playwright pass, infrastructure cleanup, build authenticity or performance
on its own. Freeze separate B0/B1 A/A noise before nonregression comparisons.
tests/e2e/fixtures/native-ingestion-h5.md
+162
View File
@@ -305,3 +305,165 @@ queue/backlog, delay and memory with identical capture hooks. Aborted profiles
remain diagnostic-only and cannot establish complete CPU/event costs. No CPU
requirement or profiling is added to this timing comparator. Local results do
not establish deployed improvement.
## Prospective H5d oracle instrumentation and verified-completion gate
H5d adds oracle instrumentation and a prospective completion gate from frozen H5c
`e431ec63d41fe90c6fd68bd2fc2bb969feeca917`. It keeps the exact H5/H4e mode,
protocol, registration, workload and deadline constants. The new driver commit
and changed nine-file measurement digest distinguish the experiment. No product
optimization, new buffering, setup/control flow, payload bound, provider fixture,
transport, public API, P0 ordering or raw-ACK durability policy changes here.
All fifty main arms, including fresh B0/B1 A/A, must be rerun on the qualified new
source; old noise or the failed H5c attempt cannot qualify H5d.
The retained H5c FULL B0 attempt reached the unchanged 1,200,000-ms observation
ceiling while fetching/validating independent durable events. Root's retained
raw evidence records all 100 runs succeeded within 75.926 seconds of the same
180-second settlement window, all 100 clock uploads ended with 51,200 emissions
and no clock errors, and only roughly 30 of 100 per-run durable reads completed
before timeout. Foreground completion remained 283.391 seconds, with only
66/512 markers at the original streaming deadline. This owning attempt failed;
complete in-journey transcript/results verification and later cases were not
finished. A separate stopped read-only supplement cannot turn it into an arm
pass. Product improvement is not established by that failure or this correction.
Frozen H5c performed five Playwright assertions for every measured normalized
delta and one Playwright deep-equality assertion for its original clock join.
It also asserted each prelude payload and each visible clock through Playwright.
H5d uses the same actual paginated API rows and provider clock maps, then validates
those retained observations through Node assertions. Only verified per-run
summaries enter the existing durable evidence array. The active failure record
also identifies the current run, pagination cursor and returned row count.
| Frozen check | H5d equivalent |
| --- | --- |
| Exact complete compact payload for every measured delta | `assertDurableRunEvents` checks the same full text, rejecting coalescing or added bytes |
| Exact run, truthy source-instance/event IDs, positive safe integer sequence in H5 | Same assertions under the unchanged `requireNativeIdentity: h5` condition |
| Original nonce, marker code and index for every durable event | Same tuple deep equality against the original provider clock map |
| Unique normalized identity tuple and complete ordered indices | Same set count and exact ordered index array; Stop permits only its canonical prefix |
| Every visible Stop marker is durable; normal result one / Stop result zero | Same prefix membership and exact accepted-result count |
| Complete normal source-clock count and closed uploads | Existing clock-readiness/closure polls remain; Node oracle also checks the normal count |
| Prelude exact dots/final readiness text, count and unique identity tuples | `assertPreludeEvents` checks the same filtered prelude rows before epoch release |
| Every visible marker has an original clock; complete normal unique visible count | `assertVisibleMarkerClocks` checks the same observations and preserves the original two-frame timestamp |
The same `/events?afterSeq=…&limit=1000` pages, advancing cursor checks and final
empty-page read remain. The 30-second per-run provider-clock availability poll
is unchanged and still occurs before the Node assertion. The 120/180/900-second
original gates, 600/180/1,200-second observation ceilings, original late Stop ACK
and lease polls, and outer resource watchdog remain unchanged. No per-event
check is dropped to make the invocation finish.
The frozen comparator9e01 validated `h5.independentDrain` correctness but omitted
its elapsed duration from quantitative gates. A synthetic RED counterexample
actually passed that comparator with faster browser/terminal delivery while full
verified completion regressed from180,010 to200,000 ms. The prospective comparator
adds `h5.independentDrain.elapsedFromEpochMs` to every non-Stop H5 case's existing
measurement, own A/A noise and nonregression loops. For loaded burst512, final B2
must also improve that duration beyond its own noise in the median and at least
four of five pairs, alongside the existing browser p95 and terminal-duration
gates. Three improving pairs do not pass. Cursor nonregression includes the same
verified-completion metric. All previous gates and timing limits remain.
Reports also derive measured normalized deltas per second from equal matched
counts divided by that independently verified completion upper bound. For the
loaded burst, N is51,200 complete measured deltas, with all100 accepted results
independently checked. The duration includes oracle, reload, screenshot, normal
API and lease verification overhead; it is not the exact last database-commit
instant. This conservative observed completion rate is descriptive: the existing
noise gate applies to duration, and no separate rate-noise threshold is invented.
Equal counts make a shorter fully verified duration monotonically higher rate.
Missing/tampered completion proof still fails the existing functional oracle.
No accepted A/A exists. The following finite browser repeatability rule is
chosen prospectively; source freezing still requires the combined root/peer review.
### Prospective main-browser A/A suitability (H5 only)
Before any B0 or B1 calibration data, choose exactly five alternating pairs and
apply these rules independently to every declared case/metric after all existing
functional, owning-report and cleanup checks. With pair durations a,b:
- Every paired difference must satisfy `abs(a-b) <= max(0.20*a, 2 ms)`.
- Sort the ten actual owning invocations by validated start. First-five versus
last-five median drift must satisfy `abs(lastMedian-firstMedian) <=
max(0.10*firstMedian, 2 ms)`.
- The two five-sample role medians must also satisfy `abs(candidateMedian-
baselineMedian) <= max(0.10*baselineMedian, 2 ms)`.
Require parseable owner start/duration, nonoverlap, registered baseline-first then
candidate-first alternation, distinct owning reports, companies and run sets.
A comparison must follow all calibration owners and reuse none of their evidence
or owned data. Synthetic oracle fixtures represent this chronology and freshness
explicitly; they do not establish that an application or provider ran.
Retain all raw samples and owner chronology. Freeze the actual maximum paired
absolute difference AND actual maximum paired relative difference. The 2-ms
floor applies only to suitability and never replaces either observed envelope.
Comparisons recompute these envelopes and suitability diagnostics from retained
samples and reject altered envelopes, rules, flags or timestamps. Unsuitable A/A
exits unsuccessfully after writing its complete diagnostic report; retain all ten
arms and diagnose before a newly reviewed prospective observation. Do not inflate
caps or silently replace an arm.
If the pooled ten-value median is at most20 ms, label that metric
representation-limited, suppress percentage-improvement claims and use its actual
frozen absolute noise for nonregression: `median(candidate-baseline) <= observed
absolute noise`. In particular, zero observed noise permits no positive regression
even when the suitability floor was2 ms. Longer metrics retain the existing
relative-noise nonregression rule. Loaded improvement still requires its existing
relative-noise median and at least4/5 individual wins; each counted win and the
median absolute improvement must additionally exceed2 ms before a numerical gain
is claimed. No sub-2ms, p99 or percentage gain is claimed for representation-limited
metrics. All original hard product/control deadlines remain unchanged.
The floor describes integer-millisecond timestamp contrast representation only.
It is not a frame period, paint guarantee, clock-accuracy bound or a bound on the
settlement/Stop observer. The unchanged Playwright polls can wait1000 ms after a
callback and incur API/scheduler time as well; no blanket1–2second tolerance is
introduced. The blocked-Stop companion keeps its separate frozen100-ms policy.
The default H4e comparator and journey semantics remain unchanged.
### Deferred diagnostic design: Playwright assertion cost only
This diagnostic has **not run**. After root review and an explicit quiet-window
release, create an owned temporary standalone Playwright test/config outside the
measured source. Do not inherit a config with `webServer`. Request no `page`,
`browser` or `request` fixture; use one Node worker, no retries, trace/video off,
and a bounded 300-second diagnostic timeout. It must start no app, database,
provider or browser. Record the temporary source/config, Node and Playwright
versions, source hashes, owning test status, raw durations and all failures.
Generate 1,000 deterministic synthetic normalized rows and a matching clock map
inside that test process. They are CPU-oracle inputs only: no provider output,
application event or persisted outcome is seeded. Use the same six checks for
both branches on the same objects:
```ts
// Planned scratch test functions, not an executed measurement.
const checkPw = (event, original, index) => {
expect(event.payload.text).toBe(`[q7${index.toString(36)}] `);
expect(event.runId).toBe("diagnostic-run");
expect(event.sourceInstanceId).toBeTruthy();
expect(event.sourceEventId).toBeTruthy();
expect(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0).toBe(true);
expect({ nonce: original.nonce, markerCode: original.markerCode, index: original.index })
.toEqual({ nonce: "diagnostic-source", markerCode: "q7", index });
};
const checkNode = (event, original, index) => {
assert.equal(event.payload.text, `[q7${index.toString(36)}] `);
assert.equal(event.runId, "diagnostic-run");
assert.ok(event.sourceInstanceId);
assert.ok(event.sourceEventId);
assert.ok(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0);
assert.deepEqual({ nonce: original.nonce, markerCode: original.markerCode, index: original.index },
{ nonce: "diagnostic-source", markerCode: "q7", index });
};
```
Warm each branch once on 100 inputs outside timing. Then record five alternating
pairs of all 1,000 inputs using monotonic `performance.now()` boundaries, retaining
each elapsed time and six-check count. Do not disable normal Playwright assertion
steps in its branch. Compare total completion duration and spread; keep diagnostic
profiles separate. This tests assertion instrumentation overhead, not application
ingestion throughput, browser responsiveness or acceptance. Full H5d qualification
and the fresh frozen fifty-arm protocol remain required regardless of its result.
tests/e2e/fixtures/native-ingestion-h5.mjs
+73
View File
@@ -334,6 +334,79 @@ export function assertRecordedFailures(proof, item, companyId) {
recordedErrorClasses: Object.fromEntries([...classes].sort(([a], [b]) => a.localeCompare(b)).map(([code, ids]) => [code, ids.size])) };
}
// H5d only changes post-observation oracle instrumentation. These functions
// check the same retained API rows and original provider clocks without making
// a Playwright step for every normalized delta. They never drive the SUT.
export function collectDurableRunEvents(rows, markerCode) {
const deltas = []; let resultAcceptedEvents = 0;
for (const row of rows) {
const event = row.payload?.prpEvent;
if (row.eventType === 'run.result.accepted' || event?.eventType === 'run.result.accepted') resultAcceptedEvents++;
if (event?.eventType !== 'item.delta') continue;
for (const match of String(event.payload?.text ?? '').matchAll(new RegExp(`\\[${markerCode}([a-z0-9]{1,3})\\]`, 'g'))) {
const index = Number.parseInt(match[1], 36);
deltas.push({ index, identity: `${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`, event });
}
}
return { deltas, resultAcceptedEvents };
}
export function assertDurableRunEvents({ collected, runId, token, markerCode, events, stop, sourceClocks, visible,
requireNativeIdentity }) {
const { deltas, resultAcceptedEvents } = collected;
for (const [ordinal,delta] of deltas.entries()) {
const { index, event } = delta, context = `Durable run ${runId} source ${token} delta ${ordinal} marker ${markerCode}:${index}`;
assert.equal(event.payload.text, `[${markerCode}${index.toString(36)}] `, `${context}: complete normalized payload`);
if (requireNativeIdentity) {
assert.equal(event.runId, runId, `${context}: exact run identity`);
assert.ok(event.sourceInstanceId, `${context}: nonempty source instance identity`);
assert.ok(event.sourceEventId, `${context}: nonempty source event identity`);
assert.ok(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0, `${context}: positive integer source sequence`);
}
const original = sourceClocks.get(index);
assert.ok(original, `${context}: missing original provider clock`);
assert.deepEqual({ nonce: original.nonce, markerCode: original.markerCode, index: original.index },
{ nonce: token, markerCode, index }, `${context}: exact original emission clock join`);
}
const identities = new Set(deltas.map(delta => delta.identity)), indices = deltas.map(delta => delta.index);
assert.equal(identities.size, deltas.length, `Durable run ${runId}: duplicate normalized source identity tuple`);
assert.deepEqual(indices, Array.from({ length: stop ? deltas.length : events }, (_, index) => index),
`Durable run ${runId}: complete ordered ${stop ? 'canonical Stop prefix' : 'measured events'}`);
if (stop) {
const present = new Set(indices);
for (const item of visible) assert.ok(present.has(item.index),
`Durable run ${runId}: visible Stop marker ${item.index} must remain durable`);
} else assert.equal(sourceClocks.size, events, `Durable run ${runId}: complete source clock count`);
assert.equal(resultAcceptedEvents, stop ? 0 : 1, `Durable run ${runId}: exact accepted result count`);
return { runId, token, markerCode, individualDeltas: deltas.length, uniqueSourceIdentities: identities.size,
matchedSourceClocks: deltas.length, ordered: true, resultAcceptedEvents };
}
export function assertPreludeEvents({ rows, runId, token, expectedEvents }) {
const identities = new Set(); let count = 0;
for (const row of rows) {
const event = row.payload?.prpEvent;
if (event?.eventType !== 'item.delta' || !String(event.itemId ?? '').startsWith('prelude-')) continue;
assert.equal(event.payload?.text, count === expectedEvents - 1 ? ` PRELUDE_${token}_READY ` : '.',
`Prelude run ${runId} delta ${count}: complete normalized payload`);
count++; identities.add(`${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`);
}
assert.equal(count, expectedEvents, `Prelude run ${runId}: exact individually persisted delta count`);
assert.equal(identities.size, count, `Prelude run ${runId}: unique source identity tuples`);
return { individualDeltas: count, uniqueSourceIdentities: identities.size };
}
export function assertVisibleMarkerClocks({ seenMarkers, sourceClocks, markerCode, expectedEvents, stop }) {
const visible = seenMarkers.map(marker => {
const original = sourceClocks.get(marker.index);
assert.ok(original, `Missing original provider clock for visible ${marker.markerCode}:${marker.index}`);
return { ...original, ...marker }; // Retain the original two-frame paint timestamp.
});
if (!stop) assert.equal(new Set(visible.map(item => item.index)).size, expectedEvents,
`Visible marker ${markerCode}: complete unique foreground count`);
return visible;
}
export function assessH5Arm(report, owner) {
assert.equal(report.protocolVersion, H5_PROTOCOL);
assert.equal(report.observationMode, H5_MODE);
tests/e2e/native-event-ingestion-blocked-stop.journey.ts
+318
View File
@@ -0,0 +1,318 @@
import { randomUUID, createHash } from "node:crypto";
import { execFileSync } from "node:child_process";
import { readFile } from "node:fs/promises";
import path from "node:path";
import { expect, test, type Request } from "@playwright/test";
import { json } from "./agent-chat.shared";
import { runtimeAuthority, processAbsent } from "./fixtures/native-ingestion-blocked-stop-authority";
import { openClock } from "./fixtures/native-ingestion-blocked-stop-clock";
import { ownedDatabase, cancellationSnapshot, blockers } from "./fixtures/native-ingestion-blocked-stop-db";
import { PROTOCOL, WORKLOAD as W, WORKLOAD_SHA256, assertHeldSample, assertIntentAudit, isAppendLockQuery, metrics, assertFunctional, assertAuthoritativeRetirement, cleanupOwnedRunBeforeClockClose }
from "./fixtures/native-ingestion-blocked-stop-protocol.mjs";
import { observeRunSettlement } from "./fixtures/native-ingestion-h5.mjs";
const delay = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));
const birth = (pid: number) => {
const value = execFileSync("ps", ["-p", String(pid), "-o", "pid=,pgid=,lstart="], { encoding: "utf8" }).trim();
const match = value.match(/^(\d+)\s+(\d+)\s+(.+)$/);
if (!match || Number(match[1]) !== pid) throw new Error("Missing exact owned process birth identity");
return { pid, group: Number(match[2]), birth: match[3] };
};
test("existing-history Stop preserves its durable barrier behind a real blocked append", async ({ page, request }, info) => {
const appRoot = path.resolve(process.env.PAPERCLIP_INGESTION_APP_ROOT ?? "");
expect(process.env.PAPERCLIP_INGESTION_SERVER_ARTIFACT).toBe("compiled");
expect(process.env.PAPERCLIP_INGESTION_COMPANION_PROTOCOL).toBe(PROTOCOL);
const report: any = { protocol: PROTOCOL, workload: W, workloadSha256: WORKLOAD_SHA256, complete: false,
variant: process.env.PAPERCLIP_INGESTION_VARIANT, startedAtMs: Date.now(), timing: {}, heldSamples: [], observerSamples: [],
observedClock: "Independent observer wall clock; PostgreSQL row recordedAt is not a commit timestamp",
measurement: { pollIntervalMs: W.pollIntervalMs, releaseOrigin: "Owned COMMIT response acknowledgement",
releaseCommitBracket: "releaseRequestedAtMs..releaseCommitAckAtMs; exact server commit instant is unknown",
absenceAndAckUncertainty: "Last unsuccessful observation through successful query/process observation; polling plus query duration",
actionToAckIncludesForcedWait: true, noUniversalClickAckDeadline: true,
recording: { trace: info.project.use.trace, video: info.project.use.video }, quantitativeAcceptance: "Requires separately frozen companion A/A and five paired arms per comparison" } };
const clock = await openClock(page);
const company = await json(await request.post("/api/companies", { data: { name: `Blocked Stop ${Date.now()}` } }));
report.companyId = company.id;
const original = await json(await request.get("/api/instance/settings/experimental"));
let holder: any, observer: any, held = false, companyRunId: string | undefined;
let stopRequest: Request | undefined, stopSettled = false;
const nonce = randomUUID().replaceAll("-", ""), key = randomUUID().replaceAll("-", ""), code = clock.nextMarkerCode();
const stopFinished = (req: Request) => {
if (req === stopRequest) { stopSettled = true; report.timing.httpSettledAtMs = Date.now(); }
};
page.on("requestfinished", stopFinished);
page.on("requestfailed", stopFinished);
try {
const files = ["native-event-ingestion-blocked-stop.journey.ts", "playwright-native-ingestion-blocked-stop.config.ts",
"fixtures/native-ingestion-blocked-stop-authority.ts", "fixtures/native-ingestion-blocked-stop-clock.ts",
"fixtures/native-ingestion-blocked-stop-db.ts", "fixtures/native-ingestion-blocked-stop-protocol.mjs",
"native-event-ingestion.journey.ts", "playwright-native-ingestion.config.ts", "fixtures/in-feed-codex.mjs",
"fixtures/native-ingestion-bin/codex", "fixtures/composer-stop-codex-loader.mjs", "fixtures/native-ingestion-bootstrap.mjs",
"fixtures/native-ingestion-remote-server.mjs", "fixtures/native-ingestion-protocol-qualification.mjs", "fixtures/native-ingestion-h5.mjs"];
report.sourceHashes = Object.fromEntries(await Promise.all(files.map(async file => [file,
createHash("sha256").update(await readFile(path.join(import.meta.dirname, file))).digest("hex")])));
const harnessRoot = path.resolve(import.meta.dirname, "../..");
report.harnessProvenance = {
sourceRevision: execFileSync("git", ["rev-parse", "HEAD"], { cwd: harnessRoot, encoding: "utf8" }).trim(),
sourceStatus: execFileSync("git", ["status", "--porcelain", "--", "tests/e2e"], { cwd: harnessRoot, encoding: "utf8" }).trim(),
driverLockSha256: createHash("sha256").update(await readFile(path.join(harnessRoot, "pnpm-lock.yaml"))).digest("hex"),
inheritedH5cRevision: "e431ec63d41fe90c6fd68bd2fc2bb969feeca917",
};
report.sourceRevision = execFileSync("git", ["rev-parse", "HEAD"], { cwd: appRoot, encoding: "utf8" }).trim();
report.buildInfo = JSON.parse(await readFile(path.join(appRoot, "server/dist/build-info.json"), "utf8"));
expect(report.buildInfo.commit).toBe(report.sourceRevision);
report.serverEntrySha256 = createHash("sha256").update(await readFile(path.join(appRoot, "server/dist/index.js"))).digest("hex");
report.health = await json(await request.get("/api/health"));
await json(await request.patch("/api/instance/settings/experimental", { data: { enableNativeRunner: true, enableWorktreeRunExecution: true } }));
await json(await request.post(`/api/companies/${company.id}/user-secret-definitions`, { data: { key: "OPENAI_API_KEY", name: "Fixture credential" } }));
await json(await request.post(`/api/companies/${company.id}/me/user-secrets`, { data: { definitionKey: "OPENAI_API_KEY", value: "not-a-real-model-key" } }));
const environments = await json(await request.get(`/api/companies/${company.id}/environments?driver=local`));
expect(environments).toHaveLength(1);
const environment = environments[0]; expect(environment.driver).toBe("local");
const agent = await json(await request.post(`/api/companies/${company.id}/agents`, { timeout: 120_000, data: {
name: "Blocked Stop local control", role: "engineer", adapterType: "paperclip_runner", defaultEnvironmentId: environment.id,
adapterConfig: { provider: "codex", model: "gpt-5.1-codex-mini", env: {
OPENAI_API_KEY: { type: "user_secret_ref", key: "OPENAI_API_KEY", version: "latest", required: true } } },
runtimeConfig: { heartbeat: { enabled: false, wakeOnDemand: true, maxConcurrentRuns: 50 } },
} }));
await expect.poll(async () => (await json(await request.get(`/api/agents/${agent.id}`))).lifecycleState,
{ timeout: 60_000 }).toBe("ready");
report.setup = { environmentId: environment.id, agentId: agent.id,
selection: "Normal local environment and control agent; existing deterministic Codex leaf command; no resolver override" };
const text = ".".repeat(W.prelude - 1) + ` PRELUDE_${nonce}_READY `
+ Array.from({ length: W.events }, (_, index) => `[${code}${index.toString(36)}] `).join("");
expect([...text].length).toBeLessThan(4000);
const state = { events: W.events, intervalMs: W.intervalMs, streams: new Map(), errors: [] } as any;
state.streams.set(nonce, { nonce, markerCode: code, opened: false, hello: false, ended: false, aborted: false,
partialBytes: 0, records: new Map(), expectedLauncher: "host-local" });
clock.clocks.set(key, state); clock.pending.set(key, new Map());
const prompt = `Measure native streaming for ${nonce.slice(0, 8)}.\n\nINGESTION_PERF nonce=${nonce} events=${W.events} intervalMs=${W.intervalMs} prelude=${W.prelude} markerCode=${code} gate=${clock.base}${key}`;
const before = new Set((await json(await request.get(`/api/companies/${company.id}/issues`))).map((row: any) => row.id));
await page.goto(`/${company.issuePrefix}/agents/${agent.id}/overview`);
await page.getByRole("link", { name: "Tasks", exact: true }).click();
await page.getByRole("button", { name: "New Task", exact: true }).last().click();
const dialog = page.getByRole("dialog");
await dialog.getByRole("textbox", { name: "editable markdown" }).fill(prompt);
await dialog.getByRole("button", { name: "Select assignee", exact: true }).click();
await page.getByRole("listbox", { name: "Assignees", exact: true }).getByRole("option")
.filter({ has: page.getByText(agent.name, { exact: true }) }).click();
await expect(dialog.getByRole("button", { name: "Select assignee" })).toContainText(agent.name);
report.setup.taskCreateActionAtMs = Date.now();
await dialog.getByRole("button", { name: "Create task", exact: true }).click();
let issue: any;
await expect.poll(async () => {
issue = (await json(await request.get(`/api/companies/${company.id}/issues`)))
.find((row: any) => !before.has(row.id) && row.description?.includes(nonce)); return Boolean(issue);
}).toBe(true);
report.issueId = issue.id;
await page.goto(`/${company.issuePrefix}/issues/${issue.identifier}`);
await expect.poll(() => clock.pending.get(key)!.has(nonce) && state.streams.get(nonce).hello,
{ timeout: W.readinessMs, intervals: [W.pollIntervalMs] }).toBe(true);
await expect(page.getByText(`PRELUDE_${nonce}_READY`, { exact: false }).first()).toBeVisible({ timeout: W.streamMs });
let run: any;
await expect.poll(async () => {
run = (await json(await request.get(`/api/companies/${company.id}/heartbeat-runs?limit=1000`)))
.find((row: any) => (row.issueId ?? row.nativeIssueId ?? row.contextSnapshot?.issueId ?? row.contextSnapshot?.taskId) === issue.id);
return Boolean(run);
}).toBe(true);
companyRunId = run.id; report.runId = run.id;
const readEvents = async () => {
const result: any[] = []; let afterSeq = 0;
for (;;) {
const rows = await json(await request.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=1000`));
if (!rows.length) return result;
result.push(...rows);
const next = Math.max(...rows.map((row: any) => row.seq)); expect(next).toBeGreaterThan(afterSeq); afterSeq = next;
}
};
const prelude = (await readEvents()).map(row => row.payload?.prpEvent)
.filter(event => event?.eventType === "item.delta" && String(event.itemId).startsWith("prelude-"));
expect(prelude).toHaveLength(W.prelude);
prelude.forEach((event, index) => {
expect(event.payload.text).toBe(index === W.prelude - 1 ? ` PRELUDE_${nonce}_READY ` : ".");
expect(event.sourceInstanceId).toBeTruthy(); expect(event.sourceEventId).toBeTruthy();
expect(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0).toBe(true);
});
expect(new Set(prelude.map(event => `${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`)).size).toBe(W.prelude);
report.prelude = { individualDeltas: prelude.length, uniqueIdentities: prelude.length, verifiedBeforeEpochAtMs: Date.now(),
method: "Normal event API read of actual emitted and committed deltas; no outcome prepopulation" };
const authority = await runtimeAuthority({ appRoot, companyId: company.id, environmentId: environment.id,
issueIds: [issue.id], transport: "local" });
const row = authority.rows[0], stream = state.streams.get(nonce);
expect(row.run_id).toBe(run.id);
expect(row.process_identity.processPid).toBe(row.process_pid);
expect(row.process_identity.processGroupId).toBe(row.process_group_id);
const provider = birth(stream.providerPid);
const processes = { runner: birth(row.process_pid), provider, group: row.process_group_id, providerGroup: provider.group };
expect(processes.runner.group).toBe(processes.group);
expect(Number.isSafeInteger(processes.providerGroup) && processes.providerGroup > 1).toBe(true);
const alive = () => !processAbsent(processes.runner.pid) && !processAbsent(-processes.group)
&& !processAbsent(processes.provider.pid) && !processAbsent(-processes.providerGroup);
expect(alive()).toBe(true); report.authorityBefore = authority; report.processesBefore = processes;
state.epochMs = Date.now() + 1000; report.epochMs = state.epochMs;
clock.pending.get(key)!.get(nonce)!.writeHead(200, { "Content-Type": "application/json" }).end(JSON.stringify({ epochMs: state.epochMs }));
const visible = () => page.evaluate(markerCode => (window as any).nativeIngestionVisible.filter((item: any) => item.markerCode === markerCode), code);
await expect.poll(async () => (await visible()).length, { timeout: W.streamMs }).toBeGreaterThanOrEqual(W.visibleBeforeLock);
report.visibleBeforeLock = await visible();
holder = await ownedDatabase(appRoot, "blocked_stop_owned_lock");
observer = await ownedDatabase(appRoot, "blocked_stop_independent_observer");
await holder.unsafe("BEGIN"); held = true;
const owned = await holder`select id, company_id, pg_backend_pid() as holder_pid from heartbeat_runs
where company_id = ${company.id} and id = ${run.id} for update`;
expect(owned).toHaveLength(1); const holderPid = owned[0].holder_pid;
report.lock = { ...owned[0], acquiredAtMs: Date.now(), scope: "One owned company/run row; no persisted mutation" };
await expect.poll(async () => {
const rows = await blockers(observer, holderPid);
report.appendBlockedBeforeStop = { observedAtMs: Date.now(), rows };
return rows.some((candidate: any) => isAppendLockQuery(candidate.query));
}, { timeout: 10_000, intervals: [W.pollIntervalMs] }).toBe(true);
const button = page.getByRole("button", { name: "Stop", exact: true }); await expect(button).toBeVisible();
const stop = page.waitForRequest(req => {
const matched = req.method() === "POST" && new URL(req.url()).pathname === `/api/heartbeat-runs/${run.id}/cancel`;
if (matched) report.timing.requestAtMs = Date.now();
return matched;
});
report.timing.actionAtMs = Date.now();
const click = button.click();
stopRequest = await stop; await click;
const emissionsAtRequest = stream.records.size;
do {
const snapshot = await cancellationSnapshot(observer, company.id, run.id);
const sample = { ...snapshot, httpPending: !stopSettled, auditIds: snapshot.audits.map((audit: any) => audit.id),
processesAlive: alive(), processes: { runner: birth(processes.runner.pid), provider: birth(processes.provider.pid),
group: processes.group, providerGroup: processes.providerGroup,
runnerGroupAlive: !processAbsent(-processes.group), providerGroupAlive: !processAbsent(-processes.providerGroup) },
clockOpen: !stream.ended && !stream.aborted,
emittedCount: stream.records.size, blockers: await blockers(observer, holderPid) };
report.heldSamples.push(sample); assertHeldSample(sample, holderPid, processes);
if (Date.now() >= report.timing.requestAtMs + W.heldAfterActualRequestMs) break;
await delay(Math.min(W.pollIntervalMs, report.timing.requestAtMs + W.heldAfterActualRequestMs - Date.now()));
} while (true);
expect(stream.records.size, "Real provider continues emitting while transcript writes and Stop await the run lock").toBeGreaterThan(emissionsAtRequest);
report.timing.releaseRequestedAtMs = Date.now();
await holder.unsafe("COMMIT"); held = false; report.timing.releaseCommitAckAtMs = Date.now();
let acknowledged = false, absent = false;
const observationStartedAtMs = Date.now();
const observe = async () => {
const snapshot = await cancellationSnapshot(observer, company.id, run.id);
if (snapshot.intent) { assertIntentAudit(snapshot); report.intentObserved ??= snapshot; }
if (!acknowledged) {
const value = snapshot.intent?.dispatchState === "acknowledged";
report.observerSamples.push({ kind: "ack-query", startedAtMs: snapshot.queryStartedAtMs,
completedAtMs: snapshot.observedAtMs, durationMs: snapshot.observedAtMs - snapshot.queryStartedAtMs, value });
if (value) {
assertIntentAudit(snapshot); acknowledged = true; report.timing.ackObservedAtMs = snapshot.observedAtMs;
report.ack = snapshot; report.timing.ackSuccessfulQueryStartedAtMs = snapshot.queryStartedAtMs;
} else report.timing.lastAckNotObservedAtMs = snapshot.observedAtMs;
}
if (!absent) {
const checkStartedAtMs = Date.now();
const value = processAbsent(processes.runner.pid) && processAbsent(-processes.group)
&& processAbsent(processes.provider.pid) && processAbsent(-processes.providerGroup);
const completedAtMs = Date.now();
report.observerSamples.push({ kind: "process-check", startedAtMs: checkStartedAtMs,
completedAtMs, durationMs: completedAtMs - checkStartedAtMs, value });
if (value) {
absent = true; report.timing.absenceObservedAtMs = completedAtMs;
report.timing.absenceSuccessfulCheckStartedAtMs = checkStartedAtMs;
report.processesAfter = { runnerAbsent: true, groupAbsent: true, providerAbsent: true, providerGroupAbsent: true,
group: processes.group, providerGroup: processes.providerGroup, checkedAtMs: completedAtMs };
} else report.timing.lastProcessesPresentAtMs = completedAtMs;
}
};
await observe();
const settlement = await observeRunSettlement({ expectedRuns: 1,
readRuns: async () => { await observe(); return [await json(await request.get(`/api/heartbeat-runs/${run.id}`))]; },
readRun: async () => json(await request.get(`/api/heartbeat-runs/${run.id}`)),
poll: (read: () => Promise<boolean>, timeout: number) => expect.poll(read, { timeout, intervals: [W.pollIntervalMs] }).toBe(true),
onOutcome: (outcome: unknown) => { report.settlement = outcome; } });
expect(settlement.runs[0].status).toBe("cancelled"); report.settledAtMs = Date.now();
const sharedDetail = await json(await request.get(`/api/heartbeat-runs/${run.id}`));
report.sharedSettlement = { runId: sharedDetail.id, companyId: sharedDetail.companyId, status: sharedDetail.status,
cancellation: sharedDetail.resultJson?.cancellation ?? null,
executionCancellation: sharedDetail.resultJson?.executionCancellation ?? null,
nativeCancellation: sharedDetail.resultJson?.nativeCancellation ?? null, observedAtMs: Date.now() };
await expect.poll(async () => { await observe(); return acknowledged; },
{ timeout: W.acknowledgementAfterSettlementMs, intervals: [W.pollIntervalMs] }).toBe(true);
await expect.poll(async () => { await observe(); return absent; },
{ timeout: W.acknowledgementAfterSettlementMs, intervals: [W.pollIntervalMs] }).toBe(true);
report.timing.postReleaseObservationStartedAtMs = observationStartedAtMs;
await expect.poll(() => stopSettled, { timeout: W.acknowledgementAfterSettlementMs,
intervals: [W.pollIntervalMs] }).toBe(true);
const response = await stopRequest.response(); expect(response?.ok()).toBe(true); report.stopHttpStatus = response!.status();
let lease: any;
await expect.poll(async () => {
lease = (await json(await request.get(`/api/environments/${environment.id}/leases`)))
.find((candidate: any) => candidate.heartbeatRunId === run.id);
return Boolean(lease?.releasedAt && ["released", "expired", "failed"].includes(lease.status) && lease.cleanupStatus !== "failed");
}, { timeout: W.leaseAfterSettlementMs }).toBe(true);
expect(lease.provider).toBe("local"); expect(lease.providerLeaseId).toBeNull();
report.lease = lease;
report.authorityAfter = await runtimeAuthority({ appRoot, companyId: company.id, environmentId: environment.id,
issueIds: [issue.id], transport: "local", afterStop: true });
report.authoritativeRetirementState = assertAuthoritativeRetirement(report);
const seen = await visible(), events = await readEvents();
const deltas = events.map(event => event.payload?.prpEvent).filter(event => event?.eventType === "item.delta"
&& String(event.payload?.text).startsWith(`[${code}`));
expect(deltas.length).toBeGreaterThanOrEqual(W.visibleBeforeLock);
deltas.forEach((event, index) => {
expect(event.payload.text).toBe(`[${code}${index.toString(36)}] `); expect(event.runId).toBe(run.id);
expect(event.sourceInstanceId).toBeTruthy(); expect(event.sourceEventId).toBeTruthy();
expect(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0).toBe(true);
expect(stream.records.has(index)).toBe(true);
});
expect(new Set(deltas.map(event => `${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`)).size).toBe(deltas.length);
seen.forEach((marker: any) => expect(marker.index).toBeLessThan(deltas.length));
expect(events.filter(event => event.eventType === "run.result.accepted" || event.payload?.prpEvent?.eventType === "run.result.accepted")).toHaveLength(0);
await page.screenshot({ path: info.outputPath("blocked-stop-existing-history.png"), fullPage: true });
await page.reload();
await expect(page.getByRole("button", { name: "Stop", exact: true })).toHaveCount(0);
await expect(page.getByText(`[${code}${seen.at(-1).index.toString(36)}]`, { exact: false }).first()).toBeVisible();
const transcriptRows = (rows: any[]) => rows.filter(row => row.payload?.prpEvent?.eventType === "item.delta")
.map(row => ({ id: row.id, seq: row.seq, event: row.payload.prpEvent }));
expect(transcriptRows(await readEvents())).toEqual(transcriptRows(events));
await expect.poll(() => stream.ended || stream.aborted, { timeout: 30_000 }).toBe(true);
expect(state.errors).toEqual([]);
report.transcript = { individualDeltas: deltas.length, uniqueSourceIdentities: deltas.length, ordered: true,
persistedAfterReload: true, resultAcceptedEvents: 0, seen, deltas, preludeCount: prelude.length };
report.clock = { ...stream, records: [...stream.records.values()] };
report.complete = true;
} catch (error) {
report.failure = { message: String(error), stack: error instanceof Error ? error.stack : undefined, atMs: Date.now() };
throw error;
} finally {
const cleanupErrors: unknown[] = [];
const cleanup = async (step: string, action: () => Promise<unknown>) => {
try { await action(); } catch (error) { cleanupErrors.push({ step, error: String(error) }); }
};
// Always release our exact owned lock before invoking any public cancellation cleanup.
if (held) await cleanup("rollback-owned-lock", async () => { await holder.unsafe("ROLLBACK"); held = false; });
if (holder) await cleanup("close-holder", () => holder.end({ timeout: 5 }));
if (observer) await cleanup("close-observer", () => observer.end({ timeout: 5 }));
report.cleanupSettlement = {};
await cleanup("settle-owned-run-before-clock-close", () => cleanupOwnedRunBeforeClockClose({
companyId: company.id, runId: companyRunId,
readRun: companyRunId ? async () => json(await request.get(`/api/heartbeat-runs/${companyRunId}`)) : undefined,
cancelRun: async () => { const response = await request.post(`/api/heartbeat-runs/${companyRunId}/cancel`);
return { status: response.status(), run: await json(response) }; },
closeClock: async () => {
for (const pending of clock.pending.values()) for (const response of pending.values()) if (!response.writableEnded) response.writeHead(503).end();
clock.gate.closeAllConnections(); await new Promise<void>(resolve => clock.gate.close(() => resolve()));
}, record: report.cleanupSettlement,
}));
await cleanup("restore-experimental-settings", async () => json(await request.patch("/api/instance/settings/experimental", {
data: { enableNativeRunner: original.enableNativeRunner, enableWorktreeRunExecution: original.enableWorktreeRunExecution } })));
report.cleanup = { complete: cleanupErrors.length === 0, errors: cleanupErrors };
report.complete &&= report.cleanup.complete; report.finishedAtMs = Date.now();
let validationError: unknown;
if (report.complete) {
try { report.metrics = metrics(report); assertFunctional(report); }
catch (error) { validationError = error; report.complete = false; report.validationFailure = String(error); }
}
await info.attach("native-ingestion-blocked-stop-evidence", { body: JSON.stringify(report, null, 2), contentType: "application/json" });
if (validationError) throw validationError;
if (!report.failure) expect(cleanupErrors).toEqual([]);
page.off("requestfinished", stopFinished); page.off("requestfailed", stopFinished);
}
});
tests/e2e/native-event-ingestion.journey.ts
+19 -43
View File
@@ -8,7 +8,8 @@ import path from "node:path";
import { expect, test } from "@playwright/test";
import { json } from "./agent-chat.shared";
import { h5Enabled, H5_MODE, H5_PROTOCOL, H4E_PROTOCOL, H5_TOPOLOGY, H5_LIMITS, H5_WORKLOAD_SHA256, ORIGINAL_BUDGETS,
observeStreaming, observeRunSettlement, originalTerminal, prepareParticipantAgents, assertParticipantTopology, assertRecordedFailures } from "./fixtures/native-ingestion-h5.mjs";
observeStreaming, observeRunSettlement, originalTerminal, prepareParticipantAgents, assertParticipantTopology, assertRecordedFailures, collectDurableRunEvents, assertDurableRunEvents,
assertPreludeEvents, assertVisibleMarkerClocks } from "./fixtures/native-ingestion-h5.mjs";
type SeenMarker = { markerCode: string; index: number; visibleMs: number };
type Emission = { nonce: string; markerCode: string; index: number; intendedMs: number; emittedMs: number; receivedAtMs: number };
@@ -511,23 +512,18 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
const listed = await json(await request.get(`/api/companies/${company.id}/heartbeat-runs?limit=1000`));
const foreground = listed.find((run: any) => (run.issueId ?? run.nativeIssueId ?? run.contextSnapshot?.issueId ?? run.contextSnapshot?.taskId) === issue.id);
expect(foreground).toBeTruthy();
const identities = new Set<string>(); let count = 0, afterSeq = 0;
const preludeRows: any[] = []; let afterSeq = 0;
while (true) {
const rows = await json(await request.get(`/api/heartbeat-runs/${foreground.id}/events?afterSeq=${afterSeq}&limit=1000`));
if (!rows.length) break;
for (const row of rows) {
const event = row.payload?.prpEvent;
if (event?.eventType !== "item.delta" || !String(event.itemId ?? "").startsWith("prelude-")) continue;
expect(event.payload?.text).toBe(count === workload.prelude - 1 ? ` PRELUDE_${nonce}_READY ` : ".");
count++; identities.add(`${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}`);
}
preludeRows.push(...rows);
const next = Math.max(...rows.map((row: { seq: number }) => row.seq));
expect(next).toBeGreaterThan(afterSeq); afterSeq = next;
}
preludeProof = { runId: foreground.id, individualDeltas: count, uniqueSourceIdentities: identities.size,
const checked = assertPreludeEvents({ rows: preludeRows, runId: foreground.id, token: nonce, expectedEvents: workload.prelude });
preludeProof = { runId: foreground.id, ...checked,
retainedGlobalHead: afterSeq, verifiedAtMs: Date.now(), method: "Independent normal event API read before common emission gate release" };
active.preludeProof = preludeProof;
expect(count).toBe(workload.prelude); expect(identities.size).toBe(count);
}
const controllerAuthority = await runtimeAuthority({
appRoot, companyId: company.id, environmentId: caseEnvironment.id, issueIds, transport: caseTransport,
@@ -645,12 +641,8 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
? seenMarkers.every(marker => state.streams.get(nonce)!.records.has(marker.index))
: stream.records.size === workload.events && stream.ended)), { timeout: 30_000 }).toBe(true);
expect(state.errors).toEqual([]);
const visible: Visible[] = seenMarkers.map(marker => {
const original = state.streams.get(nonce)!.records.get(marker.index);
expect(original, `Missing original provider clock for visible ${marker.markerCode}:${marker.index}`).toBeTruthy();
return { ...original!, ...marker }; // Keep the original paint timestamp; never restamp after clock arrival.
});
if (!stop) expect(new Set(visible.map(item => item.index)).size).toBe(workload.events);
const visible: Visible[] = assertVisibleMarkerClocks({ seenMarkers, sourceClocks: state.streams.get(nonce)!.records,
markerCode, expectedEvents: workload.events, stop });
// Terminal drainage has finished. Derive paint gaps from the retained
// observer timestamps without adding work to the streaming observer.
const paintTimes = [...new Set(visible.map(item => item.visibleMs))].sort((a, b) => a - b);
@@ -693,39 +685,23 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
const token = issuedTokens.get(taskId)!;
const code = state.streams.get(token)!.markerCode;
expect(token).toBeTruthy();
const deltas: Array<{ index: number; identity: string }> = [];
let afterSeq = 0, resultAcceptedEvents = 0;
const persistedRows: any[] = [];
let afterSeq = 0;
active.verifyingRun = { runId: run.id, nonce: token, markerCode: code, afterSeq, rows: 0 };
while (true) {
const rows = await json(await request.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=1000`));
if (!rows.length) break;
for (const row of rows) {
const event = row.payload?.prpEvent;
if (row.eventType === "run.result.accepted" || event?.eventType === "run.result.accepted") resultAcceptedEvents++;
if (event?.eventType !== "item.delta") continue;
for (const match of String(event.payload?.text ?? "").matchAll(new RegExp(`\\[${code}([a-z0-9]{1,3})\\]`, "g"))) {
const index = Number.parseInt(match[1], 36);
expect(event.payload.text, "Every durable normalized delta retains its complete compact payload").toBe(`[${code}${index.toString(36)}] `);
if (h5) { expect(event.runId).toBe(run.id); expect(event.sourceInstanceId).toBeTruthy();
expect(event.sourceEventId).toBeTruthy(); expect(Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0).toBe(true); }
deltas.push({ index, identity: `${event.sourceInstanceId}:${event.sourceEventId}:${event.sourceSeq}` });
}
}
persistedRows.push(...rows);
const next = Math.max(...rows.map((row: { seq: number }) => row.seq));
expect(next).toBeGreaterThan(afterSeq); afterSeq = next;
active.verifyingRun = { runId: run.id, nonce: token, markerCode: code, afterSeq, rows: persistedRows.length };
}
active.latestDurableRead = { runId: run.id, nonce: token, markerCode: code, deltas };
await expect.poll(() => deltas.every(delta => state.streams.get(token)!.records.has(delta.index)), { timeout: 30_000 }).toBe(true);
for (const delta of deltas) {
const original = state.streams.get(token)!.records.get(delta.index)!;
expect({ nonce: original.nonce, markerCode: original.markerCode, index: original.index })
.toEqual({ nonce: token, markerCode: code, index: delta.index });
}
expect(new Set(deltas.map(item => item.identity)).size).toBe(deltas.length);
expect(deltas.map(item => item.index)).toEqual(Array.from({ length: stop ? deltas.length : workload.events }, (_, index) => index));
if (stop) for (const item of visible) expect(deltas.some(delta => delta.index === item.index)).toBe(true);
expect(resultAcceptedEvents).toBe(stop ? 0 : 1);
durable.push({ runId: run.id, token, markerCode: code, individualDeltas: deltas.length, uniqueSourceIdentities: deltas.length,
matchedSourceClocks: deltas.length, ordered: true, resultAcceptedEvents });
const collected = collectDurableRunEvents(persistedRows, code);
active.latestDurableRead = { runId: run.id, nonce: token, markerCode: code,
deltas: collected.deltas.map(({ index, identity }) => ({ index, identity })) };
await expect.poll(() => collected.deltas.every(delta => state.streams.get(token)!.records.has(delta.index)), { timeout: 30_000 }).toBe(true);
durable.push(assertDurableRunEvents({ collected, runId: run.id, token, markerCode: code, events: workload.events, stop,
sourceClocks: state.streams.get(token)!.records, visible, requireNativeIdentity: h5 }));
}
await expect.poll(() => [...state.streams.values()].every(stream => stream.ended || stream.aborted), { timeout: 30_000 }).toBe(true);
expect(state.errors).toEqual([]);
tests/e2e/playwright-native-ingestion-blocked-stop.config.ts
+5
View File
@@ -0,0 +1,5 @@
import { defineConfig } from "@playwright/test";
import base from "./playwright-native-ingestion.config";
process.env.PAPERCLIP_INGESTION_COMPANION_PROTOCOL = "blocked-stop-history-v1";
export default defineConfig({ ...base, testMatch: "native-event-ingestion-blocked-stop.journey.ts", timeout: 900_000,
retries: 0, workers: 1, use: { ...base.use, trace: "off", video: "off" } });