mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-11 05:11:49 +01:00
test: publish fresh cursor gains and frozen application harness
This commit is contained in:
1 parent
d8cd3be0ae
commit
dfde99df5a
42 files changed
+13003
-61
No files matched your search
@@ -0,0 +1,11 @@
|
||||
# H4c application measurement qualification (2026-11-03)
|
||||
|
||||
Application timing acceptance remains pending. The frozen driver is `7d7e7a88d562a95a1d407b76474763b666c181f5` (`native-ingestion-h4c`), with protocol SHA `5db4129bbdc1fa38775121aff5d0e68a8a11762519e41cfdf36eacd90ba7256b`, comparator `01e0de114c886c255aaa7062a63f3e48788d8a83ce50921d672b2d661baef43b` and unchanged locked driver dependencies. Its exact eight protocol files, comparator and offline eligibility checks are included in this PR. Root and independent review cleared the source; the five comparison-gate tests, target typecheck, syntax and test listing passed. These source checks do not establish runtime qualification or a performance gain.
|
||||
|
||||
The [prospective registration](native-event-ingestion-2026-11-03/h4c/prospective-application-protocol.md) pins the separate latest-common-base application revisions, workloads, budgets, five alternating pairs per stage, noise review and improvement gates. The [reproduction handoff](native-event-ingestion-2026-11-03/h4c/reproduction.md) starts with a small remote3x16 qualification which exercises one real two-background activation wave, then the original local Stop. Only after it passes may one full B0 qualification run. A full functional PASS must be independently reviewed before any application A/A. Qualification runs stay outside accepted timing manifests.
|
||||
|
||||
The [comparison runner](../../scripts/native-ingestion/run-ce33-browser.mjs) is an exact copy of the prospectively reviewed private runner, SHA `2244e238642caafd488a7c86fb463376c8459ee0f06f8054555407a174698fa1`. It retains the original disposable Linux directory layout and expects independently installed/compiled frozen app roots plus their build-inventory receipt there. Its dry run starts no workloads. Each executed stage needs an explicit release JSON binding the stage, all full app and driver refs, protocol/comparator, this runner's own hash and the qualified build inventory hash. Candidate releases also bind the exact reviewed A/A noise-report SHA, verified before the first arm and immediately before comparison. The stage receipt retains the full release and release-file hash. Use new output directories and preserve all failure/cleanup evidence.
|
||||
|
||||
The preceding H4b full B0 qualification failed during unmeasured readiness:100 clock hellos but99 still-open provider GETs, with no epoch and zero measured emissions. The foreground GET had started3.485s before the shared300s readiness window. The final provider hello arrived302.208s after it; the persisted timeout error arrived300399.923ms after it. H4c replaces sequential background admission with bounded waves of two normal PATCHs, waiting for both actual provider GETs and clock hellos. Both current promises settle before any failure propagates, and a failed wave prevents later waves. All100 GETs must remain open after the closed read-only authority snapshot and before the common epoch.
|
||||
|
||||
No setup, streaming, terminal, Stop or lease timeout was extended. The six full measured workloads, event identities/payloads, rates, original source clocks, two-frame browser observation, control/durability checks and noise/improvement gates are unchanged. The prior failure remains excluded evidence, rather than a passing load sample. Remote streaming uses the normal first-party fake sandbox plugin on the same host, preserving the existing raw-recovery policy. Its void cleanup grants no remote Stop receipt: [real Daytona Stop](2026-11-03-native-event-ingestion-cloud-stop.md) is separately qualified in the batching PR. The original local-journal overload remains a separately documented limit.
|
||||
@@ -0,0 +1,79 @@
|
||||
# Interleaved append-service measurements (2026-11-03)
|
||||
|
||||
The fresh source-layer B0 A/A passed the unchanged noise gates, followed by a passing B0→B1 cursor comparison. At 10k histories, median active append cost falls about **16–22%**; at 100k, about **56–60%**. Every required long-history case beats its observed A/A noise in **5/5 individual pairs**. This measures the real append service and PostgreSQL under the preregistered interleaving. It does not establish application burst completion, browser delivery, batching performance, or production improvement. The remaining source and application gates are pending; the full task is not complete.
|
||||
|
||||
## What was measured
|
||||
|
||||
The frozen SOURCE refs are B0 `5337df423e622755d44f3ece5447d1e0beef5f40`, B1 `e9b909015ad3d517f28c45f0c66c8f41d6112b4b`, and B2 `fcb5d94848b3c4b0b32096ff5e023618e0f0637b`. Only B0 A/A and B0→B1 are reported here. The separate latest production-built APP refs are B0 `a36567f9fbf805ad72207d54f539516e3121741e`, B1 `d8cd3be0aeed4f9fb93ecc3cf725f1d46d5a0310`, and B2 `914129afb141d81c0d6d8c5f0ea11c20bd69a3da`. Source-layer results are not relabelled as measurements of those entire applications.
|
||||
|
||||
The [prospective protocol](native-event-ingestion-2026-11-03/interleaved-source/prospective-protocol.md) was frozen before the new samples: protocol SHA256 `302c11ec61d5170822c08e47f798ca582b8e5afefd9d223fb9023d9ead7a2ad3`, measurement SHA256 `c8adaf5ac16cab67513d6c8b888afa44d95fa3688d4831b8126fe3d997f1d039`, workload SHA256 `ca4894b0723d0bffff0fc88d1596d64aa8fd75aa1dd8d861372ea993db24e164`, and unchanged gate SHA256 `ded8bcbb8f05732348f6520b9c5e41b7c35bc098997eb263a9dcb174eede75fc`. The archived protocol's initial source-only status predates the separately recorded [full release](native-event-ingestion-2026-11-03/interleaved-source/full-release.md) and execution receipts.
|
||||
|
||||
Each ordinary measured arm sample contains eight interleaved blocks of 64 attempts: 512 attempts total, retaining the original 16 groups of 32 and payloads. The first arm alternates across five pairs. Two monolithic 512-attempt warmups per arm remain excluded. Ordered/warm-suffix samples insert 512 new events; **mixed replay samples insert 256 new events and reuse 256 exact duplicates**. Duplicate receipts allocate nothing. Histories and source sequences are matched at corresponding blocks, with independent identity/payload/count verification after every block and over the complete arm sample.
|
||||
|
||||
The estimand is the **sum of an arm's active append intervals**, including receipt checks and excluding generation, CPU probes, verification, trace writes and inter-arm gaps. It is not the wall time from emitting a burst through final application drainage. Raw reports separately preserve whole-pair elapsed time, first/last observed commit bounds, independent final verification, and bookkeeping gaps. There is no asynchronous source queue in this closed-loop experiment.
|
||||
|
||||
Both full reports contain 12 cases and 60 complete pairs, with 750 started/completed measured blocks, 120 final arm-verification starts and one cleanup-complete record. Each report contains 46,110 measured append attempts across both arms, including 30 single-operation late-gap attempts, plus 18,432 excluded warmup attempts. The dedicated quiet Linux host recorded Node 26.4.0, 16 CPUs, PostgreSQL 18.1, shared_buffers 128MB, work_mem 4MB and JIT on. Exact environment and commands are in the raw reports; unprofiled acceptance intervals were separate from SQL/plan diagnostics and builds.
|
||||
|
||||
## Cursor cost and repeatability
|
||||
|
||||
Values are median **active milliseconds per 512 attempts**, followed by observed min–max across five samples. Reduction is computed from the two arm medians. This is not an individual-event tail-latency distribution.
|
||||
|
||||
| Seeded history / case | B0 median [min–max] ms | B1 median [min–max] ms | Reduction | Pairs beyond noise | A/A envelope |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1,000 / ordered | 3823.2 [3660.2–3890.5] | 3554.4 [3522.7–3603.7] | 7.03% | 5/5 | 1.408% |
|
||||
| 1,000 / replay | 3164.5 [3069.6–3184.9] | 3002.7 [2943.1–3046.2] | 5.11% | 3/5 | 3.982% |
|
||||
| 1,000 / warm-suffix | 4152.8 [3944.3–4267.0] | 3542.4 [3468.8–3605.4] | 14.70% | 5/5 | 1.884% |
|
||||
| 10,000 / ordered | 4209.6 [4049.1–4255.4] | 3530.6 [3528.6–3597.9] | 16.13% | 5/5 | 1.747% |
|
||||
| 10,000 / replay | 3575.6 [3542.3–3624.2] | 3016.3 [2948.3–3071.0] | 15.64% | 5/5 | 1.630% |
|
||||
| 10,000 / warm-suffix | 4614.0 [4464.4–4736.5] | 3592.8 [3571.8–3615.2] | 22.13% | 5/5 | 11.830% |
|
||||
| 100,000 / ordered | 8029.2 [7985.5–8157.3] | 3540.0 [3503.1–3586.7] | 55.91% | 5/5 | 1.069% |
|
||||
| 100,000 / replay | 7456.0 [7350.1–7471.5] | 2989.0 [2938.5–3000.5] | 59.91% | 5/5 | 0.933% |
|
||||
| 100,000 / warm-suffix | 8347.4 [7836.9–8611.9] | 3402.4 [3119.4–3627.9] | 59.24% | 5/5 | 0.762% |
|
||||
|
||||
All nine A/A timing cases met maximum paired relative difference ≤20% and median arm drift ≤10%; their largest observed values were 11.830% and 6.133%. Every required 10k/100k case passes both the median and individual-pair improvement requirements. The 1k cases pass nonregression. In particular, 1k mixed replay has only 3/5 pairs beyond noise, so it does not meet the four-pair improvement requirement. The [summary](native-event-ingestion-2026-11-03/interleaved-source/summary.json) retains all comparisons and sample counts.
|
||||
|
||||
A separately seeded SQL diagnostic at 100k ordered history records a baseline retained-history scan of 100,032 entries versus a one-row retained-event identity lookup and bounded projection work for B1. All candidate structural guards pass. SQL calls/attempt remain 5→5 ordered and 4→4 mixed replay; warm suffix falls 6→5. Transactions remain one per attempt. These counts describe the separate diagnostic workload; fewer calls alone are not the performance result.
|
||||
|
||||
For 100k ordered samples, descriptive median backend CPU is 10.332→1.719 ms/attempt and owner-process CPU is 1.960→1.839 ms/attempt. CPU is not a separately qualified improvement gate here. Linux backend observations are tick-quantized at 10ms per block; an eight-block sum can carry up to 80ms aggregate quantization uncertainty. Owner CPU and memory describe the shared benchmark process, excluding an assembled server, Rust Runner, browser and external load driver. Heap deltas do not identify owned allocations or establish allocation efficiency. The separate 32-transaction SQL diagnostic records median lock-held-through-commit intervals of 14.17→5.89ms; acquisition query duration includes network/row-read work and is not pure lock wait.
|
||||
|
||||
Late-gap closure remains a five-pair **single-operation structural diagnostic**, outside the preregistered latency gate. Its 1k median rises 7.403→9.507ms (**+2.10ms**), and its 10k median rises 7.701→8.573ms (**+0.87ms**). At 100k it falls 24.455→9.347ms. These tradeoffs remain visible; single operations and five samples establish no tail-latency claim.
|
||||
|
||||
## Source attribution and its limits
|
||||
|
||||
[Immutable source comparisons](native-event-ingestion-2026-11-03/interleaved-source/source-app-core-equivalence.json) verify the selected append/allocator/cursor/port/collector, heartbeat event schema, canonical/lossless codec, DB client and work-signal paths between each frozen SOURCE and its corresponding APP ref. The [four-table snapshot receipt](native-event-ingestion-2026-11-03/interleaved-source/upstream-table-snapshot-equality.json) records unchanged event/cursor/range/run tables in the normal upstream merge. Static locked-graph checks for [B1](native-event-ingestion-2026-11-03/interleaved-source/source-b1-app-b1-dependency-graph.json) and [B2](native-event-ingestion-2026-11-03/interleaved-source/source-b2-app-b2-dependency-graph.json) find 28/28 DB runtime nodes, 14/14 pg nodes and 29/29 tsx-launcher nodes identical, including package-integrity/snapshot entries. drizzle-kit 0.31.10→0.31.11 and its one-run get-tsconfig development-tool edge are outside those graphs.
|
||||
|
||||
This is **selected core and runtime-DB-graph equality**, not whole application or whole transitive-closure equality. Extra vendor-index exports, Runner Codex diagnostics and ACPX environment/Claude-key handling differ outside that selection. The new run-log wrapper, native executor/finalizer and Rust changes require separate application and Stop qualification. Installed bits, OS and build provenance remain separate evidence. The source harness intentionally uses a compatible migrated APP_ROOT schema/client and unchanged support dependencies for both service arms; it does not create independently built whole applications per arm.
|
||||
|
||||
The [earlier complete monolithic A/A](native-event-ingestion-2026-11-03/interleaved-source/excluded-monolithic-service-aa-b0.json.gz), SHA256 `cfcfeb3c12812dbe48c2afe6013320922d5670a410d033d6ad9a3a2f91236ef8`, failed the unchanged noise gates and remains excluded. No candidate used it as calibration. The prospective interleaved design changes the source estimand and therefore starts a new same-workload A/A; it does not turn that failure into a pass. No specific warmup or host cause is established by the subsequent passing experiment.
|
||||
|
||||
## Reproduce and inspect
|
||||
|
||||
All raw reports and append-only traces are deterministic gzip files under [interleaved-source](native-event-ingestion-2026-11-03/interleaved-source/manifest.json). The manifest records compressed and exact decompressed hashes. The executed command receipts, repeatability result, release and quiet-return receipt are retained. `gzip -dc FILE.gz` restores the original report/trace bytes. Historical absolute Linux paths in those exact raw receipts are disposable task-owned fixture paths; no private account/credential metadata is published.
|
||||
|
||||
Use a quiet disposable Linux host with Node 26.4.0 and independently install the frozen SOURCE_B1 checkout's locked dependencies (`pnpm install --frozen-lockfile`). The benchmark starts and cleans up its own embedded PostgreSQL when `PAPERCLIP_EVENT_INGESTION_BENCHMARK_DATABASE_URL` is unset. A supplied database must be disposable, loopback-only and named with the `_test` suffix; migrations and fixtures run there. Select that exact checkout as `SOURCE_B1_ROOT`; `TOOLS_ROOT` contains the current public helper. The measurement file must be copied byte-for-byte **outside** APP_ROOT with an `.mts` extension, preserving ESM execution. Never run builds, profilers or other workloads concurrently. Use new external output paths for each experiment:
|
||||
|
||||
```sh
|
||||
# Set absolute paths; EXPERIMENT_DIR must be a new external evidence directory.
|
||||
mkdir "$EXPERIMENT_DIR"
|
||||
unset PAPERCLIP_EVENT_INGESTION_BENCHMARK_DATABASE_URL
|
||||
cp "$TOOLS_ROOT/scripts/benchmark-native-event-ingestion-interleaved.ts" "$EXPERIMENT_DIR/measurement.mts"
|
||||
sha256sum "$EXPERIMENT_DIR/measurement.mts"
|
||||
|
||||
# First qualify a new host/harness. Quick evidence cannot establish full acceptance.
|
||||
node --import "$SOURCE_B1_ROOT/server/node_modules/tsx/dist/loader.mjs" "$EXPERIMENT_DIR/measurement.mts" \
|
||||
--app-root="$SOURCE_B1_ROOT" --baseline-ref=5337df423e622755d44f3ece5447d1e0beef5f40 \
|
||||
--quick --phase=aa --output="$EXPERIMENT_DIR/quick.json"
|
||||
|
||||
# Review qualification and cleanup, then run a fresh full A/A and comparison.
|
||||
node --import "$SOURCE_B1_ROOT/server/node_modules/tsx/dist/loader.mjs" "$EXPERIMENT_DIR/measurement.mts" \
|
||||
--app-root="$SOURCE_B1_ROOT" --baseline-ref=5337df423e622755d44f3ece5447d1e0beef5f40 \
|
||||
--phase=aa --output="$EXPERIMENT_DIR/service-aa-b0.json" --require-repeatability
|
||||
node --import "$SOURCE_B1_ROOT/server/node_modules/tsx/dist/loader.mjs" "$EXPERIMENT_DIR/measurement.mts" \
|
||||
--app-root="$SOURCE_B1_ROOT" --baseline-ref=5337df423e622755d44f3ece5447d1e0beef5f40 \
|
||||
--phase=compare --noise-report="$EXPERIMENT_DIR/service-aa-b0.json" \
|
||||
--output="$EXPERIMENT_DIR/service-b0-b1.json" --require-acceptance
|
||||
```
|
||||
|
||||
The designated immutable SOURCE tags are `native-ingestion-benchmark-2026-11-03-fixture2-{b0,b1,b2}`; APP tags are `native-ingestion-benchmark-2026-11-03-ce33-{b0,b1,b2}`. Resolve them to the exact full commits above before running; their names do not replace commit verification. The pinned SOURCE_B1 root must be `e9b909015ad3d517f28c45f0c66c8f41d6112b4b`, not a newer delivery checkout. The gates remain the unchanged published module. Source-only validation uses the same command with `--check-source`; report validation uses `--validate-report=REPORT` and the matching phase, baseline and exact measurement/source bytes. The [dependency helper](native-event-ingestion-2026-11-03/interleaved-source/compare-runtime-dependencies.cjs) accepts repository, immutable before/after refs and a new output JSON path; its path/role-label-only adaptation is separately declared. It reads Git and the locked yaml parser without launching a database.
|
||||
|
||||
B1 A/A, incremental B1→B2 and combined B0→B2 source measurements, application/browser comparisons, batching/control responsiveness gates and final delivery checks remain separate. Nothing in this source report establishes completion of the full ingestion plan or a deployed-production gain.
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
# Latest common application measurement registration — prospective draft
|
||||
|
||||
No accepted browser timing is authorized by this draft. Root release follows the new H4c small and full baseline functional qualifications and this registration's review. The source-only experiment remains separately pinned at fixture2 5337/e9/fcb with its own interleaved calibration and protocol.
|
||||
|
||||
Application B0=a36567f9fbf805ad72207d54f539516e3121741e, B1=d8cd3be0aeed4f9fb93ecc3cf725f1d46d5a0310, B2=914129afb141d81c0d6d8c5f0ea11c20bd69a3da. Immutable ce33 tags, independent normal frozen installs and production builds, clean HEAD equal to server build stamp and full artifact inventories are required. These contain the same upstream base and same first-party fake-provider cleanup fixes. The new shared Rust input tree is eac015472eb88b49aa92e6f30def348aa34b4911; build the first target fresh with the package's pinned Rust1.97.1, then share only that exact input-equal target among these three checkouts. Record full protocol/build input objects, toolchain, actual binary hash and metadata. Old app/browser/Stop results do not establish current app acceptance.
|
||||
|
||||
Driver H4c=7d7e7a88d562a95a1d407b76474763b666c181f5 (native-ingestion-h4c), own frozen offline ignore-scripts dependencies. Protocol 5db4129bbdc1fa38775121aff5d0e68a8a11762519e41cfdf36eacd90ba7256b, comparator01e0de114c886c255aaa7062a63f3e48788d8a83ce50921d672b2d661baef43b, driver lock54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8. New private application-only orchestrator /private/tmp/native-ingestion-run-ce33-browser.mjs is a draft until its exact source hash and release record are reviewed. The normal H4c bootstrap and TLS remote-server assembly are mandatory; no direct compiled entry override, raw-recovery policy change, synthetic receipt, additional diagnostic provider loader/resolver override or forced external checkpoint policy. The existing normal H4c local leaf fixture remains unchanged.
|
||||
|
||||
First qualify remote3x16@25ms (one visible foreground plus a real two-background activation wave) and original local Stop1, then the full B0 six-case fixture. Smoke remains performance-ineligible; full measured workloads are unchanged. Full cases are unchanged: paced1x128@25ms; burst1x16; burst10x128; burst100x512; existing history1000 actual deltas + measured128 across10; local Stop1x512@100ms after at least3 visible. All participant GET-readiness and original clock hello messages precede one common epoch, with50ms setup-only readiness observation. Counts, rates, compact marker payloads, original source clocks, offline joins and2RAF observation are unchanged. Each remote run must have normal company/environment/lease/controller-header proof in a closed READ ONLY REPEATABLE READ transaction before epoch. Stop retains its original local journal, independent OS/PID/group/provider disappearance, durable acknowledgement and normal lease proof. Real cloud Stop with a genuine provider receipt, shared acknowledgement and independent SDK state is a separate required functional gate; there is no100-cloud-resource workload.
|
||||
|
||||
Budgets are unchanged:300s setup readiness,900s full invocation,120s streaming,180s terminal drainage,30s Stop acknowledgement and30s remote lease cleanup. These are not acceptance improvements. Trace/video/screencast and CPU/SQL profiling remain OFF during accepted comparisons. The2RAF measurement observer is declared measurement overhead. Manual screenshots follow timed drainage. Failed runs retain active DOM/clock/provider/durable evidence and cleanup receipts and cannot qualify. Every background participant needs exact source identities/payloads/counts and independent reload/persistence proof; final backlog must drain and cleanup must complete without errors.
|
||||
|
||||
Perform five fresh alternating pairs per stage, serially on the exclusive quiet VM: B0 A/A; B0→B1 nonregression; B1 A/A; B1→B2 improvement; B0→B2 improvement. Each arm uses a new disposable database/state and fixed matched payloads, load, resource limits and cache policy. No retries, omitted slow pairs or reuse of baseline arm samples across comparisons. Preserve every raw observation and command/exit receipt. Setup is outside the recorded source-emission/browser/terminal interval. The foreground browser is measured under100-run background load; this does not measure100 independent browsers.
|
||||
|
||||
Browser A/A computes a separate per-metric maximum absolute paired difference envelope. The original browser protocol has no additional fixed percentage A/A cutoff; root must inspect the complete A/A noise suitability and freeze its observed envelope before releasing candidate comparisons. A process exit alone is not a repeatability claim. Each stage release pins the orchestrator and qualified build inventory hashes; each candidate release also pins the exact reviewed A/A noise-report bytes, checked before its first arm and again before comparison. Stage receipts retain the release bytes hash and noise hash. No old A/A envelope can be reused. B0→B1 requires all fixed latency/control metrics to remain within the observed noise. B1→B2 and B0→B2 require each loaded burst512 metric (p95 emission-to-visible, p95 intended-to-visible and terminal drainage) to improve beyond observed noise in the median AND at least4/5 individual pairs. Every other metric stays within its observed A/A noise. Paced first-visible and maximum observed inter-chunk gap, Stop through acknowledgement, completion drainage and all correctness/cleanup gates remain required. Inter-chunk gaps derive after timing from distinct existing2RAF timestamps, with count/median/max observed gap; no small-sample p99 claim.
|
||||
|
||||
Measure CPU, memory, actual normalized queue depth, SQL attempts/transactions, postcommit identities, event-loop delay/utilization and app-only CPU profiles in separate, explicitly ineligible diagnostic runs after actual compiled hook qualification. Keep exact hook hashes and normal H4 assembly. Server Node measurements do not represent provider/daemon fleet CPU or total physical memory; sum PG RSS double-counts shared pages. Short Stop resource ratios are omitted. Other per-case diagnostic windows require adequate interior samples and disclosed bracket uncertainty. Diagnostic instrumentation and its overhead never qualify performance acceptance.
|
||||
|
||||
A performance, correctness, timeout or cleanup failure stops the affected sequence for diagnosis. Thresholds, workload, counts, rates and timeouts are not loosened after results. Any affected source/runtime/fixture change requires a prospective new provenance record and requalification. Documentation/evidence-only commits use recorded source/build-input equivalence and current-head checks rather than forging tested build stamps. Local measurements establish only local improvement; deployed improvement still requires exact serving revision and matched deployed windows.
|
||||
|
||||
## Prospective H4c setup correction, before any accepted application A/A
|
||||
|
||||
The retained H4b full B0 qualification failed before a measured epoch:100 clock hellos but99 still-open provider GETs. Foreground GET1793752733222 began3.485s before the shared300s readiness window; the last hello1793753035430 arrived302.208s after that foreground GET. The persisted provider timeout error occurred300399.923ms after that GET, and there were zero measured emissions. Report SHA f11de64e34a290e8fff960af2d1624e0eae04f5bbb7ea2043c9b164c48559f21 and sealed abort forensics6bf60a72d80c6881538896f57b2894b7c64a107b766d82d310b20533ad9bb448 remain excluded qualification evidence. Cleanup completed with no remaining owned processes.
|
||||
|
||||
H4c admits background tasks in waves of TWO normal PATCHs, awaiting both actual open GETs and original clock hellos before the next wave. Promise.allSettled finishes both current activations before throwing a failure and never starts a later wave after failure. The common300s deadline, individual provider300s gate wait,50ms readiness observation, measured workload/rates/payloads/source clocks/2RAF/Stop and all acceptance budgets/gates are unchanged. All100 GETs must still be open after the closed read-only authority snapshot and before the common epoch. The strict comparator binds all per-task activation/wave timing to actual run/provider clock identities and rejects old admission/noise reports. H4c first qualifies its changed setup with3x16 remote events plus the original local Stop. No whole application build or recovery policy changes. Root and independent static review cleared the final patch; final5 gate tests, target types, syntax and test listing pass. Only after actual small and full functional PASS plus root review may this new frozen driver produce a fresh application A/A.
|
||||
@@ -0,0 +1,102 @@
|
||||
# Frozen H4c driver handoff — runtime qualification pending
|
||||
|
||||
- Commit: `7d7e7a88d562a95a1d407b76474763b666c181f5`
|
||||
- Branch: `codex/native-ingestion-harness-h4c`
|
||||
- Immutable annotated tag: `native-ingestion-h4c`
|
||||
- Tag object: `96b4959d09e9ffd0e0521d17e9f970b63ec30b73`
|
||||
- Protocol SHA256: `5db4129bbdc1fa38775121aff5d0e68a8a11762519e41cfdf36eacd90ba7256b`
|
||||
- Comparator SHA256: `01e0de114c886c255aaa7062a63f3e48788d8a83ce50921d672b2d661baef43b`
|
||||
- Driver lock SHA256: `54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8`
|
||||
- Approved patch SHA256: `79cacbb7a9ed9b6026d010ffd61c4768c60bd150a54ab4b57da8806829d9d14c`
|
||||
|
||||
Root and burst independently reviewed the exact final source. Final gate suite
|
||||
passed 5/5 (10.78s), targeted noEmit types, Playwright listing, syntax and diff
|
||||
checks pass. These establish source qualification only. No H4c app or VM run
|
||||
was executed by this fixture owner. Old H3/H4/H4a/H4b refs and failed raw evidence
|
||||
remain unchanged; remote verification confirms prior tag targets.
|
||||
|
||||
Only the performance owner may stage/run at the NEXT released quiet boundary
|
||||
after the current source B1 A/A has settled and quiet checks pass. Root release:
|
||||
run the small qualification below; IF AND ONLY IF it passes, run ONE full B0
|
||||
functional qualification. No automatic retry. No accepted browser A/A or A/B
|
||||
starts until root reviews the full functional PASS. Do not put these qualification
|
||||
runs into accepted performance manifests.
|
||||
|
||||
Prepare a NEW driver checkout at the exact immutable pin, with its OWN locked
|
||||
dependencies. No app source, node_modules or compiled output sharing:
|
||||
|
||||
```sh
|
||||
git fetch origin refs/tags/native-ingestion-h4c:refs/tags/native-ingestion-h4c
|
||||
git worktree add --detach /home/ubuntu/native-ingestion/harness-h4c 7d7e7a88d562a95a1d407b76474763b666c181f5
|
||||
cd /home/ubuntu/native-ingestion/harness-h4c
|
||||
pnpm install --frozen-lockfile --offline --ignore-scripts
|
||||
```
|
||||
|
||||
The driver needs no app build. Select APP_ROOT only after the owner qualifies its
|
||||
independent locked production build and exact server/dist/build-info.json stamp.
|
||||
Use new evidence/home/data paths and an independently checked free app port.
|
||||
|
||||
Small functional qualification is **remote 3x16 events at 25ms**, one foreground
|
||||
plus one real two-background activation wave, then the unchanged one-run LOCAL
|
||||
Stop workload (512 events at 100ms, stopped after at least 3 visible markers).
|
||||
This is qualificationOnly and cannot qualify performance:
|
||||
|
||||
```sh
|
||||
env -u DATABASE_URL -u PAPERCLIP_RUNNER_STATE_DIR \
|
||||
-u PAPERCLIP_INGESTION_SERVER_COMMAND -u PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_DIR \
|
||||
-u PAPERCLIP_INGESTION_BROWSER_CAPTURE \
|
||||
PAPERCLIP_INGESTION_TRANSPORT=sandbox-remote-with-local-stop \
|
||||
PAPERCLIP_INGESTION_SMOKE=1 \
|
||||
PAPERCLIP_INGESTION_APP_ROOT="$APP_ROOT" \
|
||||
PAPERCLIP_INGESTION_EVIDENCE_DIR="$NEW_EVIDENCE_DIR" \
|
||||
PAPERCLIP_E2E_PORT="$OWNED_FREE_PORT" \
|
||||
PAPERCLIP_RUNNER_BINARY="$APP_ROOT/packages/paperclip-runner/runner/target/release/paperclip-runnerd" \
|
||||
PAPERCLIP_RUNNER_REMOTE_BINARY_PATH="$APP_ROOT/packages/paperclip-runner/runner/target/release/paperclip-runnerd" \
|
||||
PAPERCLIP_INGESTION_VARIANT=H4c-small-functional-only \
|
||||
PAPERCLIP_INGESTION_DRIVER_DEPENDENCIES='Independent H4c locked offline --ignore-scripts install; no app dependency/source/build sharing' \
|
||||
node node_modules/@playwright/test/cli.js test \
|
||||
--config tests/e2e/playwright-native-ingestion.config.ts
|
||||
```
|
||||
|
||||
The normal configured server command runs native-ingestion-bootstrap.mjs for
|
||||
configuration only, then fixtures/native-ingestion-remote-server.mjs with
|
||||
--compiled-entry=server/dist/index.js. Preserve that normal verified-TLS assembly;
|
||||
do not replace it with direct server startup. Normal matching local/remote daemon
|
||||
paths are selected above. Remote provisioning is the first-party fake plugin's
|
||||
separate OS processes on the SAME physical host; Runner/WSS/DB/browser are real.
|
||||
Fake-plugin void cleanup does not grant remote Stop authority. Real Daytona Stop
|
||||
remains an additional separate acceptance gate.
|
||||
|
||||
If the small run passes, use the same normal command with a new evidence path and
|
||||
qualified B0 APP_ROOT, remove PAPERCLIP_INGESTION_SMOKE=1, add
|
||||
`-u PAPERCLIP_INGESTION_SMOKE` to env, and use variant
|
||||
`H4c-full-B0-functional-only`. The six FULL measured fixtures are unchanged:
|
||||
paced1x128@25ms, burst1x16, burst10x128, burst100x512, retained1000+128 across10,
|
||||
and LOCAL Stop1x512@100ms. Root must inspect the complete full PASS before releasing
|
||||
fresh accepted A/A. Qualification is not a performance improvement claim.
|
||||
|
||||
H4c setup contract:
|
||||
|
||||
- Background activation uses waves of TWO normal PATCH requests, each awaiting its
|
||||
actual provider GET and clock hello with 50ms observation. The SAME shared
|
||||
300-second deadline starts before first background activation. Foreground UI
|
||||
activation remains unchanged and its earlier start is now recorded.
|
||||
- Promise.allSettled drains both wave promises before throwing a failure; later
|
||||
waves do not start after failure. No retry or budget extension.
|
||||
- All actual GETs/hellos, plus the closed READ ONLY REPEATABLE READ authority
|
||||
snapshot, precede common epoch creation/release. GET count is rechecked after
|
||||
the snapshot. The provider's AbortSignal.timeout(300000) is UNCHANGED.
|
||||
- Reports use admissionProtocol
|
||||
`two-at-a-time-provider-and-clock-readiness-before-common-epoch` and
|
||||
setupActivationConcurrency2 at top/case level. Per-case activations record each
|
||||
index/issue/token/wave, request/readiness/settlement timestamps and status;
|
||||
activationWaves record indices/start/end/statuses. Comparator strictly requires
|
||||
that exact policy and matching actual clock/authority observations.
|
||||
- Payloads/counts/rates/original emission clocks/2RAF/control/durable checks and all
|
||||
noise/4-of-5 gates remain unchanged. Budgets stay 300/900 seconds for setup/whole
|
||||
run and 120/180/30/30 seconds for stream/terminal/Stop/lease verification.
|
||||
|
||||
Previous H4b full setup failure is retained: 100 clock hellos, 99 pending GETs,
|
||||
no epoch and zero measured emissions; foreground GET began 3.485s before the shared
|
||||
readiness clock, and the final hello arrived 302.208s after that GET. H4c addresses
|
||||
that unmeasured setup admission path; fresh runtime evidence is still required.
|
||||
@@ -0,0 +1,12 @@
|
||||
# Source-only evidence index
|
||||
|
||||
This bundle contains the fresh interleaved B0 A/A and B0→B1 real-PostgreSQL append-service comparison. See [the report](../../2026-11-03-native-event-ingestion-interleaved.md) for commands, results, gates and limitations. It does not establish application/browser/batching or production improvement, and the full task remains incomplete.
|
||||
|
||||
- `measurement-source.ts`, `prospective-protocol.md` and `gates.mjs` preserve the exact prospective measurement, protocol and unchanged acceptance logic. The canonical measurement also lives at `scripts/benchmark-native-event-ingestion-interleaved.ts`.
|
||||
- `service-*.json.gz` and `.blocks.jsonl.gz` restore exact original bytes with `gzip -dc`. Gzip metadata is deterministic (mtime zero). The command, repeatability, release and quiet receipts are unchanged originals.
|
||||
- `summary.json` and `cursor-result-table.json` are compact derived views. Acceptance uses the raw samples and unchanged gates. Twelve cases have five samples per arm; late-gap is structural-only. Mixed replay is 256 new and 256 exact duplicate events per 512 attempts.
|
||||
- `excluded-monolithic-service-aa-b0.json.gz` is a complete earlier failed A/A, explicitly excluded from calibration. A passing new interleaving does not establish its failure cause.
|
||||
- The selected source/table and static locked dependency-graph receipts support attribution only within their named scope. `compare-runtime-dependencies.cjs` is the original static helper with explicit repository/ref/output arguments and role labels; its projection record preserves both hashes. No whole-application or whole-closure equivalence is claimed.
|
||||
- `manifest.json` records every packaged file, plus the two owning files outside this directory. Compressed entries also record exact decompressed hashes and sizes. No raw measurement projection/redaction was required. Original absolute Linux paths are task-owned disposable fixture locations, not reproduction defaults.
|
||||
|
||||
Packaging checks parsed every JSON/JSONL file, recomputed all twelve case medians and 120 sample event counts, reapplied the unchanged A/A gates (new PASS, excluded old FAIL), verified exact source/protocol/gate hashes and compared compressed payloads with their originals. Source/helper syntax checks launch no application or database. All packaged/decompressed content was scanned for the actual approved qualification credential and credential/account/private-user-path patterns: zero matches. This scan is an evidence hygiene check, not a new performance measurement.
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
const {execFileSync}=require('node:child_process');
|
||||
const {createHash}=require('node:crypto');
|
||||
const fs=require('node:fs');
|
||||
const path=require('node:path');
|
||||
const cwd=process.argv[2];
|
||||
if (!cwd || !path.isAbsolute(cwd)) throw new Error('Pass an absolute repository checkout');
|
||||
const YAML=require(path.join(cwd,'node_modules/.pnpm/yaml@2.9.1/node_modules/yaml/dist/index.js'));
|
||||
const refs={before:process.argv[3],after:process.argv[4]};
|
||||
if (!Object.values(refs).every(value => /^[a-f0-9]{40}$/.test(value))) throw new Error('Pass two immutable full commit IDs');
|
||||
const hash=v=>createHash('sha256').update(typeof v==='string'?v:JSON.stringify(v)).digest('hex');
|
||||
const get=(ref,path)=>execFileSync('git',['show',ref+':'+path],{cwd,encoding:'utf8',maxBuffer:20*1024*1024});
|
||||
const docs=Object.fromEntries(Object.entries(refs).map(([key,ref])=>[key,YAML.parse(get(ref,'pnpm-lock.yaml'))]));
|
||||
function resolve(d,name,version){let key=name+'@'+version;if(d.snapshots[key])return key;key=key.replace(/\(patch_hash=[^)]+\)/g,'');if(d.snapshots[key])return key;throw new Error('Unresolved '+key);}
|
||||
function graph(d,roots){const out={};function add(key){if(out[key])return;const snapshot=d.snapshots[key];const base=key.replace(/\(.*/,'');out[key]={snapshot,package:d.packages[base]??null};for(const [name,version] of Object.entries({...snapshot.dependencies,...snapshot.optionalDependencies})){if(version.startsWith('link:'))continue;add(resolve(d,name,version));}}roots.forEach(k=>add(k));return out;}
|
||||
const details={};for(const [label,d]of Object.entries(docs)){
|
||||
const imports={};for(const importer of ['.','packages/db','server']){const i=d.importers[importer];imports[importer]={};for(const cat of ['dependencies','devDependencies','optionalDependencies'])for(const name of ['drizzle-orm','postgres','pg','embedded-postgres','tsx','drizzle-kit'])if(i?.[cat]?.[name])imports[importer][name]={category:cat,...i[cat][name]};}
|
||||
const db=d.importers['packages/db'];const roots=['drizzle-orm','postgres','embedded-postgres'].map(n=>resolve(d,n,db.dependencies[n].version));const tsx=resolve(d,'tsx',db.devDependencies.tsx.version);const pg=Object.keys(d.snapshots).find(x=>/^pg@8\.18\.0\(/.test(x))??'pg@8.18.0';
|
||||
const runtime=graph(d,roots);const launcher=graph(d,[tsx]);const pgGraph=graph(d,[pg]);
|
||||
const parents=Object.entries(d.snapshots).filter(([_,s])=>s.dependencies?.['get-tsconfig']||s.optionalDependencies?.['get-tsconfig']).map(([key,s])=>({key,version:s.dependencies?.['get-tsconfig']??s.optionalDependencies['get-tsconfig']}));
|
||||
details[label]={imports,roots,tsx,pg,runtime,launcher,pgGraph,getTsconfigParents:parents};
|
||||
}
|
||||
function diff(a,b){return {removed:Object.keys(a).filter(k=>!b[k]),added:Object.keys(b).filter(k=>!a[k]),changed:Object.keys(a).filter(k=>b[k]&&JSON.stringify(a[k])!==JSON.stringify(b[k]))};}
|
||||
const comparisons={};for(const name of ['runtime','launcher','pgGraph'])comparisons[name]={nodeCounts:[Object.keys(details.before[name]).length,Object.keys(details.after[name]).length],...diff(details.before[name],details.after[name])};
|
||||
const sourcePaths=['packages/db/src/run-event-payload.ts','server/src/services/native-runtime/canonical.ts','packages/db/src/client.ts','patches/postgres@3.4.9.patch','patches/embedded-postgres@18.1.0-beta.16.patch'];const sources={};for(const p of sourcePaths){try{sources[p]=Object.fromEntries(Object.entries(refs).map(([k,v])=>[k,hash(get(v,p))]));}catch(e){sources[p]={error:String(e)}}}
|
||||
const toolingEdges=Object.fromEntries(Object.entries(docs).map(([label,d])=>[label,Object.entries(d.snapshots).filter(([_,x])=>x.dependencies?.['@esbuild-kit/esm-loader']).map(([key,x])=>({key,esmLoader:x.dependencies['@esbuild-kit/esm-loader']}))]));
|
||||
const result={refs,comparisons,details,toolingEdges,sources,patchConfigEqual:get(refs.before,'package.json')===get(refs.after,'package.json'),limits:'Static locked dependency-graph and immutable source comparison; installed bits, OS and Node versions remain experiment provenance requirements.'};
|
||||
if (!process.argv[5]) throw new Error('Pass a new output JSON path');
|
||||
fs.writeFileSync(process.argv[5],JSON.stringify(result,null,2)+'\n',{flag:'wx'});
|
||||
console.log(JSON.stringify({comparisons,toolingEdges,imports:details.after.imports,getTsconfigParents:Object.fromEntries(Object.entries(details).map(([k,v])=>[k,v.getTsconfigParents])),sources,patchConfigEqual:result.patchConfigEqual},null,2));
|
||||
+134
@@ -0,0 +1,134 @@
|
||||
[
|
||||
{
|
||||
"history": 1000,
|
||||
"scenario": "ordered",
|
||||
"baselineMedianActiveMs": 3823.2426909999995,
|
||||
"cursorMedianActiveMs": 3554.4297090000036,
|
||||
"reductionPercent": 7.031020621128439,
|
||||
"pairedBeyondNoise": 5,
|
||||
"baselineNoisePercent": 1.4079047972212446,
|
||||
"gate": "nonregression",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 1000,
|
||||
"scenario": "replay",
|
||||
"baselineMedianActiveMs": 3164.471254999982,
|
||||
"cursorMedianActiveMs": 3002.7086030000064,
|
||||
"reductionPercent": 5.11183824926154,
|
||||
"pairedBeyondNoise": 3,
|
||||
"baselineNoisePercent": 3.9822120206785225,
|
||||
"gate": "nonregression",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 1000,
|
||||
"scenario": "late-gap",
|
||||
"baselineMedianActiveMs": 7.402558999994653,
|
||||
"cursorMedianActiveMs": 9.506674999996903,
|
||||
"reductionPercent": -28.424170614564098,
|
||||
"pairedBeyondNoise": 0,
|
||||
"baselineNoisePercent": 15.945874769827368,
|
||||
"gate": "structural-only",
|
||||
"passed": null
|
||||
},
|
||||
{
|
||||
"history": 1000,
|
||||
"scenario": "warm-suffix",
|
||||
"baselineMedianActiveMs": 4152.802396999992,
|
||||
"cursorMedianActiveMs": 3542.40763099998,
|
||||
"reductionPercent": 14.698382144090575,
|
||||
"pairedBeyondNoise": 5,
|
||||
"baselineNoisePercent": 1.8842439321434798,
|
||||
"gate": "nonregression",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 10000,
|
||||
"scenario": "ordered",
|
||||
"baselineMedianActiveMs": 4209.649118000001,
|
||||
"cursorMedianActiveMs": 3530.6375980000594,
|
||||
"reductionPercent": 16.129884010915838,
|
||||
"pairedBeyondNoise": 5,
|
||||
"baselineNoisePercent": 1.7474622955774097,
|
||||
"gate": "improvement",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 10000,
|
||||
"scenario": "replay",
|
||||
"baselineMedianActiveMs": 3575.641203999985,
|
||||
"cursorMedianActiveMs": 3016.318643000035,
|
||||
"reductionPercent": 15.6425807034064,
|
||||
"pairedBeyondNoise": 5,
|
||||
"baselineNoisePercent": 1.6304743570125815,
|
||||
"gate": "improvement",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 10000,
|
||||
"scenario": "late-gap",
|
||||
"baselineMedianActiveMs": 7.701110000023618,
|
||||
"cursorMedianActiveMs": 8.573146000009729,
|
||||
"reductionPercent": -11.323510506711836,
|
||||
"pairedBeyondNoise": 0,
|
||||
"baselineNoisePercent": 23.08985200646279,
|
||||
"gate": "structural-only",
|
||||
"passed": null
|
||||
},
|
||||
{
|
||||
"history": 10000,
|
||||
"scenario": "warm-suffix",
|
||||
"baselineMedianActiveMs": 4614.03645500011,
|
||||
"cursorMedianActiveMs": 3592.809363000095,
|
||||
"reductionPercent": 22.133052089203552,
|
||||
"pairedBeyondNoise": 5,
|
||||
"baselineNoisePercent": 11.830110496052082,
|
||||
"gate": "improvement",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 100000,
|
||||
"scenario": "ordered",
|
||||
"baselineMedianActiveMs": 8029.154580999981,
|
||||
"cursorMedianActiveMs": 3540.048427999951,
|
||||
"reductionPercent": 55.91007257006802,
|
||||
"pairedBeyondNoise": 5,
|
||||
"baselineNoisePercent": 1.0693214938958222,
|
||||
"gate": "improvement",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 100000,
|
||||
"scenario": "replay",
|
||||
"baselineMedianActiveMs": 7456.020750999975,
|
||||
"cursorMedianActiveMs": 2989.0107820000267,
|
||||
"reductionPercent": 59.911447649885474,
|
||||
"pairedBeyondNoise": 5,
|
||||
"baselineNoisePercent": 0.9332642395176317,
|
||||
"gate": "improvement",
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 100000,
|
||||
"scenario": "late-gap",
|
||||
"baselineMedianActiveMs": 24.45493699994404,
|
||||
"cursorMedianActiveMs": 9.347156999981962,
|
||||
"reductionPercent": 61.77803688472658,
|
||||
"pairedBeyondNoise": 5,
|
||||
"baselineNoisePercent": 39.768871890266304,
|
||||
"gate": "structural-only",
|
||||
"passed": null
|
||||
},
|
||||
{
|
||||
"history": 100000,
|
||||
"scenario": "warm-suffix",
|
||||
"baselineMedianActiveMs": 8347.43800100009,
|
||||
"cursorMedianActiveMs": 3402.361554999836,
|
||||
"reductionPercent": 59.240648992035574,
|
||||
"pairedBeyondNoise": 5,
|
||||
"baselineNoisePercent": 0.7615089382403567,
|
||||
"gate": "improvement",
|
||||
"passed": true
|
||||
}
|
||||
]
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"originalSha256": "3225b891fa2d7f76408037100dc65a17f6fcd25ab9b070c419095ef2994f0ad9",
|
||||
"publicSha256": "c3f8207cc2199be4eec09b1b4cb08781ff6658e05415b304d75b41aa961d1c3a",
|
||||
"adaptation": "Repository, immutable before/after refs and exclusive output are arguments; role labels renamed before/after; static graph comparison logic unchanged. Read-only Git, no benchmark/app/provider work."
|
||||
}
|
||||
BIN
Binary file not shown.
@@ -0,0 +1,15 @@
|
||||
# Lead release of the full interleaved source experiment
|
||||
|
||||
Recorded 2026-11-03 after the lead independently reviewed the complete quick report and append-only trace. This file records an existing explicit lead instruction; it does not grant a new permission by inference.
|
||||
|
||||
The lead verified quick report b4eac4e185d6afa569794a17831851efd99b0f99a75a089f706072d35b3a2592 and trace8aa0701e8a84e256fc307a17e57cd36c4b02a44f7db1dd37dbb492900346731b:5pairs,80completed64-eventblocks,2048warmupattempts,5120measuredattempts, matched per-block retained histories,10finalverifications and complete cleanup. The1kordered qualification maximum paired difference4.976572685859885% and median arm drift0.416295124480865% passed the unchanged subset limits. Quick remains ineligible for full repeatability or optimization acceptance.
|
||||
|
||||
Exact frozen applications remain B0=5337df423e622755d44f3ece5447d1e0beef5f40, B1=e9b909015ad3d517f28c45f0c66c8f41d6112b4b, B2=fcb5d94848b3c4b0b32096ff5e023618e0f0637b. Their normal production builds/stamps/independent dependencies and source guards passed. Later public doc/evidence-only heads do not replace these measured application identities.
|
||||
|
||||
Measurement SHA c8adaf5ac16cab67513d6c8b888afa44d95fa3688d4831b8126fe3d997f1d039; prospective protocol SHA302c11ec61d5170822c08e47f798ca582b8e5afefd9d223fb9023d9ead7a2ad3; FULL workload SHAca4894b0723d0bffff0fc88d1596d64aa8fd75aa1dd8d861372ea993db24e164; unchanged gate SHAded8bcbb8f05732348f6520b9c5e41b7c35bc098997eb263a9dcb174eede75fc. Original48e monolithic evidence remains preserved and cannot supply new noise.
|
||||
|
||||
Released automation: native-ingestion-run-interleaved-service.mjs SHAecf9adcf9fef440fc10c049199dabb8ece59f6811bef526f2c775435a1c62f8b; detached launcher SHA152d477f2b3bbe4a29c12feca38b0a0c52bc2ca6488027e2724f006042bc3abd.
|
||||
|
||||
The lead explicitly released these5stages sequentially on the exclusive quiet VM: freshB0A/A; onlyifPASS B0→B1; freshB1A/A; onlyifPASS B1→B2; B0→B2. All9timingcases must pass maximum paired relative difference<=20% and median arm drift<=10%; candidate median and at least4of5 INDIVIDUAL paired improvements must exceed the observed corresponding noise. No threshold change, outlier exclusion, old noise, competing load/profile/build/transfer, or unfinished cleanup is allowed. Any failed gate or cleanup stops the series for diagnosis.
|
||||
|
||||
The source estimand is per-arm active append cost summed across8interleaved64-attempt blocks. Whole pair, commit, independent drain, bookkeeping and all block/failure evidence remain recorded. It does not establish application burst completion/browser latency. H4b normal functional qualification and subsequent browser A/A/comparisons remain separately gated. The independent real-cloud Stop journey is a separate functional qualification.
|
||||
@@ -0,0 +1,51 @@
|
||||
// Shared by the real owner-path harness and compiled-browser comparator.
|
||||
export const SERVICE_REPEATABILITY_LIMITS = Object.freeze({
|
||||
maximumPairedRelativeDifference: 0.20,
|
||||
maximumMedianArmDrift: 0.10,
|
||||
});
|
||||
|
||||
const median = values => [...values].sort((a, b) => a - b)[Math.floor(values.length / 2)];
|
||||
|
||||
export function noiseExceedingPairedWins(improvements, noiseEnvelopeRelative) {
|
||||
if (!Number.isFinite(noiseEnvelopeRelative) || noiseEnvelopeRelative < 0
|
||||
|| improvements.length < 5 || !improvements.every(Number.isFinite)) {
|
||||
throw new Error("Finite noise and at least five paired improvements are required");
|
||||
}
|
||||
const requiredPairedWins = Math.ceil(improvements.length * 0.8);
|
||||
const count = improvements.filter(value => value > noiseEnvelopeRelative).length;
|
||||
return { noiseExceedingPairedWins: count, requiredPairedWins,
|
||||
pairedWinRequirementPassed: count >= requiredPairedWins };
|
||||
}
|
||||
|
||||
export function evaluateServiceRepeatability(report) {
|
||||
const failures = [];
|
||||
if (report.phase !== "aa") failures.push("Only A/A calibrates repeatability");
|
||||
if (report.incomplete !== false || report.cleanupComplete !== true) failures.push("Evidence and resource cleanup must be complete");
|
||||
const workload = report.workload ?? {};
|
||||
if (workload.quick !== false) failures.push("Smoke cannot qualify repeatability");
|
||||
if (!Number.isInteger(workload.pairs) || workload.pairs < 5) failures.push("At least five alternating pairs are required");
|
||||
if (workload.events !== 32 || workload.groups !== 16) failures.push("The frozen 16 groups of 32 events are required");
|
||||
const histories = [1000, 10000, 100000], scenarios = ["ordered", "replay", "late-gap", "warm-suffix"];
|
||||
if (!histories.every(value => workload.histories?.includes(value))
|
||||
|| !scenarios.every(value => workload.scenarios?.includes(value))) failures.push("The full fixed workload matrix is required");
|
||||
const cases = [];
|
||||
for (const history of histories) for (const scenario of scenarios) {
|
||||
const rows = (report.results ?? []).filter(item => item.history === history && item.scenario === scenario);
|
||||
const row = rows[0], baseline = row?.baseline?.samples, repeat = row?.candidate?.samples;
|
||||
if (rows.length !== 1 || !Array.isArray(baseline) || !Array.isArray(repeat)
|
||||
|| baseline.length !== workload.pairs || repeat.length !== workload.pairs
|
||||
|| ![...baseline, ...repeat].every(item => Number.isFinite(item.elapsedMs) && item.elapsedMs > 0)) {
|
||||
failures.push(`Missing or invalid paired samples: ${history}/${scenario}`); continue;
|
||||
}
|
||||
if (scenario === "late-gap") continue;
|
||||
const maximumPairedRelativeDifference = Math.max(...baseline.map((sample, index) => Math.abs(repeat[index].elapsedMs - sample.elapsedMs) / sample.elapsedMs));
|
||||
const baselineMedian = median(baseline.map(item => item.elapsedMs));
|
||||
const medianArmDrift = Math.abs(median(repeat.map(item => item.elapsedMs)) - baselineMedian) / baselineMedian;
|
||||
cases.push({ history, scenario, maximumPairedRelativeDifference, medianArmDrift,
|
||||
passed: maximumPairedRelativeDifference <= SERVICE_REPEATABILITY_LIMITS.maximumPairedRelativeDifference
|
||||
&& medianArmDrift <= SERVICE_REPEATABILITY_LIMITS.maximumMedianArmDrift });
|
||||
}
|
||||
return { limits: SERVICE_REPEATABILITY_LIMITS, eligible: failures.length === 0,
|
||||
eligibilityFailures: failures, cases, passed: failures.length === 0 && cases.length === 9 && cases.every(item => item.passed),
|
||||
excluded: "Late-gap single-operation latency is structural-only; its complete paired evidence is still required" };
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
{
|
||||
"schema": "paperclip.native-ingestion.source-evidence.v1",
|
||||
"scope": "Frozen SOURCE B0 A/A and B0\u2192B1 cursor active append-service cost only; no browser/batching/production/full-task completion claim",
|
||||
"sourceRefs": {
|
||||
"B0": "5337df423e622755d44f3ece5447d1e0beef5f40",
|
||||
"B1": "e9b909015ad3d517f28c45f0c66c8f41d6112b4b",
|
||||
"B2": "fcb5d94848b3c4b0b32096ff5e023618e0f0637b"
|
||||
},
|
||||
"separateAppRefs": {
|
||||
"B0": "a36567f9fbf805ad72207d54f539516e3121741e",
|
||||
"B1": "d8cd3be0aeed4f9fb93ecc3cf725f1d46d5a0310",
|
||||
"B2": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da"
|
||||
},
|
||||
"protocolSha256": "302c11ec61d5170822c08e47f798ca582b8e5afefd9d223fb9023d9ead7a2ad3",
|
||||
"measurementSha256": "c8adaf5ac16cab67513d6c8b888afa44d95fa3688d4831b8126fe3d997f1d039",
|
||||
"workloadSha256": "ca4894b0723d0bffff0fc88d1596d64aa8fd75aa1dd8d861372ea993db24e164",
|
||||
"gateSha256": "ded8bcbb8f05732348f6520b9c5e41b7c35bc098997eb263a9dcb174eede75fc",
|
||||
"rawProjection": "None. Exact original decompressed bytes retained; historical Linux paths are disposable owned fixture locations. Adapted static helper declares its original/public hashes separately.",
|
||||
"excludedPriorAaSha256": "cfcfeb3c12812dbe48c2afe6013320922d5670a410d033d6ad9a3a2f91236ef8",
|
||||
"files": [
|
||||
{
|
||||
"path": "README.md",
|
||||
"bytes": 2545,
|
||||
"sha256": "1f70726000fea330bc3251b20c43cfde80aae255110320be8fb5d301a62a9c90"
|
||||
},
|
||||
{
|
||||
"path": "compare-runtime-dependencies.cjs",
|
||||
"bytes": 4390,
|
||||
"sha256": "c3f8207cc2199be4eec09b1b4cb08781ff6658e05415b304d75b41aa961d1c3a"
|
||||
},
|
||||
{
|
||||
"path": "cursor-result-table.json",
|
||||
"bytes": 3869,
|
||||
"sha256": "026fc500c3c56fc2e84fb46ca43eed8b54f8647fa3d6054a48e4513dbec1bdab"
|
||||
},
|
||||
{
|
||||
"path": "dependency-helper-projection.json",
|
||||
"bytes": 394,
|
||||
"sha256": "39bdb57f8ce001f82daf9c41bb91acfdfa344fcc61f1b11fed2f90a466b195da"
|
||||
},
|
||||
{
|
||||
"path": "excluded-monolithic-service-aa-b0.json.gz",
|
||||
"bytes": 160758,
|
||||
"sha256": "dd1508b0fece27dade320fcddeb4bf0e81ecef7c3dffd01b7ee110dbd9949258",
|
||||
"decompressedBytes": 3175715,
|
||||
"decompressedSha256": "cfcfeb3c12812dbe48c2afe6013320922d5670a410d033d6ad9a3a2f91236ef8"
|
||||
},
|
||||
{
|
||||
"path": "full-release.md",
|
||||
"bytes": 2794,
|
||||
"sha256": "904434a688bbfc0b1de0673d06941881a28f6ec81b8b6153d733b8ab58a4803e"
|
||||
},
|
||||
{
|
||||
"path": "gates.mjs",
|
||||
"bytes": 3553,
|
||||
"sha256": "ded8bcbb8f05732348f6520b9c5e41b7c35bc098997eb263a9dcb174eede75fc"
|
||||
},
|
||||
{
|
||||
"path": "measurement-source.ts",
|
||||
"bytes": 51618,
|
||||
"sha256": "c8adaf5ac16cab67513d6c8b888afa44d95fa3688d4831b8126fe3d997f1d039"
|
||||
},
|
||||
{
|
||||
"path": "packaging-verification.json",
|
||||
"bytes": 1176,
|
||||
"sha256": "143fb11e55a78977d4d6bced8a236e98ed4b46e855889f7c94e4b0736d8cee86"
|
||||
},
|
||||
{
|
||||
"path": "prospective-protocol.md",
|
||||
"bytes": 13425,
|
||||
"sha256": "302c11ec61d5170822c08e47f798ca582b8e5afefd9d223fb9023d9ead7a2ad3"
|
||||
},
|
||||
{
|
||||
"path": "root-release.json",
|
||||
"bytes": 920,
|
||||
"sha256": "85477defaf8e9b26b25b95fc15b3eb3124e6b7b809d76a40cce86b83c776feae"
|
||||
},
|
||||
{
|
||||
"path": "service-aa-b0-commands.json",
|
||||
"bytes": 949,
|
||||
"sha256": "fd94dbf1fa4875ab689712ff259bdae52fa6eed667664c9b5dc3c044a6dcf3a6"
|
||||
},
|
||||
{
|
||||
"path": "service-aa-b0.json.blocks.jsonl.gz",
|
||||
"bytes": 153031,
|
||||
"sha256": "799782ba8c0fe609567c22ed3ddbe831b9a561872ad4ea9b0ae9c28b24379331",
|
||||
"decompressedBytes": 1493345,
|
||||
"decompressedSha256": "744620da2dfff459f1fff9574837d11bb60a2f243923f526529b6a476d653148"
|
||||
},
|
||||
{
|
||||
"path": "service-aa-b0.json.gz",
|
||||
"bytes": 357150,
|
||||
"sha256": "05beeda33d75794169199d37f6d721f87474c9043dfb1bcd70b35398efc63a7b",
|
||||
"decompressedBytes": 4328699,
|
||||
"decompressedSha256": "1d88dea61820d5555af08dc1f6079f3820ccb0e3b429105951ad06eca426d08a"
|
||||
},
|
||||
{
|
||||
"path": "service-aa-b0.repeatability.json",
|
||||
"bytes": 2177,
|
||||
"sha256": "ebdae0417c10b81550e6791d2c246becae8fc7edeee31fa7a103f2142d7abef1"
|
||||
},
|
||||
{
|
||||
"path": "service-b0-b1-commands.json",
|
||||
"bytes": 1063,
|
||||
"sha256": "a10f64f755425852252bafd5fc2397dcc62565be45e2733f2e86a95ec50cb5e5"
|
||||
},
|
||||
{
|
||||
"path": "service-b0-b1.json.blocks.jsonl.gz",
|
||||
"bytes": 153649,
|
||||
"sha256": "6502f2f3e41363e584d52988090d7df16420d7a4c5a84f57abb0fdc8d8f50b98",
|
||||
"decompressedBytes": 1493184,
|
||||
"decompressedSha256": "382b0f7ca51cc855c0bf0d8b1685cb01cbf0dde1d55ecf4c90dc527c32db7f57"
|
||||
},
|
||||
{
|
||||
"path": "service-b0-b1.json.gz",
|
||||
"bytes": 375350,
|
||||
"sha256": "00f8e4a49a77aaa65417fcf5b328ccdcb50795675bd490332a8849f817b6df12",
|
||||
"decompressedBytes": 4776091,
|
||||
"decompressedSha256": "283eef0a9d5cf3b39bb7ebf7444ced40025f499e51847c2cc205f3101c1f1470"
|
||||
},
|
||||
{
|
||||
"path": "service-b0-b1.quiet.json",
|
||||
"bytes": 316,
|
||||
"sha256": "f50d903380fc1daff6c2d4c1b27311513b2caaaa2b770f8c444718dab4571fc6"
|
||||
},
|
||||
{
|
||||
"path": "source-app-core-equivalence.json",
|
||||
"bytes": 29118,
|
||||
"sha256": "2407b0f92eb70778d910ae8130d74cc64d9e47ba4aaed09d2ff54414e3b28e61"
|
||||
},
|
||||
{
|
||||
"path": "source-b1-app-b1-dependency-graph.json",
|
||||
"bytes": 82820,
|
||||
"sha256": "c6f8ad70c3cf774a0ebe29f71b5847b14430a34bcb3532ef2dd1d66816220552"
|
||||
},
|
||||
{
|
||||
"path": "source-b2-app-b2-dependency-graph.json",
|
||||
"bytes": 82820,
|
||||
"sha256": "6ba611d3331121146f2c7d744fea8f5fc00eff348cb2bdcb8d0975d68886f7b8"
|
||||
},
|
||||
{
|
||||
"path": "summary.json",
|
||||
"bytes": 14339,
|
||||
"sha256": "8c2a5c760175867a36fe4222b444fb0ce7234142d934699cbf0deeb4478db3b4"
|
||||
},
|
||||
{
|
||||
"path": "upstream-dependency-graph.json",
|
||||
"bytes": 82860,
|
||||
"sha256": "733f23dea1a6657fd75b9394a12451ddd7d78938dde9011ffa5b421092d8cdc9"
|
||||
},
|
||||
{
|
||||
"path": "upstream-table-snapshot-equality.json",
|
||||
"bytes": 1409,
|
||||
"sha256": "1f8eaede3ce8ec77b83909efc465d5adbdef97595fba8f0ee6c2e8c0e19b3774"
|
||||
}
|
||||
],
|
||||
"externalOwnedFiles": [
|
||||
{
|
||||
"repositoryPath": "doc/benchmarks/2026-11-03-native-event-ingestion-interleaved.md",
|
||||
"bytes": 13105,
|
||||
"sha256": "69f0b1895e1c72f9ddfaeb31c2f0152cf63fed077e780832aaeff687112f4e0d"
|
||||
},
|
||||
{
|
||||
"repositoryPath": "scripts/benchmark-native-event-ingestion-interleaved.ts",
|
||||
"bytes": 51618,
|
||||
"sha256": "c8adaf5ac16cab67513d6c8b888afa44d95fa3688d4831b8126fe3d997f1d039"
|
||||
}
|
||||
]
|
||||
}
|
||||
+703
@@ -0,0 +1,703 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { promises as fs } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import { availableParallelism, loadavg, tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { monitorEventLoopDelay, performance } from "node:perf_hooks";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import type { Db } from "../packages/db/src/index.js";
|
||||
import type { AppendHeartbeatRunEventInput, AppendHeartbeatRunEventResult } from "../server/src/services/heartbeat-run-events.js";
|
||||
|
||||
// The exact bytes of this file may be copied outside APP_ROOT as .mts. Only
|
||||
// erased type imports are relative to the measurement file; runtime support is
|
||||
// always owned by the explicitly selected, frozen application checkout.
|
||||
const arg = (name: string) => process.argv.find(value => value.startsWith(`--${name}=`))?.slice(name.length + 3);
|
||||
const measurementFile = fileURLToPath(import.meta.url);
|
||||
const actualSelfSha256 = createHash("sha256").update(await fs.readFile(measurementFile)).digest("hex");
|
||||
const root = path.resolve(arg("app-root") ?? fileURLToPath(new URL("../", import.meta.url)));
|
||||
const appImport = (file: string) => import(pathToFileURL(path.join(root, file)).href);
|
||||
const serverRequire = createRequire(path.join(root, "server/package.json"));
|
||||
const gatesPath = "scripts/native-ingestion-performance-gates.mjs";
|
||||
const frozenGatesSha256 = "ded8bcbb8f05732348f6520b9c5e41b7c35bc098997eb263a9dcb174eede75fc";
|
||||
const gatesSha256 = createHash("sha256").update(await fs.readFile(path.join(root, gatesPath))).digest("hex");
|
||||
assert.equal(gatesSha256, frozenGatesSha256, "The preregistered acceptance gates must remain unchanged");
|
||||
const { evaluateServiceRepeatability, noiseExceedingPairedWins } = await appImport(gatesPath);
|
||||
const sourcePath = "server/src/services/heartbeat-run-events.ts";
|
||||
const cursorSourcePath = "server/src/services/heartbeat/native-event-cursors.ts";
|
||||
const git = (args: string[]) => execFileSync("git", args, { cwd: root, encoding: "utf8" });
|
||||
const quick = process.argv.includes("--quick");
|
||||
const phase = arg("phase") ?? "compare";
|
||||
assert.ok(["aa", "compare", "batch"].includes(phase), "--phase must be aa, compare, or batch");
|
||||
if (process.argv.includes("--require-repeatability")) assert.equal(phase, "aa", "Repeatability requires the A/A phase");
|
||||
const baselineRef = arg("baseline-ref");
|
||||
assert.ok(baselineRef, "Pass a pinned --baseline-ref=<commit>, including report validation");
|
||||
const baselineRevision = git(["rev-parse", "--verify", `${baselineRef}^{commit}`]).trim();
|
||||
const histories = (arg("histories") ?? (quick ? "1000" : "1000,10000,100000")).split(",").map(Number);
|
||||
const pairs = Number(arg("pairs") ?? 5);
|
||||
const events = Number(arg("events") ?? 32);
|
||||
const groups = Number(arg("groups") ?? 16);
|
||||
assert.deepEqual(histories, quick ? [1000] : [1000, 10000, 100000], "Retained-history matrix is frozen");
|
||||
assert.equal(pairs, 5, "The prospective schedule requires five measured pairs");
|
||||
assert.equal(events, 32, "Generator groups retain 32 attempts");
|
||||
assert.equal(groups, 16, "Each non-late-gap sample retains 16 generator groups");
|
||||
const warmups = 2, blockAttempts = 64, blocksPerArm = 8, groupsPerBlock = blockAttempts / events;
|
||||
const scenarios = quick ? ["ordered"] : ["ordered", "replay", "late-gap", "warm-suffix"];
|
||||
const workload = { histories, pairs, events, groups, scenarios, payloadTextBytes: 128, warmSourceOffset: 1000,
|
||||
seed: "native-event-ingestion-v1", retainedHistory: "vacuum-analyzed", quick,
|
||||
schedule: "interleaved-8x64-v1", blockAttempts, blocksPerArm, groupsPerBlock, warmups,
|
||||
firstArm: "alternating-per-pair", estimand: "per-arm sum of active append intervals" };
|
||||
const workloadSha256 = createHash("sha256").update(JSON.stringify(workload)).digest("hex");
|
||||
|
||||
function assertProtocolIdentity(recorded: any) {
|
||||
assert.equal(recorded.actualSelfSha256, actualSelfSha256, "Measurement bytes changed; prior evidence cannot qualify this file");
|
||||
assert.equal(recorded.workloadSha256, workloadSha256, "Workload or schedule changed; rerun A/A");
|
||||
assert.equal(createHash("sha256").update(JSON.stringify(recorded.workload)).digest("hex"), workloadSha256, "Recorded workload does not match its hash");
|
||||
assert.equal(recorded.baselineRevision, baselineRevision, "Exact pinned baseline changed; rerun A/A");
|
||||
assert.equal(recorded.gatesSha256, gatesSha256, "Acceptance gates changed; rerun A/A");
|
||||
assert.equal(recorded.candidateSourceHashes?.[gatesPath], gatesSha256, "Recorded gate source must match APP_ROOT");
|
||||
assert.ok(recorded.candidateSourceHashes?.[sourcePath], "Report must hash the actual APP_ROOT append service");
|
||||
}
|
||||
function assertServiceAcceptance(recorded: any) {
|
||||
assertProtocolIdentity(recorded);
|
||||
assert.equal(recorded.cleanupComplete, true, "Owned benchmark resources have not finished cleanup");
|
||||
assert.ok(["compare", "batch"].includes(recorded.phase), "A/A calibration is not performance acceptance");
|
||||
assert.equal(recorded.workload?.quick, false, "Qualification smoke cannot establish acceptance");
|
||||
assert.ok(recorded.workload.pairs >= 5, "Acceptance requires at least five paired samples");
|
||||
assert.ok([1000, 10000, 100000].every(history => recorded.workload.histories.includes(history)), "Acceptance requires the complete retained-history matrix");
|
||||
assert.ok(["ordered", "replay", "late-gap", "warm-suffix"].every(scenario => recorded.workload.scenarios.includes(scenario)), "Acceptance requires every fixed service scenario");
|
||||
assert.ok(recorded.comparison?.length > 0, "Acceptance requires a same-workload A/A noise comparison");
|
||||
assert.ok(recorded.comparison.every((entry: { acceptancePassed: boolean | null }) => entry.acceptancePassed !== false), "Service performance gate failed; inspect comparison and retained failures");
|
||||
}
|
||||
if (arg("validate-report")) {
|
||||
const recorded = JSON.parse(await fs.readFile(arg("validate-report")!, "utf8"));
|
||||
assertProtocolIdentity(recorded);
|
||||
assert.equal(recorded.incomplete, false, "Incomplete evidence cannot qualify a revision");
|
||||
assert.ok(Object.keys(recorded.candidateSourceHashes ?? {}).length > 0, "Report must include supporting source hashes");
|
||||
for (const [file, expected] of Object.entries(recorded.candidateSourceHashes)) {
|
||||
assert.equal(createHash("sha256").update(await fs.readFile(path.join(root, file))).digest("hex"), expected, `Measured source changed: ${file}`);
|
||||
}
|
||||
if (process.argv.includes("--require-acceptance")) {
|
||||
assertServiceAcceptance(recorded);
|
||||
}
|
||||
if (process.argv.includes("--require-repeatability")) {
|
||||
assert.equal(evaluateServiceRepeatability(recorded).passed, true, "A/A repeatability gate failed; retain evidence and diagnose before candidate measurements");
|
||||
}
|
||||
console.log(JSON.stringify({ valid: true, phase: recorded.phase, baselineRevision: recorded.baselineRevision,
|
||||
candidateRevision: recorded.candidateRevision, scope: recorded.scope }));
|
||||
process.exit(0);
|
||||
}
|
||||
if (process.argv.includes("--check-source")) {
|
||||
// Source-only smoke: import the unchanged gates and resolve support files,
|
||||
// without importing the DB client, starting PostgreSQL or creating fixtures.
|
||||
for (const file of ["packages/db/src/index.ts", "packages/db/src/work-signals.ts", sourcePath,
|
||||
"server/src/services/native-runtime/canonical.ts", "server/src/__tests__/helpers/external-test-database.ts"]) {
|
||||
await fs.access(path.join(root, file));
|
||||
}
|
||||
for (const specifier of ["drizzle-orm/postgres-js", "@paperclipai/db", "drizzle-orm"]) serverRequire.resolve(specifier);
|
||||
console.log(JSON.stringify({ sourceOnly: true, appRoot: root, measurementFile, actualSelfSha256,
|
||||
workloadSha256, baselineRevision, gatesSha256, quick, performanceAcceptanceEligible: false }));
|
||||
process.exit(0);
|
||||
}
|
||||
const baselineSource = git(["show", `${baselineRevision}:${sourcePath}`]);
|
||||
const candidateRevision = git(["rev-parse", "HEAD"]).trim();
|
||||
const { agents, applyPendingMigrations, companies, createDb, heartbeatRuns, startEmbeddedPostgresTestDatabase } =
|
||||
await appImport("packages/db/src/index.ts") as typeof import("../packages/db/src/index.js");
|
||||
const { installDatabaseWorkSignals } = await appImport("packages/db/src/work-signals.ts") as typeof import("../packages/db/src/work-signals.js");
|
||||
const { resolveExternalTestDatabaseUrl } = await appImport("server/src/__tests__/helpers/external-test-database.ts") as typeof import("../server/src/__tests__/helpers/external-test-database.js");
|
||||
const { nativeSha256 } = await appImport("server/src/services/native-runtime/canonical.ts") as typeof import("../server/src/services/native-runtime/canonical.js");
|
||||
const { drizzle } = await import(pathToFileURL(serverRequire.resolve("drizzle-orm/postgres-js")).href);
|
||||
// GNU ps reports CPU time in whole seconds. Read Linux process clock ticks
|
||||
// directly; resolve their frequency once, outside every measured interval.
|
||||
const databaseCpuClockTicks = process.platform === "linux"
|
||||
? Number(execFileSync("getconf", ["CLK_TCK"], { encoding: "utf8" }).trim()) : null;
|
||||
if (databaseCpuClockTicks !== null) assert.ok(Number.isFinite(databaseCpuClockTicks) && databaseCpuClockTicks > 0);
|
||||
const databaseCpuResolutionMillis = databaseCpuClockTicks !== null ? 1000 / databaseCpuClockTicks
|
||||
: process.platform === "darwin" ? 10 : null;
|
||||
const output = arg("output");
|
||||
assert.ok(output, "An --output=<new-report.json> path is required to retain evidence");
|
||||
const traceFile = path.resolve(arg("trace-output") ?? `${output}.blocks.jsonl`);
|
||||
assert.ok(path.resolve(output) !== traceFile, "Summary and append-only block trace must use separate paths");
|
||||
function assertOutsideApp(file: string) {
|
||||
const relative = path.relative(root, path.resolve(file));
|
||||
assert.ok(relative === ".." || relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative), "Keep measurement evidence outside frozen APP_ROOT");
|
||||
}
|
||||
assertOutsideApp(traceFile); assertOutsideApp(output);
|
||||
await fs.writeFile(output, "", { flag: "wx" });
|
||||
// Exclusive creation prevents an accidental rerun from overwriting prior evidence.
|
||||
const trace = await fs.open(traceFile, "ax");
|
||||
let traceWriteMillis = 0;
|
||||
async function appendTrace(entry: Record<string, unknown>) {
|
||||
const started = performance.now();
|
||||
await trace.appendFile(`${JSON.stringify({ at: new Date().toISOString(), monotonicMs: started, ...entry })}\n`);
|
||||
traceWriteMillis += performance.now() - started;
|
||||
}
|
||||
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_EVENT_INGESTION_BENCHMARK_DATABASE_URL");
|
||||
if (externalUrl) assert.ok(["127.0.0.1", "localhost"].includes(new URL(externalUrl).hostname), "Use a disposable local database");
|
||||
|
||||
type Append = (db: Db, input: AppendHeartbeatRunEventInput) => Promise<AppendHeartbeatRunEventResult>;
|
||||
type AppendBatch = (db: Db, inputs: AppendHeartbeatRunEventInput[]) => Promise<AppendHeartbeatRunEventResult[]>;
|
||||
type EventModule = { appendHeartbeatRunEvent: Append; appendHeartbeatRunEvents?: AppendBatch };
|
||||
type Fixture = { companyId: string; agentId: string; runId: string; source: string; history: number; sourceOffset: number; next: number };
|
||||
type Query = { sql: string; params: unknown[]; elapsedMs?: number; returnedRows?: number };
|
||||
type Sample = {
|
||||
elapsedMs: number; eventsPerSecond: number; newEvents: number; duplicateEvents: number;
|
||||
appCpuMicros: number; databaseCpuMillis: number | null; heapDeltaBytes: number; rssBytes: number;
|
||||
eventLoopUtilization: number; eventLoopDelayMaxMs: number | null;
|
||||
retainedHistoryBefore: number; retainedHistoryAfter: number;
|
||||
attemptedEvents: number; nextBefore: number; nextAfter: number;
|
||||
appendStartedMonotonicMs: number; appendEndedMonotonicMs: number;
|
||||
firstCommitObservedMonotonicMs: number; lastCommitObservedMonotonicMs: number;
|
||||
verificationMillis: number; verificationEndedMonotonicMs: number;
|
||||
};
|
||||
type TimedResult = { sample: Sample; values: AppendHeartbeatRunEventInput[]; receipts: AppendHeartbeatRunEventResult[] };
|
||||
type Arm = "baseline" | "candidate";
|
||||
type CommitObservations = { first?: number; last?: number };
|
||||
const median = (values: number[]) => [...values].sort((a, b) => a - b)[Math.floor(values.length / 2)]!;
|
||||
const summary = (samples: Sample[]) => ({
|
||||
medianMs: median(samples.map(sample => sample.elapsedMs)),
|
||||
minMs: Math.min(...samples.map(sample => sample.elapsedMs)), maxMs: Math.max(...samples.map(sample => sample.elapsedMs)),
|
||||
medianEventsPerSecond: median(samples.map(sample => sample.eventsPerSecond)),
|
||||
medianAppCpuMicrosPerEvent: median(samples.map(sample => sample.appCpuMicros / (sample.newEvents + sample.duplicateEvents))),
|
||||
samples,
|
||||
});
|
||||
const baselineDir = await fs.mkdtemp(path.join(tmpdir(), "native-event-interleaved-baseline-"));
|
||||
await fs.writeFile(path.join(baselineDir, "package.json"), JSON.stringify({ type: "module" }));
|
||||
await fs.mkdir(path.join(baselineDir, "server"));
|
||||
// Keep original ESM package conditions and the same dependency instances as the
|
||||
// candidate. Only this disposable private snapshot receives a dependency link.
|
||||
await fs.symlink(path.join(root, "server/node_modules"), path.join(baselineDir, "server/node_modules"), "dir");
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: Db | undefined;
|
||||
const companiesToRemove: string[] = [];
|
||||
const setupStarted = performance.now();
|
||||
const loadBefore = loadavg();
|
||||
const report: Record<string, unknown> = {
|
||||
scope: "Real append service / real PostgreSQL; sum of per-arm active append intervals under 8x64 interleaving. Not assembled application burst completion, browser, or hosted acceptance.",
|
||||
appRoot: root, measurementFile, actualSelfSha256, gatesSha256, traceFile,
|
||||
originalHarnessProvenance: { path: "scripts/benchmark-native-event-ingestion.ts", provenanceOnly: true,
|
||||
sha256: "48e967a5e7a8c803ff3bf4890a37ff70564ba1a618d83bcff5bb110b35ee8133" },
|
||||
performanceAcceptanceEligible: !quick,
|
||||
qualification: quick ? "Ineligible 1k ordered A/A source qualification only; full accepted series requires ROOT RELEASE" : "Full source workload; application acceptance remains separate",
|
||||
phase, baselineRevision, candidateRevision, workload, workloadSha256,
|
||||
baselineSourceSha256: createHash("sha256").update(baselineSource).digest("hex"),
|
||||
measurement: { warmups, pairedOrder: "8x64 blocks interleaved; first arm alternates per pair", cache: "warm; seed and projection preparation excluded", databaseCpuResolutionMillis,
|
||||
databaseCpuSource: databaseCpuClockTicks === null ? "ps process CPU time" : "/proc/PID/stat utime+stime clock ticks",
|
||||
databaseCpuAggregateWorstQuantizationMillis: databaseCpuResolutionMillis === null ? null : blocksPerArm * databaseCpuResolutionMillis,
|
||||
databaseCpuCaution: "Each block is tick-quantized (typically 10ms on Linux); an eight-block sum has up to eight ticks of quantization uncertainty. Do not infer sub-tick CPU precision.",
|
||||
memoryCaution: "Heap deltas, RSS and process CPU describe shared-process intervals; heap deltas do not identify owned allocations.",
|
||||
activeInterval: "invoke() including append receipt checks, excluding generator, CPU probes, independent verification, trace writes and inter-arm gaps",
|
||||
pairInterval: "pair plan through final independent database verification; final pair-complete log and scenario diagnostics are outside this interval" },
|
||||
environment: { node: process.version, platform: process.platform, architecture: process.arch, cpus: availableParallelism(), loadBefore },
|
||||
results: [], pairTimeline: [], incomplete: true, cleanupComplete: false,
|
||||
};
|
||||
let calibratedNoise: any;
|
||||
|
||||
async function save() {
|
||||
report.traceWriteMillis = traceWriteMillis;
|
||||
if (output) await fs.writeFile(output, JSON.stringify(report, null, 2));
|
||||
}
|
||||
|
||||
function backendCpuMillis(pid: number): number | null {
|
||||
if (process.platform === "win32") return null;
|
||||
if (databaseCpuClockTicks !== null) {
|
||||
const stat = readFileSync(`/proc/${pid}/stat`, "utf8");
|
||||
// Fields after the parenthesized comm begin at field 3 (state).
|
||||
const fields = stat.slice(stat.lastIndexOf(")") + 2).trim().split(/\s+/);
|
||||
return (Number(fields[11]) + Number(fields[12])) / databaseCpuClockTicks * 1000;
|
||||
}
|
||||
const value = execFileSync("ps", ["-o", "time=", "-p", String(pid)], { encoding: "utf8" }).trim();
|
||||
const parts = value.split(":").map(Number);
|
||||
return parts.length >= 2 && parts.every(Number.isFinite)
|
||||
? parts.reduce((total, part) => total * 60 + part, 0) * 1000 : null;
|
||||
}
|
||||
|
||||
function input(fixture: Fixture, sequence: number): AppendHeartbeatRunEventInput {
|
||||
const envelope = {
|
||||
schema: "paperclip.prp.event.v1", schemaVersion: 1, runId: fixture.runId,
|
||||
normalizedSessionId: fixture.runId, sourceKind: "runner", sourceInstanceId: fixture.source,
|
||||
sourceEventId: `${fixture.source}:${sequence}`, sourceSeq: sequence,
|
||||
eventType: "item.delta", priority: 1, emittedAt: "2026-11-03T00:00:00.000Z",
|
||||
turnId: "benchmark-turn", itemId: "benchmark-item",
|
||||
payload: { kind: "agentMessage", channel: "final", text: "x".repeat(workload.payloadTextBytes) },
|
||||
};
|
||||
return {
|
||||
companyId: fixture.companyId, agentId: fixture.agentId, runId: fixture.runId,
|
||||
eventType: envelope.eventType, stream: "system", level: "info", payload: { prpEvent: envelope },
|
||||
nativeSource: {
|
||||
sourceInstanceId: fixture.source, sourceEventId: envelope.sourceEventId, sourceSeq: sequence,
|
||||
protocolSchemaVersion: 1, canonicalPayload: envelope,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// Query diagnostics run separately from unprofiled timing. Wrapping the actual
|
||||
// prepared execution captures completed query duration, including transactions.
|
||||
function diagnosticDb(target: Db, queries: Query[], transactions: Array<{ elapsedMs: number; lockAcquisitionQueryMs?: number; lockHeldThroughCommitMs?: number }>): Db {
|
||||
const recorded = installDatabaseWorkSignals(drizzle(target.$client, { schema: target._.fullSchema }), target) as Db;
|
||||
type Session = { prepareQuery: (...args: any[]) => { execute: (...args: any[]) => Promise<unknown> } };
|
||||
const decorate = (connection: Db, transaction?: { acquisitionMs?: number; acquiredAt?: number }) => {
|
||||
const session = (connection as unknown as { session: Session }).session;
|
||||
const prepare = session.prepareQuery;
|
||||
session.prepareQuery = function (this: Session, ...args: any[]) {
|
||||
const prepared = prepare.apply(this, args), execute = prepared.execute;
|
||||
const query = args[0] as Query;
|
||||
prepared.execute = async function (this: unknown, ...values: any[]) {
|
||||
const started = performance.now();
|
||||
try {
|
||||
const result = await execute.apply(this, values);
|
||||
const elapsedMs = performance.now() - started;
|
||||
queries.push({ sql: query.sql, params: query.params, elapsedMs, returnedRows: Array.isArray(result) ? result.length : undefined });
|
||||
if (transaction && /\bfor update\b/i.test(query.sql)) {
|
||||
transaction.acquisitionMs = elapsedMs;
|
||||
transaction.acquiredAt = performance.now();
|
||||
}
|
||||
return result;
|
||||
} catch (error) {
|
||||
queries.push({ sql: query.sql, params: query.params, elapsedMs: performance.now() - started });
|
||||
throw error;
|
||||
}
|
||||
};
|
||||
return prepared;
|
||||
};
|
||||
const original = connection.transaction.bind(connection);
|
||||
connection.transaction = ((callback: (tx: Db) => Promise<unknown>, options?: Parameters<Db["transaction"]>[1]) => {
|
||||
const started = performance.now(), state: { acquisitionMs?: number; acquiredAt?: number } = {};
|
||||
return original(async tx => callback(decorate(tx as unknown as Db, state)), options).finally(() => {
|
||||
transactions.push({ elapsedMs: performance.now() - started, lockAcquisitionQueryMs: state.acquisitionMs,
|
||||
lockHeldThroughCommitMs: state.acquiredAt === undefined ? undefined : performance.now() - state.acquiredAt });
|
||||
});
|
||||
}) as unknown as typeof connection.transaction;
|
||||
return connection;
|
||||
};
|
||||
return decorate(recorded);
|
||||
}
|
||||
|
||||
try {
|
||||
await appendTrace({ kind: "run-start", measurementFile, actualSelfSha256, workloadSha256, baselineRevision, gatesSha256, workload });
|
||||
// Snapshot changed supporting service modules too: a B1→B2 comparison must
|
||||
// not accidentally import B2 behavior behind its nominally B1 entry point.
|
||||
// The database schema/client intentionally remain the compatible migrated
|
||||
// superset for both arms; the benchmark is about the event owner itself.
|
||||
const changed = git(["diff", "--name-only", baselineRevision, "--", "server/src/services"])
|
||||
.trim().split("\n").filter(file => file.endsWith(".ts") && !file.endsWith(".test.ts"));
|
||||
const snapshots = new Map<string, string>([[sourcePath, baselineSource]]);
|
||||
let baselineCursorSource: string | undefined;
|
||||
try { baselineCursorSource = git(["show", `${baselineRevision}:${cursorSourcePath}`]); }
|
||||
catch { /* The pre-cursor baseline has no projection to prepare. */ }
|
||||
if (baselineCursorSource) snapshots.set(cursorSourcePath, baselineCursorSource);
|
||||
for (const file of changed) {
|
||||
if (snapshots.has(file)) continue;
|
||||
try { snapshots.set(file, git(["show", `${baselineRevision}:${file}`])); }
|
||||
catch { /* A newly added service cannot have been imported by the baseline. */ }
|
||||
}
|
||||
for (const [file, source] of snapshots) {
|
||||
const destination = path.join(baselineDir, file);
|
||||
await fs.mkdir(path.dirname(destination), { recursive: true });
|
||||
const sourceUrl = pathToFileURL(path.join(root, file));
|
||||
await fs.writeFile(destination, source.replace(/(from\s+|import\s+)["']([^"']+)["']/g, (_match, prefix: string, specifier: string) => {
|
||||
if (specifier.startsWith("node:")) return `${prefix}${JSON.stringify(specifier)}`;
|
||||
if (!specifier.startsWith(".")) return `${prefix}${JSON.stringify(specifier)}`;
|
||||
const actual = fileURLToPath(new URL(specifier, sourceUrl));
|
||||
const relative = path.relative(root, actual).replace(/\.js$/, ".ts");
|
||||
const selected = snapshots.has(relative) ? path.join(baselineDir, relative) : actual;
|
||||
return `${prefix}${JSON.stringify(pathToFileURL(selected).href)}`;
|
||||
}));
|
||||
}
|
||||
report.baselineServiceSnapshots = Object.fromEntries([...snapshots].map(([file, source]) => [file, createHash("sha256").update(source).digest("hex")]));
|
||||
const baselineFile = path.join(baselineDir, sourcePath);
|
||||
const baseline = await import(pathToFileURL(baselineFile).href) as EventModule;
|
||||
const candidate = phase === "aa" ? baseline : await appImport(sourcePath) as EventModule;
|
||||
report.projectionPriming = { baseline: Boolean(baselineCursorSource), candidate: phase === "aa" ? Boolean(baselineCursorSource) : true };
|
||||
report.effectiveCandidateRevision = phase === "aa" ? baselineRevision : candidateRevision;
|
||||
if (phase === "batch") assert.equal(typeof candidate.appendHeartbeatRunEvents, "function", "Batch phase requires appendHeartbeatRunEvents");
|
||||
report.candidateSourceSha256 = createHash("sha256").update(await fs.readFile(path.join(root, sourcePath))).digest("hex");
|
||||
report.baselineSnapshotSupport = { appRoot: root, dependencyResolution: "Original bare imports use normal ESM resolution through a private server/node_modules symlink to APP_ROOT/server/node_modules; relative unchanged support stays in APP_ROOT; DB schema/client are the APP_ROOT compatible migrated superset", privateSnapshotDirectory: baselineDir };
|
||||
const proofFiles = [...new Set([sourcePath, gatesPath, "packages/db/src/index.ts", "packages/db/src/work-signals.ts",
|
||||
"server/src/services/native-runtime/canonical.ts", "server/src/__tests__/helpers/external-test-database.ts",
|
||||
...git(["ls-files", "--", "server/src/services", "packages/db/src/schema"]).trim().split("\n"),
|
||||
...git(["ls-files", "--others", "--exclude-standard", "--", "server/src/services", "packages/db/src/schema"]).trim().split("\n"),
|
||||
])].filter(file => /\.(?:ts|mjs)$/.test(file) && !file.endsWith(".test.ts"));
|
||||
report.candidateSourceHashes = Object.fromEntries(await Promise.all(proofFiles.map(async file => [file,
|
||||
createHash("sha256").update(await fs.readFile(path.join(root, file))).digest("hex")])));
|
||||
report.candidateDirtyDiff = git(["diff", "HEAD", "--", sourcePath, "server/src/services/heartbeat", "packages/db/src/schema"]);
|
||||
if (arg("noise-report")) {
|
||||
calibratedNoise = JSON.parse(await fs.readFile(arg("noise-report")!, "utf8"));
|
||||
assertProtocolIdentity(calibratedNoise);
|
||||
assert.equal(calibratedNoise.phase, "aa"); assert.equal(calibratedNoise.incomplete, false);
|
||||
assert.equal(calibratedNoise.baselineSourceSha256, report.baselineSourceSha256, "Pinned baseline source changed after calibration");
|
||||
assert.equal(evaluateServiceRepeatability(calibratedNoise).passed, true,
|
||||
"A/A repeatability must pass before candidate measurements start");
|
||||
}
|
||||
if (!externalUrl) database = await startEmbeddedPostgresTestDatabase("native-event-ingestion-benchmark-");
|
||||
if (externalUrl) await applyPendingMigrations(externalUrl);
|
||||
db = createDb(externalUrl ?? database!.connectionString, { maxConnections: 1 });
|
||||
const [settings] = await db.$client.unsafe("select version() postgres_version, pg_backend_pid() backend_pid, current_setting('shared_buffers') shared_buffers, current_setting('work_mem') work_mem, current_setting('jit') jit");
|
||||
report.database = settings;
|
||||
|
||||
async function fixture(history: number, scenario = "ordered"): Promise<Fixture> {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), source = randomUUID();
|
||||
companiesToRemove.push(companyId);
|
||||
await db!.insert(companies).values({ id: companyId, name: "Synthetic native event benchmark", issuePrefix: `E${companyId.replaceAll("-", "")}` });
|
||||
await db!.insert(agents).values({ id: agentId, companyId, name: "Synthetic event source", adapterType: "paperclip_runner" });
|
||||
await db!.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nextEventSeq: history + 1 });
|
||||
const sourceOffset = scenario === "late-gap" ? 1 : scenario === "warm-suffix" ? workload.warmSourceOffset : 0;
|
||||
await db!.$client.unsafe(`insert into heartbeat_run_events
|
||||
(company_id, run_id, agent_id, seq, event_type, stream, level, payload, source_instance_id, source_event_id, source_seq, source_payload_sha256, protocol_schema_version)
|
||||
select $1::uuid, $2::uuid, $3::uuid, n, 'item.delta', 'system', 'info', jsonb_build_object('historicalSyntheticText', repeat('x', $5::integer)),
|
||||
$4, $4 || ':' || (n + $6::integer)::text, n + $6::integer, repeat('0', 64), 1
|
||||
from generate_series(1, $7::integer) n`, [companyId, runId, agentId, source, workload.payloadTextBytes, sourceOffset, history]);
|
||||
await db!.$client.unsafe("VACUUM ANALYZE heartbeat_run_events");
|
||||
return { companyId, agentId, runId, source, history, sourceOffset, next: history + sourceOffset + 1 };
|
||||
}
|
||||
|
||||
async function prepare(f: Fixture, arm: Arm) {
|
||||
const usesBaseline = arm === "baseline" || phase === "aa";
|
||||
if (usesBaseline && !baselineCursorSource) return;
|
||||
// Seed setup is an old-version append. Prime using the real fallback and
|
||||
// bounded rebuild before steady-state timing; cold behavior is traced below.
|
||||
const module = await import(usesBaseline
|
||||
? pathToFileURL(path.join(baselineDir, cursorSourcePath)).href
|
||||
: pathToFileURL(path.join(root, cursorSourcePath)).href) as Record<string, unknown>;
|
||||
const rebuild = module.rebuildNativeEventCursorPage as undefined | ((db: Db, binding: { companyId: string; runId: string }) => Promise<{ pending: boolean }>);
|
||||
assert.equal(typeof rebuild, "function", "Cursor owner must expose rebuildNativeEventCursorPage for seeded-history preparation");
|
||||
await db!.$client.unsafe("update heartbeat_runs set event_cursor_rebuild_pending=true where id=$1::uuid", [f.runId]);
|
||||
for (let page = 0; page <= Math.ceil((f.history + 1) / 512) + 2; page++) {
|
||||
const result = await rebuild!(db!, { companyId: f.companyId, runId: f.runId });
|
||||
if (!result.pending) return;
|
||||
}
|
||||
throw new Error("Cursor fixture failed to finish bounded rebuild");
|
||||
}
|
||||
|
||||
function emittedInputs(f: Fixture, scenario: string, repetitions = 1) {
|
||||
if (scenario === "late-gap") return [input(f, 1)];
|
||||
const newCount = scenario === "replay" ? events / 2 : events;
|
||||
const values: AppendHeartbeatRunEventInput[] = [];
|
||||
for (let group = 0; group < repetitions; group++) {
|
||||
const unique = Array.from({ length: newCount }, (_, n) => input(f, f.next + n));
|
||||
f.next += newCount;
|
||||
values.push(...unique);
|
||||
if (scenario === "replay") values.push(...unique);
|
||||
}
|
||||
return values;
|
||||
}
|
||||
|
||||
async function invoke(target: Db, arm: Arm, values: AppendHeartbeatRunEventInput[], commits?: CommitObservations) {
|
||||
const owner = arm === "baseline" ? baseline : candidate;
|
||||
const receipts: AppendHeartbeatRunEventResult[] = [];
|
||||
const receive = (received: AppendHeartbeatRunEventResult[]) => {
|
||||
if (commits && received.some(receipt => receipt.disposition === "committed")) {
|
||||
const observed = performance.now(); commits.first ??= observed; commits.last = observed;
|
||||
}
|
||||
receipts.push(...received);
|
||||
};
|
||||
if (phase === "batch" && arm === "candidate") {
|
||||
for (let offset = 0; offset < values.length; offset += 64) {
|
||||
const chunk = values.slice(offset, offset + 64);
|
||||
assert.ok(chunk.reduce((bytes, value) => bytes + Buffer.byteLength(JSON.stringify(value.nativeSource!.canonicalPayload)), 0) <= 64 * 1024);
|
||||
receive(await owner.appendHeartbeatRunEvents!(target, chunk));
|
||||
}
|
||||
}
|
||||
else for (const value of values) receive([await owner.appendHeartbeatRunEvent(target, value)]);
|
||||
assert.equal(receipts.length, values.length);
|
||||
assert.ok(receipts.every(receipt => receipt.disposition === "committed" || receipt.disposition === "duplicate"));
|
||||
const expectedCursor = Math.max(...values.map(value => value.nativeSource!.sourceSeq));
|
||||
assert.ok(receipts.every(receipt => receipt.highestContiguousSourceSeq <= expectedCursor || values.length === 1));
|
||||
return receipts;
|
||||
}
|
||||
|
||||
async function verifyDurable(f: Fixture, scenario: string, values: AppendHeartbeatRunEventInput[], receipts: AppendHeartbeatRunEventResult[]) {
|
||||
assert.equal(receipts.at(-1)!.highestContiguousSourceSeq,
|
||||
scenario === "late-gap" ? f.history + 1 : scenario === "warm-suffix" ? 0 : f.next - 1);
|
||||
const expectedRows = scenario === "late-gap" ? f.history + 1 : f.next - f.sourceOffset - 1;
|
||||
const [durable] = await db!.$client.unsafe("select count(*)::integer as events, max(seq)::bigint as last from heartbeat_run_events where run_id=$1::uuid", [f.runId]);
|
||||
assert.equal(Number(durable!.events), expectedRows, "Independent history read must account for every committed event");
|
||||
assert.equal(Number(durable!.last), expectedRows, "Run sequences must not skip or overlap");
|
||||
const expectedBySequence = new Map(values.map(value => [value.nativeSource!.sourceSeq, value]));
|
||||
const sequences = [...expectedBySequence.keys()];
|
||||
const persisted = await db!.$client.unsafe(`select source_instance_id, source_event_id, source_seq,
|
||||
source_payload_sha256, protocol_schema_version, payload from heartbeat_run_events
|
||||
where run_id=$1::uuid and source_instance_id=$2 and source_seq between $3::bigint and $4::bigint
|
||||
order by source_seq`, [f.runId, f.source, Math.min(...sequences), Math.max(...sequences)]);
|
||||
assert.equal(persisted.length, expectedBySequence.size);
|
||||
for (const row of persisted) {
|
||||
const expected = expectedBySequence.get(Number(row.source_seq))!;
|
||||
assert.equal(row.source_event_id, expected.nativeSource!.sourceEventId);
|
||||
assert.equal(row.source_instance_id, expected.nativeSource!.sourceInstanceId);
|
||||
assert.equal(row.protocol_schema_version, expected.nativeSource!.protocolSchemaVersion);
|
||||
assert.equal(row.source_payload_sha256, nativeSha256(expected.nativeSource!.canonicalPayload));
|
||||
assert.deepEqual(row.payload, expected.payload, "Durable event payload must match independently of its append receipt");
|
||||
}
|
||||
}
|
||||
|
||||
async function timed(f: Fixture, arm: Arm, scenario: string, repetitions = groups, context: Record<string, unknown> = {}): Promise<TimedResult> {
|
||||
const nextBefore = f.next, retainedHistoryBefore = f.next - f.sourceOffset - 1;
|
||||
const values = emittedInputs(f, scenario, repetitions);
|
||||
assert.equal(values.length, scenario === "late-gap" ? 1 : repetitions * events);
|
||||
const expectedNew = scenario === "replay" ? values.length / 2 : values.length;
|
||||
assert.equal(f.next, scenario === "late-gap" ? nextBefore : nextBefore + expectedNew, "Generator next must progress monotonically without gaps");
|
||||
const pgBefore = backendCpuMillis(Number(settings!.backend_pid));
|
||||
const beforeMemory = process.memoryUsage(), cpuBefore = process.cpuUsage(), eluBefore = performance.eventLoopUtilization();
|
||||
const loop = monitorEventLoopDelay({ resolution: 10 });
|
||||
loop.enable();
|
||||
const started = performance.now();
|
||||
let receipts: AppendHeartbeatRunEventResult[], ended: number;
|
||||
const commits: CommitObservations = {};
|
||||
try { receipts = await invoke(db!, arm, values, commits); ended = performance.now(); }
|
||||
catch (error) {
|
||||
const failedEnded = performance.now(); loop.disable();
|
||||
await appendTrace({ kind: "append-failed", ...context, arm, scenario, history: f.history,
|
||||
appendStartedMonotonicMs: started, appendEndedMonotonicMs: failedEnded, elapsedMs: failedEnded - started,
|
||||
attemptedEvents: values.length, nextBefore, nextAfter: f.next,
|
||||
retainedHistoryBefore, firstCommitObservedMonotonicMs: commits.first, lastCommitObservedMonotonicMs: commits.last });
|
||||
throw error;
|
||||
} finally { loop.disable(); }
|
||||
const elapsedMs = ended - started;
|
||||
const cpu = process.cpuUsage(cpuBefore), memory = process.memoryUsage();
|
||||
const eventLoopUtilization = performance.eventLoopUtilization(eluBefore).utilization;
|
||||
const pgAfter = backendCpuMillis(Number(settings!.backend_pid));
|
||||
const newEvents = receipts.filter(receipt => receipt.disposition === "committed").length;
|
||||
assert.equal(newEvents, expectedNew);
|
||||
assert.ok(commits.first !== undefined && commits.last !== undefined, "Every scenario commits at least one event");
|
||||
const verificationStarted = performance.now();
|
||||
await verifyDurable(f, scenario, values, receipts);
|
||||
const verificationEnded = performance.now();
|
||||
const sample: Sample = { elapsedMs, eventsPerSecond: values.length / elapsedMs * 1000, newEvents,
|
||||
duplicateEvents: values.length - newEvents, appCpuMicros: cpu.user + cpu.system,
|
||||
databaseCpuMillis: pgBefore === null || pgAfter === null ? null : pgAfter - pgBefore,
|
||||
heapDeltaBytes: memory.heapUsed - beforeMemory.heapUsed, rssBytes: memory.rss,
|
||||
eventLoopUtilization, eventLoopDelayMaxMs: loop.count ? loop.max / 1e6 : null,
|
||||
retainedHistoryBefore, retainedHistoryAfter: retainedHistoryBefore + newEvents,
|
||||
attemptedEvents: values.length, nextBefore, nextAfter: f.next,
|
||||
appendStartedMonotonicMs: started, appendEndedMonotonicMs: ended,
|
||||
firstCommitObservedMonotonicMs: commits.first, lastCommitObservedMonotonicMs: commits.last,
|
||||
verificationMillis: verificationEnded - verificationStarted, verificationEndedMonotonicMs: verificationEnded };
|
||||
return { sample, values, receipts };
|
||||
}
|
||||
|
||||
function aggregate(blocks: Sample[], scenario: string): Sample {
|
||||
const sum = (key: "elapsedMs" | "newEvents" | "duplicateEvents" | "appCpuMicros" | "heapDeltaBytes" | "attemptedEvents" | "verificationMillis") => blocks.reduce((total, block) => total + block[key], 0);
|
||||
assert.equal(blocks.length, scenario === "late-gap" ? 1 : blocksPerArm);
|
||||
const attemptedEvents = sum("attemptedEvents"), newEvents = sum("newEvents");
|
||||
assert.equal(attemptedEvents, scenario === "late-gap" ? 1 : 512, "Every attempted event must remain in the sample");
|
||||
assert.equal(newEvents, scenario === "replay" ? 256 : attemptedEvents);
|
||||
assert.equal(sum("duplicateEvents"), scenario === "replay" ? 256 : 0);
|
||||
for (let index = 1; index < blocks.length; index++) {
|
||||
assert.equal(blocks[index]!.retainedHistoryBefore, blocks[index - 1]!.retainedHistoryAfter, "Retained history must be continuous between blocks");
|
||||
assert.equal(blocks[index]!.nextBefore, blocks[index - 1]!.nextAfter, "Generator next must be continuous between blocks");
|
||||
}
|
||||
const first = blocks[0]!, last = blocks.at(-1)!, elapsedMs = sum("elapsedMs");
|
||||
const delays = blocks.flatMap(block => block.eventLoopDelayMaxMs === null ? [] : [block.eventLoopDelayMaxMs]);
|
||||
return { elapsedMs, eventsPerSecond: attemptedEvents / elapsedMs * 1000, attemptedEvents, newEvents,
|
||||
duplicateEvents: sum("duplicateEvents"), appCpuMicros: sum("appCpuMicros"),
|
||||
databaseCpuMillis: blocks.some(block => block.databaseCpuMillis === null) ? null : blocks.reduce((total, block) => total + block.databaseCpuMillis!, 0),
|
||||
heapDeltaBytes: sum("heapDeltaBytes"), rssBytes: Math.max(...blocks.map(block => block.rssBytes)),
|
||||
eventLoopUtilization: blocks.reduce((total, block) => total + block.eventLoopUtilization * block.elapsedMs, 0) / elapsedMs,
|
||||
eventLoopDelayMaxMs: delays.length ? Math.max(...delays) : null,
|
||||
retainedHistoryBefore: first.retainedHistoryBefore, retainedHistoryAfter: last.retainedHistoryAfter,
|
||||
nextBefore: first.nextBefore, nextAfter: last.nextAfter, appendStartedMonotonicMs: first.appendStartedMonotonicMs,
|
||||
appendEndedMonotonicMs: last.appendEndedMonotonicMs,
|
||||
firstCommitObservedMonotonicMs: first.firstCommitObservedMonotonicMs, lastCommitObservedMonotonicMs: last.lastCommitObservedMonotonicMs,
|
||||
verificationMillis: sum("verificationMillis"),
|
||||
verificationEndedMonotonicMs: last.verificationEndedMonotonicMs };
|
||||
}
|
||||
|
||||
// Qualify before entering the full benchmark. All comparisons preserve the
|
||||
// original append receipts; a failed fixture never becomes a timing sample.
|
||||
for (const history of histories) for (const scenario of scenarios) {
|
||||
const samples: Record<Arm, Sample[]> = { baseline: [], candidate: [] };
|
||||
let a: Fixture | undefined, b: Fixture | undefined;
|
||||
if (scenario !== "late-gap") {
|
||||
a = await fixture(history, scenario); b = await fixture(history, scenario);
|
||||
await prepare(a, "baseline"); await prepare(b, "candidate");
|
||||
for (let warmup = 0; warmup < warmups; warmup++) {
|
||||
for (const arm of ["baseline", "candidate"] as const) {
|
||||
await appendTrace({ kind: "warmup-start", history, scenario, warmup, arm, attempts: 512 });
|
||||
const { sample } = await timed(arm === "baseline" ? a : b, arm, scenario);
|
||||
await appendTrace({ kind: "warmup-complete", history, scenario, warmup, arm, ...sample });
|
||||
}
|
||||
}
|
||||
}
|
||||
for (let pair = 0; pair < pairs; pair++) {
|
||||
if (scenario === "late-gap") {
|
||||
a = await fixture(history, scenario); b = await fixture(history, scenario);
|
||||
await prepare(a, "baseline"); await prepare(b, "candidate");
|
||||
}
|
||||
const firstArm: Arm = pair % 2 ? "candidate" : "baseline";
|
||||
const armOrder: Arm[] = firstArm === "baseline" ? ["baseline", "candidate"] : ["candidate", "baseline"];
|
||||
const count = scenario === "late-gap" ? 1 : blocksPerArm;
|
||||
const blocks: Record<Arm, Sample[]> = { baseline: [], candidate: [] };
|
||||
const allValues: Record<Arm, AppendHeartbeatRunEventInput[]> = { baseline: [], candidate: [] };
|
||||
const allReceipts: Record<Arm, AppendHeartbeatRunEventResult[]> = { baseline: [], candidate: [] };
|
||||
const pairStarted = performance.now(), traceMillisBefore = traceWriteMillis;
|
||||
// The complete plan persists before the first append. Any missing start or
|
||||
// completion remains visible as an unstarted/interrupted block; no retry,
|
||||
// delay, outlier discard or growing-trace rewrite occurs.
|
||||
const plan = Array.from({ length: count }, (_, block) => armOrder.map(arm => ({ block, arm,
|
||||
attempts: scenario === "late-gap" ? 1 : blockAttempts }))).flat();
|
||||
await appendTrace({ kind: "pair-plan", history, scenario, pair, firstArm, pairStartedMonotonicMs: pairStarted, plan });
|
||||
for (let block = 0; block < count; block++) {
|
||||
assert.equal(a!.next - a!.sourceOffset, b!.next - b!.sourceOffset, "Corresponding blocks must start at the same retained history");
|
||||
for (const arm of armOrder) {
|
||||
const f = arm === "baseline" ? a! : b!;
|
||||
const context = { history, scenario, pair, block, arm, firstArm,
|
||||
retainedHistoryBefore: f.next - f.sourceOffset - 1, nextBefore: f.next };
|
||||
await appendTrace({ kind: "block-start", ...context });
|
||||
try {
|
||||
const measured = await timed(f, arm, scenario, scenario === "late-gap" ? 1 : groupsPerBlock, context);
|
||||
blocks[arm].push(measured.sample);
|
||||
allValues[arm].push(...measured.values); allReceipts[arm].push(...measured.receipts);
|
||||
await appendTrace({ kind: "block-complete", ...context, ...measured.sample });
|
||||
} catch (error) {
|
||||
await appendTrace({ kind: "block-failed", ...context, failedMonotonicMs: performance.now(),
|
||||
error: error instanceof Error ? { name: error.name, message: error.message, stack: error.stack } : String(error) });
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
const finalVerification: Record<string, { startedMonotonicMs: number; endedMonotonicMs: number; elapsedMs: number; attemptedEvents: number; boundary: string }> = {};
|
||||
for (const arm of armOrder) {
|
||||
await appendTrace({ kind: "final-verification-start", history, scenario, pair, arm });
|
||||
const started = performance.now();
|
||||
await verifyDurable(arm === "baseline" ? a! : b!, scenario, allValues[arm], allReceipts[arm]);
|
||||
const ended = performance.now();
|
||||
finalVerification[arm] = { startedMonotonicMs: started, endedMonotonicMs: ended, elapsedMs: ended - started,
|
||||
attemptedEvents: allValues[arm].length, boundary: "All append promises have committed; independent durable count, order, identity and payload verification; no asynchronous source queue or browser drain" };
|
||||
}
|
||||
const pairEnded = performance.now();
|
||||
const activeAppendMillis = [...blocks.baseline, ...blocks.candidate].reduce((total, block) => total + block.elapsedMs, 0);
|
||||
const blockVerificationMillis = [...blocks.baseline, ...blocks.candidate].reduce((total, block) => total + block.verificationMillis, 0);
|
||||
const finalVerificationMillis = Object.values(finalVerification).reduce((total, value) => total + value.elapsedMs, 0);
|
||||
const perArm = Object.fromEntries((["baseline", "candidate"] as const).map(arm => {
|
||||
const sample = aggregate(blocks[arm], scenario); samples[arm].push(sample);
|
||||
return [arm, { activeAppendMillis: sample.elapsedMs, firstAppendStartMonotonicMs: sample.appendStartedMonotonicMs,
|
||||
firstCommitObservedMonotonicMs: sample.firstCommitObservedMonotonicMs, lastCommitObservedMonotonicMs: sample.lastCommitObservedMonotonicMs,
|
||||
firstAppendToLastCommitWallMillis: sample.lastCommitObservedMonotonicMs - sample.appendStartedMonotonicMs,
|
||||
firstToLastCommitWallMillis: sample.lastCommitObservedMonotonicMs - sample.firstCommitObservedMonotonicMs,
|
||||
commitObservationBoundary: "Host receipt resolution after commit; a batch observes its committed receipts together, not individual database commit timestamps",
|
||||
blocks: blocks[arm] }];
|
||||
}));
|
||||
const timeline = { history, scenario, pair, firstArm, wholeElapsedMs: pairEnded - pairStarted,
|
||||
startedMonotonicMs: pairStarted, endedMonotonicMs: pairEnded, perArm, finalIndependentDrainVerification: finalVerification,
|
||||
activeAppendMillis, blockVerificationMillis, finalVerificationMillis,
|
||||
bookkeepingAndGapMillis: pairEnded - pairStarted - activeAppendMillis - blockVerificationMillis - finalVerificationMillis,
|
||||
traceWriteMillisWithinPair: traceWriteMillis - traceMillisBefore };
|
||||
(report.pairTimeline as unknown[]).push(timeline);
|
||||
await appendTrace({ kind: "pair-complete", ...timeline });
|
||||
// Save only compact scenario summaries after diagnostics, never the growing
|
||||
// block trace on every block. JSONL above is the authoritative failure log.
|
||||
}
|
||||
const diagnostics: Record<string, unknown> = {};
|
||||
for (const arm of ["baseline", "candidate"] as const) {
|
||||
const f = await fixture(history, scenario); await prepare(f, arm);
|
||||
const queries: Query[] = [], transactions: Array<{ elapsedMs: number }> = [];
|
||||
const recorded = diagnosticDb(db, queries, transactions);
|
||||
const values = emittedInputs(f, scenario), receipts = await invoke(recorded, arm, values);
|
||||
const plans: unknown[] = [];
|
||||
for (const query of queries) {
|
||||
if (plans.length >= 16 || plans.some((plan: any) => plan.sql === query.sql)) continue;
|
||||
const read = /^select\b/i.test(query.sql);
|
||||
// Never reexecute a mutation merely to profile it. Read plans report
|
||||
// actual rows/buffers; mutation plans describe shape without effects.
|
||||
const [plan] = await db.$client.unsafe(`EXPLAIN (${read ? "ANALYZE, BUFFERS, " : ""}FORMAT JSON) ${query.sql}`, query.params as never[]);
|
||||
plans.push({ sql: query.sql, analyzed: read, plan: plan!["QUERY PLAN"] });
|
||||
}
|
||||
const scanNodes = (node: any): any[] => [
|
||||
...(node["Relation Name"] === "heartbeat_run_events" ? [node] : []),
|
||||
...(node.Plans ?? []).flatMap(scanNodes),
|
||||
];
|
||||
const historyScans = (plans as Array<{ analyzed: boolean; plan: Array<{ Plan: unknown }> }>).filter(plan => plan.analyzed)
|
||||
.flatMap(plan => scanNodes(plan.plan[0].Plan));
|
||||
const usesProjection = arm === "baseline" || phase === "aa" ? Boolean(baselineCursorSource) : true;
|
||||
const examinedRows = (node: any) => (node["Actual Rows"] + (node["Rows Removed by Filter"] ?? 0)
|
||||
+ (node["Rows Removed by Index Recheck"] ?? 0)) * node["Actual Loops"];
|
||||
if (usesProjection) {
|
||||
assert.ok(historyScans.every(node => node["Node Type"] !== "Seq Scan" && examinedRows(node) <= 64),
|
||||
"A ready cursor must not scan retained event history, including the optimized baseline arm");
|
||||
assert.ok(!queries.some(query => /count\(\*\)/i.test(query.sql) && /heartbeat_run_events/.test(query.sql)),
|
||||
"A ready cursor unexpectedly used the retained-history aggregate fallback");
|
||||
}
|
||||
diagnostics[arm] = { sqlCalls: queries.length, transactions: transactions.length, sqlCallsPerEvent: queries.length / values.length,
|
||||
transactionsPerEvent: transactions.length / values.length, transactionTimings: transactions,
|
||||
lockTimingBoundary: "Acquisition query duration includes network/row-read cost; held interval ends after commit. It is not a measurement of pure lock wait.",
|
||||
persistedEvents: receipts.filter(receipt => receipt.disposition === "committed").length,
|
||||
usesProjection, structuralCacheBoundPassed: usesProjection ? true : null,
|
||||
historyScanRows: historyScans.map(examinedRows), queries, plans };
|
||||
}
|
||||
const baselineStats = summary(samples.baseline), candidateStats = summary(samples.candidate);
|
||||
const relativePairedDifference = samples.baseline.map((sample, n) => (samples.candidate[n]!.elapsedMs - sample.elapsedMs) / sample.elapsedMs);
|
||||
(report.results as unknown[]).push({ history, scenario,
|
||||
latencyGate: scenario === "late-gap" ? "structural-only" : history >= 10_000 ? "improvement" : "nonregression",
|
||||
baseline: baselineStats, candidate: candidateStats, diagnostics,
|
||||
pairedCandidateWins: relativePairedDifference.filter(value => value < 0).length,
|
||||
medianLatencyChangePercent: 100 * (candidateStats.medianMs / baselineStats.medianMs - 1),
|
||||
aaNoiseEnvelopeRelative: phase === "aa" ? Math.max(...relativePairedDifference.map(Math.abs)) : undefined,
|
||||
});
|
||||
await save();
|
||||
}
|
||||
report.setupAndExecutionMillis = performance.now() - setupStarted;
|
||||
report.loadAfter = loadavg();
|
||||
report.incomplete = false;
|
||||
report.acceptance = phase === "aa" ? "Baseline repeatability only; this does not establish an improvement" : "Compare against the frozen same-workload A/A report; throughput improvement needs at least four of five paired wins beyond baseline noise. Browser, control and full-app gates remain separate.";
|
||||
if (arg("noise-report")) {
|
||||
const noise = calibratedNoise;
|
||||
report.noiseReport = arg("noise-report");
|
||||
report.comparison = (report.results as any[]).map(result => {
|
||||
const control = noise.results.find((row: any) => row.history === result.history && row.scenario === result.scenario);
|
||||
assert.ok(control);
|
||||
const improvement = (result.baseline.medianMs - result.candidate.medianMs) / result.baseline.medianMs;
|
||||
const paired = noiseExceedingPairedWins(result.baseline.samples.map((sample: Sample, index: number) =>
|
||||
(sample.elapsedMs - result.candidate.samples[index].elapsedMs) / sample.elapsedMs), control.aaNoiseEnvelopeRelative);
|
||||
const improvedBeyondNoise = paired.pairedWinRequirementPassed && improvement > control.aaNoiseEnvelopeRelative;
|
||||
const noRegressionBeyondNoise = -improvement <= control.aaNoiseEnvelopeRelative;
|
||||
return { history: result.history, scenario: result.scenario, pairedCandidateWins: result.pairedCandidateWins,
|
||||
noiseExceedingPairedWins: paired.noiseExceedingPairedWins, requiredPairedWins: paired.requiredPairedWins,
|
||||
latencyGate: result.latencyGate,
|
||||
improvementRelative: improvement, noiseEnvelopeRelative: control.aaNoiseEnvelopeRelative,
|
||||
improvedBeyondNoise, noRegressionBeyondNoise,
|
||||
acceptancePassed: result.latencyGate === "structural-only" ? null
|
||||
: result.latencyGate === "improvement" ? improvedBeyondNoise : noRegressionBeyondNoise };
|
||||
});
|
||||
}
|
||||
await save();
|
||||
} catch (error) {
|
||||
report.incomplete = true;
|
||||
report.failure = error instanceof Error ? { name: error.name, message: error.message, stack: error.stack } : String(error);
|
||||
await appendTrace({ kind: "run-failed", failure: report.failure });
|
||||
throw error;
|
||||
} finally {
|
||||
try {
|
||||
// The owned cluster is removed as a whole below. Per-company agent deletes
|
||||
// would repeatedly scan other fixtures through foreign-key checks. An
|
||||
// explicitly supplied test database retains unrelated data, so remove all
|
||||
// our event rows first, then their parents in separate phases.
|
||||
if (externalUrl) {
|
||||
for (const table of ["heartbeat_run_events", "heartbeat_runs", "agents", "companies"]) {
|
||||
for (const companyId of companiesToRemove) {
|
||||
await db?.$client.unsafe(`delete from ${table} where ${table === "companies" ? "id" : "company_id"}=$1::uuid`, [companyId]);
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
try { await db?.$client.end({ timeout: 1 }); }
|
||||
finally {
|
||||
try { await database?.cleanup(); }
|
||||
finally { await fs.rm(baselineDir, { recursive: true, force: true }); }
|
||||
}
|
||||
}
|
||||
report.cleanupComplete = true;
|
||||
await appendTrace({ kind: "cleanup-complete", incomplete: report.incomplete });
|
||||
await save();
|
||||
await trace.close();
|
||||
}
|
||||
if (phase === "aa") {
|
||||
report.repeatability = evaluateServiceRepeatability(report);
|
||||
await save();
|
||||
}
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
if (process.argv.includes("--require-acceptance")) assertServiceAcceptance(report);
|
||||
if (process.argv.includes("--require-repeatability")) {
|
||||
assert.equal(evaluateServiceRepeatability(report).passed, true, "A/A repeatability gate failed; retain evidence and diagnose before candidate measurements");
|
||||
}
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"scope": "Offline packaging checks only; no new database, application, provider or benchmark execution",
|
||||
"exactFrozenHashesVerified": {
|
||||
"measurement-source.ts": "c8adaf5ac16cab67513d6c8b888afa44d95fa3688d4831b8126fe3d997f1d039",
|
||||
"prospective-protocol.md": "302c11ec61d5170822c08e47f798ca582b8e5afefd9d223fb9023d9ead7a2ad3",
|
||||
"gates.mjs": "ded8bcbb8f05732348f6520b9c5e41b7c35bc098997eb263a9dcb174eede75fc"
|
||||
},
|
||||
"allFiveCompressedPayloadsEqualOriginalBytes": true,
|
||||
"newAaRepeatabilityPassed": true,
|
||||
"excludedOldAaRepeatabilityPassed": false,
|
||||
"candidatePassedTimingCases": 9,
|
||||
"candidateStructuralCases": 3,
|
||||
"candidateFailedCases": 0,
|
||||
"allTwelveMediansAnd120SampleCountsVerified": true,
|
||||
"syntaxChecks": [
|
||||
"node --check scripts/benchmark-native-event-ingestion-interleaved.ts",
|
||||
"node --check compare-runtime-dependencies.cjs",
|
||||
"node --check gates.mjs"
|
||||
],
|
||||
"scan": {
|
||||
"files": 26,
|
||||
"decompressedBytes": 15713136,
|
||||
"actualCredentialSourcePresent": true,
|
||||
"actualCredentialMatches": 0,
|
||||
"patternMatches": {
|
||||
"credential_literal": 0,
|
||||
"vault_or_person_path": 0,
|
||||
"nonloopback_credential_url": 0
|
||||
}
|
||||
}
|
||||
}
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
# Prospective native event ingestion source protocol
|
||||
|
||||
Status: source implementation and local source checks only. No source workload, VM measurement, application build or accepted measurement series is authorized by this document. Root owns VM operations. After read-only review, the first measurement step is exactly one bounded, ineligible `--quick --phase=aa` qualification at 1,000 ordered retained events. A full series requires explicit ROOT RELEASE after that qualification and its evidence have been reviewed. Never start another fixture while cleanup is unresolved.
|
||||
|
||||
The complete earlier monolithic A/A experiment failed the unchanged 20% maximum paired-difference / 10% median-arm-drift noise gates. Its failures remain evidence. This prospective schedule changes the source estimand and gets a new source hash and workload hash; it neither removes those failures nor retroactively qualifies their timings. The old harness/report identified by SHA-256 `48e967a5e7a8c803ff3bf4890a37ff70564ba1a618d83bcff5bb110b35ee8133` is provenance only. It cannot supply A/A calibration or acceptance for the new schedule.
|
||||
|
||||
## Frozen application references and ownership
|
||||
|
||||
Application refs accepted by root, not measured locally here:
|
||||
|
||||
- B0: `5337df423e622755d44f3ece5447d1e0beef5f40`
|
||||
- B1: `e9b909015ad3d517f28c45f0c66c8f41d6112b4b`
|
||||
- B2: `fcb5d94848b3c4b0b32096ff5e023618e0f0637b`
|
||||
|
||||
The source harness is `scripts/benchmark-native-event-ingestion-interleaved.ts`. The original `scripts/benchmark-native-event-ingestion.ts` and `scripts/native-ingestion-performance-gates.mjs` stay untouched. The gate helper must retain SHA-256 `ded8bcbb8f05732348f6520b9c5e41b7c35bc098997eb263a9dcb174eede75fc`; the new harness asserts it before any DB workload.
|
||||
|
||||
The harness owns only synthetic companies/agents/runs/events and its private baseline source snapshot. Root owns the selected frozen app checkout, its dependency/build identity, VM access, fixture sequence and release decisions. No production source changes, app builds, measurement execution or VM calls were part of implementation.
|
||||
|
||||
## Schedule and preserved workload
|
||||
|
||||
Full matrix: retained histories 1,000 / 10,000 / 100,000, with ordered / replay / late-gap / warm-suffix scenarios. Seed, 128-byte text, source identities, payload shape, source offset, generator order and vacuum/analyze preparation are inherited from the frozen harness.
|
||||
|
||||
For every non-late-gap case, prepare two independent fixtures at the same retained history, then run the original two warmups of 512 attempts per arm. Warmups remain monolithic, excluded from measured samples and logged separately. Five measured pairs follow. Each pair contains eight corresponding blocks per arm; each block emits exactly two original generator groups of 32 attempts (64 total). The order is:
|
||||
|
||||
- Pair 0, 2, 4: B64, C64, B64, C64, repeated until both arms reach 512 attempts.
|
||||
- Pair 1, 3: C64, B64, C64, B64, repeated until both arms reach 512 attempts.
|
||||
|
||||
The first arm is fixed within a pair and alternates between pairs. There are no deliberate timer delays, retries, discarded blocks, outlier exclusions or reduced counts. Each corresponding block starts at the same retained history. The generator's `next` advances without gaps and is continuous across blocks. Ordered and warm-suffix samples commit exactly 512 new events. Replay preserves each original 16-new/16-replay group, yielding exactly 256 new and 256 duplicate attempts per 512-attempt arm sample.
|
||||
|
||||
Late-gap retains a fresh fixture and one operation per arm per pair, no warmups, five complete pairs, and structural-only latency treatment. Candidate batch invocation never exceeds 64 events or the inherited 64-KiB canonical-payload cap. Ordinary compare and A/A continue invoking individual appends.
|
||||
|
||||
Mandatory qualification uses `--quick --phase=aa`: only 1,000 ordered history, still 16×32 attempts, two warmups per arm and five measured pairs. It is explicitly ineligible for repeatability or performance acceptance. Event/group/pair/history CLI overrides that reduce or change the frozen design are rejected. Quick is not a route to relaxed gates.
|
||||
|
||||
## Estimand, timing boundaries and precision
|
||||
|
||||
The comparison sample is the sum of eight per-arm ACTIVE append intervals. Wall time includes `invoke()` and its append receipt checks; it excludes generator work, independent durable verification, backend CPU probes, trace writes and inter-arm gaps. Per-arm process and database CPU observations are summed over corresponding block observation windows. This estimand is source service cost under the stated interleaving, not assembled application burst completion, browser delivery latency, provider throughput, or hosted performance acceptance.
|
||||
|
||||
For every measured block retain arm, scenario, seeded history, actual retained history before/after, `next` before/after, attempted/new/duplicate counts, active start/end/duration, process CPU, database CPU, memory, event-loop observations, and independent verification duration. Host commit observations are recorded when committed append receipts resolve. A batch acknowledges its committed receipts together; these are not individual database commit timestamps.
|
||||
|
||||
Each measured pair additionally records whole elapsed time from pair-plan bookkeeping through final independent durable verification; per-arm first-append-to-last-commit and first-to-last-observed-commit wall time; active wall sums; block verification sums; final verification time; trace write time; and the residual `bookkeepingAndGapMillis`. That residual explicitly includes generator work, CPU probes, trace I/O, aggregation overhead within the pair and inter-arm gaps. The final `pair-complete` trace write and later scenario SQL/plan diagnostics are outside whole-pair time and remain within total trace/execution accounting.
|
||||
|
||||
Append promises resolve after commit, with no asynchronous source queue. The final independent drain/verification is a separately timed read of durable row count, run sequence extent, source identities, payload hashes and payloads for the entire arm's attempted sample. It does not claim browser or application delivery drained.
|
||||
|
||||
Linux PostgreSQL CPU comes from `/proc/PID/stat` clock ticks using the runtime `CLK_TCK`. Typical per-block resolution is 10 ms. Eight block differences have worst aggregate quantization of eight ticks (typically 80 ms); retain raw tick-quantized observations and avoid sub-tick precision claims. A single late-gap sample has one interval. Heap deltas, RSS, process CPU and event-loop observations describe a shared process; heap delta is not owned allocation or a reliable allocation-efficiency estimate.
|
||||
|
||||
## Correctness, SQL diagnostics, evidence and cleanup
|
||||
|
||||
All original append receipt checks, independent durable identity/payload/count checks, SQL capture, EXPLAIN plan handling, cursor structural bounds, and cleanup logic remain. Verification operates after each 64-attempt block and once again over the complete pair's arm sample. No mutation is rerun for profiling. Ready projections must have no retained-history sequential scans, no aggregate fallback and at most the original 64 examined rows on analyzed retained-history read nodes. Late-gap remains structural-only.
|
||||
|
||||
`--output` is mandatory. Both summary and `--trace-output` paths must be outside the frozen app; defaults use `<output>.blocks.jsonl`. Outputs must be new paths. A new append-only JSONL file is exclusively created, with a complete pair block plan persisted before that pair's first append. Each start/completion/failure is appended. Planned blocks lacking a start are unstarted evidence; a started block lacking completion is interrupted evidence. Append failures retain observed duration/commit bounds before rethrowing. A failed run stays incomplete and cannot qualify. The trace is never rewritten from a growing in-memory array on each block. Scenario summaries can be rewritten separately, with the append-only trace authoritative for interrupted work.
|
||||
|
||||
All owned DB clients and embedded clusters/private source snapshots are cleaned up in `finally`. An explicitly supplied disposable localhost DB removes only owned event rows, runs, agents and companies in the inherited order. The summary and append-only trace retain cleanup status. Cleanup must complete before another fixture starts. External supplied URLs must pass the inherited disposable-database helper and localhost restriction.
|
||||
|
||||
## Private execution and source provenance
|
||||
|
||||
Copy the exact public `.ts` bytes to a private `.mts` path outside APP_ROOT. The extension is mandatory because an external `.ts` without a module package context can be transpiled as CommonJS and reject top-level await. `--app-root=<frozen-app>` controls Git, `createRequire`, DB/client/work-signals imports, canonical hashing, external-test-database helper, acceptance gates and candidate service imports. Static relative imports are erased types only. The measurement records its actual executing file path and SHA separately from the original harness's provenance hash.
|
||||
|
||||
Baseline service snapshots live under a private temporary directory with its own `package.json` containing `type: module`. Baseline relative imports select corresponding snapshot files when present and otherwise explicit APP_ROOT support paths. Original bare imports remain unchanged and retain normal ESM package conditions. Only the disposable private snapshot gets a `server/node_modules` symlink to `APP_ROOT/server/node_modules`, preserving the same dependency instances and export conditions as the candidate. No dependency link or file is installed in the frozen app. The harness itself uses an APP_ROOT-based `createRequire` for its inherited diagnostics dependency resolution. The compatible migrated APP_ROOT DB schema/client and unchanged support modules intentionally support both arms, as in the original source harness. This source-only shared-support provenance is explicit; it does not claim an independent application artifact per arm.
|
||||
|
||||
`candidateSourceHashes` hashes actual APP_ROOT service modules and schema files (plus DB/client/work-signals, canonical helper, external-DB helper and gate source), not the absent public measurement path in a frozen app. `actualSelfSha256` hashes the exact private executing measurement bytes. Report validation and noise loading require matching self hash, recomputed workload hash, exact baseline revision and unchanged gate hash. Noise loading additionally requires identical pinned baseline source bytes and a complete, eligible, passing same-workload A/A report. Report validation checks APP_ROOT supporting hashes. Old monolithic or differently copied measurement evidence cannot qualify.
|
||||
|
||||
## Local checks and next commands
|
||||
|
||||
The following are source-only checks; they do not open PostgreSQL or append synthetic events:
|
||||
|
||||
```sh
|
||||
cp scripts/benchmark-native-event-ingestion-interleaved.ts /private/tmp/native-ingestion-interleaved.mts
|
||||
node --import "$APP_ROOT/server/node_modules/tsx/dist/loader.mjs" /private/tmp/native-ingestion-interleaved.mts \
|
||||
--app-root="$APP_ROOT" --baseline-ref="$BASELINE_REVISION" --quick --phase=aa --check-source
|
||||
```
|
||||
|
||||
Read-only review precedes the first qualification. Root chooses a new external output directory and one accepted baseline ref, ensures the VM is quiet and all prior fixture resources are cleaned, then runs exactly one bounded quick A/A qualification using the same command without `--check-source` and with `--output=<new-external-report.json>`. Do not add `--require-repeatability` to quick: the complete-matrix gate correctly rejects quick evidence. Review its complete block log, payload/order/SQL diagnostics and cleanup receipt before ROOT RELEASE.
|
||||
|
||||
After ROOT RELEASE, a full same-workload A/A must pass every unchanged 20% paired and 10% median drift limit on all nine timing cases, with complete late-gap structural evidence. Candidate comparisons require that exact full A/A report via `--noise-report`. Improvement cases still need the median improvement beyond the A/A envelope and at least four of five individual paired improvements beyond that envelope; 1k cases retain nonregression treatment. No threshold is loosened. Every source validation must use the same private `.mts` bytes, baseline ref and matching `--quick` status as its report. Application/browser/stop/control acceptance remains a separate required layer.
|
||||
|
||||
Implementation evidence (2026-11-03, local source only): the new harness SHA-256 is `c8adaf5ac16cab67513d6c8b888afa44d95fa3688d4831b8126fe3d997f1d039`. A byte-identical external `.mts` copy passed source-only execution. The focused TypeScript invocation over this harness plus the repository Express request augmentation passed without emitting or building. Thirteen synthetic source/report checks passed, including old-self/workload/baseline/gate rejection, incomplete/cleanup rejection, both excessive-noise failures and the inclusive 20%/10% gate boundary. A private ESM snapshot import check proved reference equality with APP_ROOT's actual ESM Drizzle and DB exports; no DB connection was opened. Full workload SHA-256 is `ca4894b0723d0bffff0fc88d1596d64aa8fd75aa1dd8d861372ea993db24e164`; quick workload SHA-256 is `86ccdcfbd02cc1d16bf75223337305e3c10f29ca80da3bec74db28afe0ab7cc6`. These checks are not performance measurements or qualification evidence. No VM workload, full build or performance series ran during implementation.
|
||||
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"recordedAt": "2026-11-03T23:44:11.117779+00:00",
|
||||
"leadExplicitRelease": true,
|
||||
"releaseFile": "/home/ubuntu/native-ingestion/helpers/native-ingestion-interleaved-full-release.md",
|
||||
"hashes": {
|
||||
"native-ingestion-run-interleaved-service.mjs": "ecf9adcf9fef440fc10c049199dabb8ece59f6811bef526f2c775435a1c62f8b",
|
||||
"native-ingestion-launch-interleaved-service.py": "152d477f2b3bbe4a29c12feca38b0a0c52bc2ca6488027e2724f006042bc3abd",
|
||||
"native-ingestion-interleaved-full-release.md": "904434a688bbfc0b1de0673d06941881a28f6ec81b8b6153d733b8ab58a4803e"
|
||||
},
|
||||
"measurementSourceSha256": "c8adaf5ac16cab67513d6c8b888afa44d95fa3688d4831b8126fe3d997f1d039",
|
||||
"protocolSha256": "302c11ec61d5170822c08e47f798ca582b8e5afefd9d223fb9023d9ead7a2ad3",
|
||||
"allPriorFixturesCleanupVerified": true,
|
||||
"stages": [
|
||||
"service-aa-b0",
|
||||
"service-b0-b1",
|
||||
"service-aa-b1",
|
||||
"service-b1-b2",
|
||||
"service-b0-b2"
|
||||
]
|
||||
}
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
[
|
||||
{
|
||||
"index": 1,
|
||||
"stage": "service-aa-b0",
|
||||
"cwd": "/home/ubuntu/native-ingestion/fixture2-cursors",
|
||||
"executable": "/opt/native-ingestion/node-v26.4.0-linux-x64/bin/node",
|
||||
"args": [
|
||||
"--import",
|
||||
"./server/node_modules/tsx/dist/loader.mjs",
|
||||
"/home/ubuntu/native-ingestion/helpers/benchmark-native-event-ingestion-interleaved.mts",
|
||||
"--app-root=/home/ubuntu/native-ingestion/fixture2-cursors",
|
||||
"--baseline-ref=5337df423e622755d44f3ece5447d1e0beef5f40",
|
||||
"--phase=aa",
|
||||
"--output=/home/ubuntu/native-ingestion/evidence/fixture2/interleaved-service/service-aa-b0.json",
|
||||
"--require-repeatability"
|
||||
],
|
||||
"environmentOverrides": {},
|
||||
"startedAt": "2026-11-03T23:44:11.280Z",
|
||||
"log": "/home/ubuntu/native-ingestion/evidence/fixture2/interleaved-service/service-aa-b0.log",
|
||||
"childPid": 93475,
|
||||
"finishedAt": "2026-11-03T23:56:55.856Z",
|
||||
"code": 0,
|
||||
"signal": null
|
||||
}
|
||||
]
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
+76
@@ -0,0 +1,76 @@
|
||||
{
|
||||
"report": "/home/ubuntu/native-ingestion/evidence/fixture2/interleaved-service/service-aa-b0.json",
|
||||
"limits": {
|
||||
"maximumPairedRelativeDifference": 0.2,
|
||||
"maximumMedianArmDrift": 0.1
|
||||
},
|
||||
"eligible": true,
|
||||
"eligibilityFailures": [],
|
||||
"cases": [
|
||||
{
|
||||
"history": 1000,
|
||||
"scenario": "ordered",
|
||||
"maximumPairedRelativeDifference": 0.014079047972212447,
|
||||
"medianArmDrift": 0.012275859436574102,
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 1000,
|
||||
"scenario": "replay",
|
||||
"maximumPairedRelativeDifference": 0.03982212020678522,
|
||||
"medianArmDrift": 0.001509348380207063,
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 1000,
|
||||
"scenario": "warm-suffix",
|
||||
"maximumPairedRelativeDifference": 0.018842439321434798,
|
||||
"medianArmDrift": 0.012182094590722805,
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 10000,
|
||||
"scenario": "ordered",
|
||||
"maximumPairedRelativeDifference": 0.017474622955774097,
|
||||
"medianArmDrift": 0.0016529478530073213,
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 10000,
|
||||
"scenario": "replay",
|
||||
"maximumPairedRelativeDifference": 0.016304743570125815,
|
||||
"medianArmDrift": 0.0064856671560466796,
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 10000,
|
||||
"scenario": "warm-suffix",
|
||||
"maximumPairedRelativeDifference": 0.11830110496052082,
|
||||
"medianArmDrift": 0.061328132942894305,
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 100000,
|
||||
"scenario": "ordered",
|
||||
"maximumPairedRelativeDifference": 0.010693214938958221,
|
||||
"medianArmDrift": 0.009308917031380358,
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 100000,
|
||||
"scenario": "replay",
|
||||
"maximumPairedRelativeDifference": 0.009332642395176317,
|
||||
"medianArmDrift": 0.0028535574402402016,
|
||||
"passed": true
|
||||
},
|
||||
{
|
||||
"history": 100000,
|
||||
"scenario": "warm-suffix",
|
||||
"maximumPairedRelativeDifference": 0.007615089382403567,
|
||||
"medianArmDrift": 0.005994934834498225,
|
||||
"passed": true
|
||||
}
|
||||
],
|
||||
"passed": true,
|
||||
"excluded": "Late-gap single-operation latency is structural-only; its complete paired evidence is still required"
|
||||
}
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
[
|
||||
{
|
||||
"index": 1,
|
||||
"stage": "service-b0-b1",
|
||||
"cwd": "/home/ubuntu/native-ingestion/fixture2-cursors",
|
||||
"executable": "/opt/native-ingestion/node-v26.4.0-linux-x64/bin/node",
|
||||
"args": [
|
||||
"--import",
|
||||
"./server/node_modules/tsx/dist/loader.mjs",
|
||||
"/home/ubuntu/native-ingestion/helpers/benchmark-native-event-ingestion-interleaved.mts",
|
||||
"--app-root=/home/ubuntu/native-ingestion/fixture2-cursors",
|
||||
"--baseline-ref=5337df423e622755d44f3ece5447d1e0beef5f40",
|
||||
"--phase=compare",
|
||||
"--output=/home/ubuntu/native-ingestion/evidence/fixture2/interleaved-service/service-b0-b1.json",
|
||||
"--noise-report=/home/ubuntu/native-ingestion/evidence/fixture2/interleaved-service/service-aa-b0.json",
|
||||
"--require-acceptance"
|
||||
],
|
||||
"environmentOverrides": {},
|
||||
"startedAt": "2026-11-04T00:20:41.130Z",
|
||||
"log": "/home/ubuntu/native-ingestion/evidence/fixture2/interleaved-service/service-b0-b1.log",
|
||||
"childPid": 112072,
|
||||
"finishedAt": "2026-11-04T00:31:49.701Z",
|
||||
"code": 0,
|
||||
"signal": null
|
||||
}
|
||||
]
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
+8
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"checkedAt": "2026-11-04T00:32:24.827992+00:00",
|
||||
"complete": true,
|
||||
"sourceReportSha256": "283eef0a9d5cf3b39bb7ebf7444ced40025f499e51847c2cc205f3101c1f1470",
|
||||
"blocksSha256": "382b0f7ca51cc855c0bf0d8b1685cb01cbf0dde1d55ecf4c90dc527c32db7f57",
|
||||
"cleanupComplete": true,
|
||||
"noCompetingOwnedProcesses": true
|
||||
}
|
||||
+637
@@ -0,0 +1,637 @@
|
||||
{
|
||||
"scope": "Read-only immutable Git source comparison for selected core paths only; not whole application or transitive closure equivalence",
|
||||
"refs": {
|
||||
"SOURCE_B0": "5337df423e622755d44f3ece5447d1e0beef5f40",
|
||||
"SOURCE_B1": "e9b909015ad3d517f28c45f0c66c8f41d6112b4b",
|
||||
"SOURCE_B2": "fcb5d94848b3c4b0b32096ff5e023618e0f0637b",
|
||||
"APP_B0": "a36567f9fbf805ad72207d54f539516e3121741e",
|
||||
"APP_B1": "d8cd3be0aeed4f9fb93ecc3cf725f1d46d5a0310",
|
||||
"APP_B2": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da"
|
||||
},
|
||||
"pairs": [
|
||||
{
|
||||
"lane": "B0",
|
||||
"before": "5337df423e622755d44f3ece5447d1e0beef5f40",
|
||||
"after": "a36567f9fbf805ad72207d54f539516e3121741e",
|
||||
"corePaths": [
|
||||
{
|
||||
"path": "server/src/services/heartbeat-run-events.ts",
|
||||
"beforeSha256": "9f38c6c875918f54ae1505a5a8c8dc38ac705185b9dc5ac4cb92f2ecadc69fc1",
|
||||
"afterSha256": "9f38c6c875918f54ae1505a5a8c8dc38ac705185b9dc5ac4cb92f2ecadc69fc1",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/heartbeat/native-event-cursors.ts",
|
||||
"beforeSha256": null,
|
||||
"afterSha256": null,
|
||||
"equal": true,
|
||||
"absentBoth": true
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/heartbeat/native-event-cursor-work.ts",
|
||||
"beforeSha256": null,
|
||||
"afterSha256": null,
|
||||
"equal": true,
|
||||
"absentBoth": true
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/paperclip-control-plane-port.ts",
|
||||
"beforeSha256": "b0b017e19b6808c600a18ea4f47d728e8d41608b3b26b5c42d4a2eafe0a8588c",
|
||||
"afterSha256": "b0b017e19b6808c600a18ea4f47d728e8d41608b3b26b5c42d4a2eafe0a8588c",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-run-coordinator-store.ts",
|
||||
"beforeSha256": "458a65844efedc35130f701f11e689560ab0d1c131f347d0741346faeb2fe9b0",
|
||||
"afterSha256": "458a65844efedc35130f701f11e689560ab0d1c131f347d0741346faeb2fe9b0",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/native-session-runtime.ts",
|
||||
"beforeSha256": "e916df31032785a9f4ce98df8d703c974391dbbb0ddf7a4e34ea43d231f9abdb",
|
||||
"afterSha256": "e916df31032785a9f4ce98df8d703c974391dbbb0ddf7a4e34ea43d231f9abdb",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/native-event-persistence.ts",
|
||||
"beforeSha256": null,
|
||||
"afterSha256": null,
|
||||
"equal": true,
|
||||
"absentBoth": true
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/contracts/control-plane-port.ts",
|
||||
"beforeSha256": "f200d8b9a9bc8cdcad918761e8ad3776663f4d129a8e20e8289b5148f6509923",
|
||||
"afterSha256": "f200d8b9a9bc8cdcad918761e8ad3776663f4d129a8e20e8289b5148f6509923",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts",
|
||||
"beforeSha256": "fb3ee3d87ad7824cc8a986bc433f00c73150a70f51e6f6dfe67ed5da976a350c",
|
||||
"afterSha256": "fb3ee3d87ad7824cc8a986bc433f00c73150a70f51e6f6dfe67ed5da976a350c",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/client.ts",
|
||||
"beforeSha256": "a4f89bf5ed8144c685d3e2113ca377f98734df8f68300f8d381ed5eaf509dfa0",
|
||||
"afterSha256": "a4f89bf5ed8144c685d3e2113ca377f98734df8f68300f8d381ed5eaf509dfa0",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/work-signals.ts",
|
||||
"beforeSha256": "f402f868c4cbbb19f78a8cfccc1cfe226d102a93ccff8efcf06a604ed09dc6f0",
|
||||
"afterSha256": "f402f868c4cbbb19f78a8cfccc1cfe226d102a93ccff8efcf06a604ed09dc6f0",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/run-event-payload.ts",
|
||||
"beforeSha256": "88b89c582d292ce91a1e053ad4d8b1ab73316dff035eee3dab65b0fe9c2717c4",
|
||||
"afterSha256": "88b89c582d292ce91a1e053ad4d8b1ab73316dff035eee3dab65b0fe9c2717c4",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/heartbeat_runs.ts",
|
||||
"beforeSha256": "8b4bfa006d999d4e27c8dc3678431544fb77d167e623b8421370571ab15b0f2d",
|
||||
"afterSha256": "8b4bfa006d999d4e27c8dc3678431544fb77d167e623b8421370571ab15b0f2d",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/heartbeat_run_events.ts",
|
||||
"beforeSha256": "f887d01bfb4173faa88f62f9df680b7710b57336221af70fc988e35968998388",
|
||||
"afterSha256": "f887d01bfb4173faa88f62f9df680b7710b57336221af70fc988e35968998388",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/heartbeat_run_event_cursors.ts",
|
||||
"beforeSha256": null,
|
||||
"afterSha256": null,
|
||||
"equal": true,
|
||||
"absentBoth": true
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/native_runner_checkpoints.ts",
|
||||
"beforeSha256": "5691cfd49e50434d3d9d3acf95cbeaa3436c1aa734876d0b3a29d2e731c50c64",
|
||||
"afterSha256": "5691cfd49e50434d3d9d3acf95cbeaa3436c1aa734876d0b3a29d2e731c50c64",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/native_run_results.ts",
|
||||
"beforeSha256": "a6b790e6dbf42a4eba34057738e7db6cc13e0e762f9d005db225733a8e37b449",
|
||||
"afterSha256": "a6b790e6dbf42a4eba34057738e7db6cc13e0e762f9d005db225733a8e37b449",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/native_run_finalizations.ts",
|
||||
"beforeSha256": "d9431cecdd839994a1b3fd142a8741d9cbee7da39e297265c269abc344cbf940",
|
||||
"afterSha256": "d9431cecdd839994a1b3fd142a8741d9cbee7da39e297265c269abc344cbf940",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-runner-checkpoint.ts",
|
||||
"beforeSha256": "3eb4c694b44ac9932d2c395438c1664d78bb1d00d8371359b062a1af5f039ad7",
|
||||
"afterSha256": "3eb4c694b44ac9932d2c395438c1664d78bb1d00d8371359b062a1af5f039ad7",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-runner-recovery.ts",
|
||||
"beforeSha256": null,
|
||||
"afterSha256": null,
|
||||
"equal": true,
|
||||
"absentBoth": true
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/live-events.ts",
|
||||
"beforeSha256": "63ddefcd3bbd7c5dcdc59e248d2d78f14f5b668d38b8585cac8a0d417e0ec115",
|
||||
"afterSha256": "63ddefcd3bbd7c5dcdc59e248d2d78f14f5b668d38b8585cac8a0d417e0ec115",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/canonical.ts",
|
||||
"beforeSha256": "6d2fa843f7710f55ced02715111d80e467a8a4ad0929cfcdef3a7bbaf7f3f993",
|
||||
"afterSha256": "6d2fa843f7710f55ced02715111d80e467a8a4ad0929cfcdef3a7bbaf7f3f993",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
}
|
||||
],
|
||||
"allPresentCorePathsByteEqual": true,
|
||||
"explicitOutsideCorePaths": [
|
||||
{
|
||||
"path": "server/src/vendor/paperclip-runner/index.ts",
|
||||
"beforeSha256": "991a7fd75ea23226a6c4c6a4b8992d6b28bd2c9825aca889b68e60e145c574c0",
|
||||
"afterSha256": "c14c8e4ab505164bc84c4d0305f614b2fd6a91cd6daf128ae98c9a9a63f54e6e",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/live/runnerd-codex-transport.ts",
|
||||
"beforeSha256": "3813d933b871ad15d052a8f03246b30bbaf03b989c3ac6436e184dda46186561",
|
||||
"afterSha256": "e8e878ac88f09673d6e33abba98c9841218a55522bd3d752171f5fb14a5c5f5a",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/adapter-utils/src/acpx-engine/execute.ts",
|
||||
"beforeSha256": "00635e57744c7407070618453852ede5002382159a81a4f31ceeac0551d3e725",
|
||||
"afterSha256": "f7e6dc787eaddb8767bd0b9f52d4ec50ec57c9add4b9fbe94ace6d4745343fdc",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/run-log-store.ts",
|
||||
"beforeSha256": "39c8da235b46a81ea478a6fa3a3e43acedd107cc4835c37912452451ffe050c0",
|
||||
"afterSha256": "b7ab36de4df505492ee109cbac377b359126a3a29c48db343e1ce82a44d9a488",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-run-finalizer.ts",
|
||||
"beforeSha256": "7706b8632519e6bfeb06e9b224e49a14fa1d8b1e4b0a1e3d5cdf025864cf29f8",
|
||||
"afterSha256": "07bc941d5796870fbd6d96e2a30f8409b389c4f1fc7232c745d1a83d376b9b4d",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-session-executor.ts",
|
||||
"beforeSha256": "0abf1aed7d1a8637e5469b4e8ee3a5a018e571c675078621dc30c0d54e09b48c",
|
||||
"afterSha256": "9f94c6d5b22b4a429c9d86eee8bb3f29644d3a45040730bbe4c8db3a5399e6e6",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"lane": "B1",
|
||||
"before": "e9b909015ad3d517f28c45f0c66c8f41d6112b4b",
|
||||
"after": "d8cd3be0aeed4f9fb93ecc3cf725f1d46d5a0310",
|
||||
"corePaths": [
|
||||
{
|
||||
"path": "server/src/services/heartbeat-run-events.ts",
|
||||
"beforeSha256": "9dfb7309a1e3b8feee654c645f13b603d48ebadc08a70783269caa8ef79daef5",
|
||||
"afterSha256": "9dfb7309a1e3b8feee654c645f13b603d48ebadc08a70783269caa8ef79daef5",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/heartbeat/native-event-cursors.ts",
|
||||
"beforeSha256": "3eda23353aff44b4c8b75a19f3628ebb59dd57874b5e6dee62425b6ee1305d42",
|
||||
"afterSha256": "3eda23353aff44b4c8b75a19f3628ebb59dd57874b5e6dee62425b6ee1305d42",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/heartbeat/native-event-cursor-work.ts",
|
||||
"beforeSha256": "b4ac1ed260a1cb66419a9b3e57f674aeb0bcf911c9a65a63e5a4c460a89522f9",
|
||||
"afterSha256": "b4ac1ed260a1cb66419a9b3e57f674aeb0bcf911c9a65a63e5a4c460a89522f9",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/paperclip-control-plane-port.ts",
|
||||
"beforeSha256": "b0b017e19b6808c600a18ea4f47d728e8d41608b3b26b5c42d4a2eafe0a8588c",
|
||||
"afterSha256": "b0b017e19b6808c600a18ea4f47d728e8d41608b3b26b5c42d4a2eafe0a8588c",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-run-coordinator-store.ts",
|
||||
"beforeSha256": "9cd659ee0d5f01d43fe2ba40c575b15ce65d2c535075a1d4d77ce45155676254",
|
||||
"afterSha256": "9cd659ee0d5f01d43fe2ba40c575b15ce65d2c535075a1d4d77ce45155676254",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/native-session-runtime.ts",
|
||||
"beforeSha256": "e916df31032785a9f4ce98df8d703c974391dbbb0ddf7a4e34ea43d231f9abdb",
|
||||
"afterSha256": "e916df31032785a9f4ce98df8d703c974391dbbb0ddf7a4e34ea43d231f9abdb",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/native-event-persistence.ts",
|
||||
"beforeSha256": null,
|
||||
"afterSha256": null,
|
||||
"equal": true,
|
||||
"absentBoth": true
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/contracts/control-plane-port.ts",
|
||||
"beforeSha256": "f200d8b9a9bc8cdcad918761e8ad3776663f4d129a8e20e8289b5148f6509923",
|
||||
"afterSha256": "f200d8b9a9bc8cdcad918761e8ad3776663f4d129a8e20e8289b5148f6509923",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts",
|
||||
"beforeSha256": "fb3ee3d87ad7824cc8a986bc433f00c73150a70f51e6f6dfe67ed5da976a350c",
|
||||
"afterSha256": "fb3ee3d87ad7824cc8a986bc433f00c73150a70f51e6f6dfe67ed5da976a350c",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/client.ts",
|
||||
"beforeSha256": "a4f89bf5ed8144c685d3e2113ca377f98734df8f68300f8d381ed5eaf509dfa0",
|
||||
"afterSha256": "a4f89bf5ed8144c685d3e2113ca377f98734df8f68300f8d381ed5eaf509dfa0",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/work-signals.ts",
|
||||
"beforeSha256": "f402f868c4cbbb19f78a8cfccc1cfe226d102a93ccff8efcf06a604ed09dc6f0",
|
||||
"afterSha256": "f402f868c4cbbb19f78a8cfccc1cfe226d102a93ccff8efcf06a604ed09dc6f0",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/run-event-payload.ts",
|
||||
"beforeSha256": "88b89c582d292ce91a1e053ad4d8b1ab73316dff035eee3dab65b0fe9c2717c4",
|
||||
"afterSha256": "88b89c582d292ce91a1e053ad4d8b1ab73316dff035eee3dab65b0fe9c2717c4",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/heartbeat_runs.ts",
|
||||
"beforeSha256": "d56a9a7a76933a9a2a117df939a6457440ff0b388ff6665a165fd2e756434e93",
|
||||
"afterSha256": "d56a9a7a76933a9a2a117df939a6457440ff0b388ff6665a165fd2e756434e93",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/heartbeat_run_events.ts",
|
||||
"beforeSha256": "f887d01bfb4173faa88f62f9df680b7710b57336221af70fc988e35968998388",
|
||||
"afterSha256": "f887d01bfb4173faa88f62f9df680b7710b57336221af70fc988e35968998388",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/heartbeat_run_event_cursors.ts",
|
||||
"beforeSha256": "2abce984dd65dfb275f5ce6545fcc6258c1d22d6e69374dead5caa27b1d96baf",
|
||||
"afterSha256": "2abce984dd65dfb275f5ce6545fcc6258c1d22d6e69374dead5caa27b1d96baf",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/native_runner_checkpoints.ts",
|
||||
"beforeSha256": "5691cfd49e50434d3d9d3acf95cbeaa3436c1aa734876d0b3a29d2e731c50c64",
|
||||
"afterSha256": "5691cfd49e50434d3d9d3acf95cbeaa3436c1aa734876d0b3a29d2e731c50c64",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/native_run_results.ts",
|
||||
"beforeSha256": "a6b790e6dbf42a4eba34057738e7db6cc13e0e762f9d005db225733a8e37b449",
|
||||
"afterSha256": "a6b790e6dbf42a4eba34057738e7db6cc13e0e762f9d005db225733a8e37b449",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/native_run_finalizations.ts",
|
||||
"beforeSha256": "d9431cecdd839994a1b3fd142a8741d9cbee7da39e297265c269abc344cbf940",
|
||||
"afterSha256": "d9431cecdd839994a1b3fd142a8741d9cbee7da39e297265c269abc344cbf940",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-runner-checkpoint.ts",
|
||||
"beforeSha256": "3eb4c694b44ac9932d2c395438c1664d78bb1d00d8371359b062a1af5f039ad7",
|
||||
"afterSha256": "3eb4c694b44ac9932d2c395438c1664d78bb1d00d8371359b062a1af5f039ad7",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-runner-recovery.ts",
|
||||
"beforeSha256": null,
|
||||
"afterSha256": null,
|
||||
"equal": true,
|
||||
"absentBoth": true
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/live-events.ts",
|
||||
"beforeSha256": "63ddefcd3bbd7c5dcdc59e248d2d78f14f5b668d38b8585cac8a0d417e0ec115",
|
||||
"afterSha256": "63ddefcd3bbd7c5dcdc59e248d2d78f14f5b668d38b8585cac8a0d417e0ec115",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/canonical.ts",
|
||||
"beforeSha256": "6d2fa843f7710f55ced02715111d80e467a8a4ad0929cfcdef3a7bbaf7f3f993",
|
||||
"afterSha256": "6d2fa843f7710f55ced02715111d80e467a8a4ad0929cfcdef3a7bbaf7f3f993",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
}
|
||||
],
|
||||
"allPresentCorePathsByteEqual": true,
|
||||
"explicitOutsideCorePaths": [
|
||||
{
|
||||
"path": "server/src/vendor/paperclip-runner/index.ts",
|
||||
"beforeSha256": "991a7fd75ea23226a6c4c6a4b8992d6b28bd2c9825aca889b68e60e145c574c0",
|
||||
"afterSha256": "c14c8e4ab505164bc84c4d0305f614b2fd6a91cd6daf128ae98c9a9a63f54e6e",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/live/runnerd-codex-transport.ts",
|
||||
"beforeSha256": "3813d933b871ad15d052a8f03246b30bbaf03b989c3ac6436e184dda46186561",
|
||||
"afterSha256": "e8e878ac88f09673d6e33abba98c9841218a55522bd3d752171f5fb14a5c5f5a",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/adapter-utils/src/acpx-engine/execute.ts",
|
||||
"beforeSha256": "00635e57744c7407070618453852ede5002382159a81a4f31ceeac0551d3e725",
|
||||
"afterSha256": "f7e6dc787eaddb8767bd0b9f52d4ec50ec57c9add4b9fbe94ace6d4745343fdc",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/run-log-store.ts",
|
||||
"beforeSha256": "39c8da235b46a81ea478a6fa3a3e43acedd107cc4835c37912452451ffe050c0",
|
||||
"afterSha256": "b7ab36de4df505492ee109cbac377b359126a3a29c48db343e1ce82a44d9a488",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-run-finalizer.ts",
|
||||
"beforeSha256": "7706b8632519e6bfeb06e9b224e49a14fa1d8b1e4b0a1e3d5cdf025864cf29f8",
|
||||
"afterSha256": "07bc941d5796870fbd6d96e2a30f8409b389c4f1fc7232c745d1a83d376b9b4d",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-session-executor.ts",
|
||||
"beforeSha256": "0abf1aed7d1a8637e5469b4e8ee3a5a018e571c675078621dc30c0d54e09b48c",
|
||||
"afterSha256": "9f94c6d5b22b4a429c9d86eee8bb3f29644d3a45040730bbe4c8db3a5399e6e6",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"lane": "B2",
|
||||
"before": "fcb5d94848b3c4b0b32096ff5e023618e0f0637b",
|
||||
"after": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da",
|
||||
"corePaths": [
|
||||
{
|
||||
"path": "server/src/services/heartbeat-run-events.ts",
|
||||
"beforeSha256": "699f1eda0806de94b49ea11f2dee56e9a3a7687eb94050feec2fa41e60089aed",
|
||||
"afterSha256": "699f1eda0806de94b49ea11f2dee56e9a3a7687eb94050feec2fa41e60089aed",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/heartbeat/native-event-cursors.ts",
|
||||
"beforeSha256": "3eda23353aff44b4c8b75a19f3628ebb59dd57874b5e6dee62425b6ee1305d42",
|
||||
"afterSha256": "3eda23353aff44b4c8b75a19f3628ebb59dd57874b5e6dee62425b6ee1305d42",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/heartbeat/native-event-cursor-work.ts",
|
||||
"beforeSha256": "b4ac1ed260a1cb66419a9b3e57f674aeb0bcf911c9a65a63e5a4c460a89522f9",
|
||||
"afterSha256": "b4ac1ed260a1cb66419a9b3e57f674aeb0bcf911c9a65a63e5a4c460a89522f9",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/paperclip-control-plane-port.ts",
|
||||
"beforeSha256": "d5dee911af746f010d78162f02c3f866892e41c41fb41fcb8fbeeefe3a816573",
|
||||
"afterSha256": "d5dee911af746f010d78162f02c3f866892e41c41fb41fcb8fbeeefe3a816573",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-run-coordinator-store.ts",
|
||||
"beforeSha256": "9cd659ee0d5f01d43fe2ba40c575b15ce65d2c535075a1d4d77ce45155676254",
|
||||
"afterSha256": "9cd659ee0d5f01d43fe2ba40c575b15ce65d2c535075a1d4d77ce45155676254",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/native-session-runtime.ts",
|
||||
"beforeSha256": "fc46351d0e554c970715da291768d148c6fed7cd44e2401f00293001cd85c627",
|
||||
"afterSha256": "fc46351d0e554c970715da291768d148c6fed7cd44e2401f00293001cd85c627",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/native-event-persistence.ts",
|
||||
"beforeSha256": "1237704030d270992612c06cfd4f9872145f28c7caaf2fbd130378f31d84216d",
|
||||
"afterSha256": "1237704030d270992612c06cfd4f9872145f28c7caaf2fbd130378f31d84216d",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/contracts/control-plane-port.ts",
|
||||
"beforeSha256": "96a35ce3208f9edbc5aac26dbef0e44184a14906953af43f7506ac23f1b255ec",
|
||||
"afterSha256": "96a35ce3208f9edbc5aac26dbef0e44184a14906953af43f7506ac23f1b255ec",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts",
|
||||
"beforeSha256": "fb3ee3d87ad7824cc8a986bc433f00c73150a70f51e6f6dfe67ed5da976a350c",
|
||||
"afterSha256": "fb3ee3d87ad7824cc8a986bc433f00c73150a70f51e6f6dfe67ed5da976a350c",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/client.ts",
|
||||
"beforeSha256": "a4f89bf5ed8144c685d3e2113ca377f98734df8f68300f8d381ed5eaf509dfa0",
|
||||
"afterSha256": "a4f89bf5ed8144c685d3e2113ca377f98734df8f68300f8d381ed5eaf509dfa0",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/work-signals.ts",
|
||||
"beforeSha256": "f402f868c4cbbb19f78a8cfccc1cfe226d102a93ccff8efcf06a604ed09dc6f0",
|
||||
"afterSha256": "f402f868c4cbbb19f78a8cfccc1cfe226d102a93ccff8efcf06a604ed09dc6f0",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/run-event-payload.ts",
|
||||
"beforeSha256": "88b89c582d292ce91a1e053ad4d8b1ab73316dff035eee3dab65b0fe9c2717c4",
|
||||
"afterSha256": "88b89c582d292ce91a1e053ad4d8b1ab73316dff035eee3dab65b0fe9c2717c4",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/heartbeat_runs.ts",
|
||||
"beforeSha256": "d56a9a7a76933a9a2a117df939a6457440ff0b388ff6665a165fd2e756434e93",
|
||||
"afterSha256": "d56a9a7a76933a9a2a117df939a6457440ff0b388ff6665a165fd2e756434e93",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/heartbeat_run_events.ts",
|
||||
"beforeSha256": "f887d01bfb4173faa88f62f9df680b7710b57336221af70fc988e35968998388",
|
||||
"afterSha256": "f887d01bfb4173faa88f62f9df680b7710b57336221af70fc988e35968998388",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/heartbeat_run_event_cursors.ts",
|
||||
"beforeSha256": "2abce984dd65dfb275f5ce6545fcc6258c1d22d6e69374dead5caa27b1d96baf",
|
||||
"afterSha256": "2abce984dd65dfb275f5ce6545fcc6258c1d22d6e69374dead5caa27b1d96baf",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/native_runner_checkpoints.ts",
|
||||
"beforeSha256": "5691cfd49e50434d3d9d3acf95cbeaa3436c1aa734876d0b3a29d2e731c50c64",
|
||||
"afterSha256": "5691cfd49e50434d3d9d3acf95cbeaa3436c1aa734876d0b3a29d2e731c50c64",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/native_run_results.ts",
|
||||
"beforeSha256": "a6b790e6dbf42a4eba34057738e7db6cc13e0e762f9d005db225733a8e37b449",
|
||||
"afterSha256": "a6b790e6dbf42a4eba34057738e7db6cc13e0e762f9d005db225733a8e37b449",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/db/src/schema/native_run_finalizations.ts",
|
||||
"beforeSha256": "d9431cecdd839994a1b3fd142a8741d9cbee7da39e297265c269abc344cbf940",
|
||||
"afterSha256": "d9431cecdd839994a1b3fd142a8741d9cbee7da39e297265c269abc344cbf940",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-runner-checkpoint.ts",
|
||||
"beforeSha256": "3eb4c694b44ac9932d2c395438c1664d78bb1d00d8371359b062a1af5f039ad7",
|
||||
"afterSha256": "3eb4c694b44ac9932d2c395438c1664d78bb1d00d8371359b062a1af5f039ad7",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-runner-recovery.ts",
|
||||
"beforeSha256": null,
|
||||
"afterSha256": null,
|
||||
"equal": true,
|
||||
"absentBoth": true
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/live-events.ts",
|
||||
"beforeSha256": "63ddefcd3bbd7c5dcdc59e248d2d78f14f5b668d38b8585cac8a0d417e0ec115",
|
||||
"afterSha256": "63ddefcd3bbd7c5dcdc59e248d2d78f14f5b668d38b8585cac8a0d417e0ec115",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/canonical.ts",
|
||||
"beforeSha256": "6d2fa843f7710f55ced02715111d80e467a8a4ad0929cfcdef3a7bbaf7f3f993",
|
||||
"afterSha256": "6d2fa843f7710f55ced02715111d80e467a8a4ad0929cfcdef3a7bbaf7f3f993",
|
||||
"equal": true,
|
||||
"absentBoth": false
|
||||
}
|
||||
],
|
||||
"allPresentCorePathsByteEqual": true,
|
||||
"explicitOutsideCorePaths": [
|
||||
{
|
||||
"path": "server/src/vendor/paperclip-runner/index.ts",
|
||||
"beforeSha256": "991a7fd75ea23226a6c4c6a4b8992d6b28bd2c9825aca889b68e60e145c574c0",
|
||||
"afterSha256": "c14c8e4ab505164bc84c4d0305f614b2fd6a91cd6daf128ae98c9a9a63f54e6e",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/paperclip-runner/src/live/runnerd-codex-transport.ts",
|
||||
"beforeSha256": "3813d933b871ad15d052a8f03246b30bbaf03b989c3ac6436e184dda46186561",
|
||||
"afterSha256": "e8e878ac88f09673d6e33abba98c9841218a55522bd3d752171f5fb14a5c5f5a",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "packages/adapter-utils/src/acpx-engine/execute.ts",
|
||||
"beforeSha256": "00635e57744c7407070618453852ede5002382159a81a4f31ceeac0551d3e725",
|
||||
"afterSha256": "f7e6dc787eaddb8767bd0b9f52d4ec50ec57c9add4b9fbe94ace6d4745343fdc",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/run-log-store.ts",
|
||||
"beforeSha256": "39c8da235b46a81ea478a6fa3a3e43acedd107cc4835c37912452451ffe050c0",
|
||||
"afterSha256": "b7ab36de4df505492ee109cbac377b359126a3a29c48db343e1ce82a44d9a488",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-run-finalizer.ts",
|
||||
"beforeSha256": "7706b8632519e6bfeb06e9b224e49a14fa1d8b1e4b0a1e3d5cdf025864cf29f8",
|
||||
"afterSha256": "07bc941d5796870fbd6d96e2a30f8409b389c4f1fc7232c745d1a83d376b9b4d",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
},
|
||||
{
|
||||
"path": "server/src/services/native-runtime/native-session-executor.ts",
|
||||
"beforeSha256": "0abf1aed7d1a8637e5469b4e8ee3a5a018e571c675078621dc30c0d54e09b48c",
|
||||
"afterSha256": "9f94c6d5b22b4a429c9d86eee8bb3f29644d3a45040730bbe4c8db3a5399e6e6",
|
||||
"equal": false,
|
||||
"absentBoth": false
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"limits": [
|
||||
"Extra vendor index exports, Runner Codex diagnostics and ACPX environment/Claude-key handling differ outside this core selection",
|
||||
"Run-log wrapper, native executor and native finalizer are real application-path changes; fresh app/Stop proof remains separate",
|
||||
"Locked DB/pg/tsx graph comparisons are separate static receipts, not installed-binary/OS/build/runtime equality",
|
||||
"Heartbeat event cursor/range table declarations share one schema source; native checkpoints/results/finalizations and heartbeatRuns are separately compared"
|
||||
]
|
||||
}
|
||||
+2803
File diff suppressed because it is too large.
Load diff
+2803
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,559 @@
|
||||
{
|
||||
"scope": "Real source append service only, active append interval sums under preregistered8x64 interleaving; no browser/batching/production claim",
|
||||
"baselineRevision": "5337df423e622755d44f3ece5447d1e0beef5f40",
|
||||
"candidateRevision": "e9b909015ad3d517f28c45f0c66c8f41d6112b4b",
|
||||
"measurementSha256": "c8adaf5ac16cab67513d6c8b888afa44d95fa3688d4831b8126fe3d997f1d039",
|
||||
"protocolSha256": "302c11ec61d5170822c08e47f798ca582b8e5afefd9d223fb9023d9ead7a2ad3",
|
||||
"workloadSha256": "ca4894b0723d0bffff0fc88d1596d64aa8fd75aa1dd8d861372ea993db24e164",
|
||||
"gateSha256": "ded8bcbb8f05732348f6520b9c5e41b7c35bc098997eb263a9dcb174eede75fc",
|
||||
"aaRepeatabilityPassed": true,
|
||||
"aaMaximumPairedNoiseAcrossNineCasesPercent": 11.830110496052082,
|
||||
"aaMaximumMedianDriftAcrossNineCasesPercent": 6.132813294289431,
|
||||
"samplesPerCaseArm": 5,
|
||||
"traceCompleteness": {
|
||||
"service-aa-b0": {
|
||||
"recordTypes": {
|
||||
"run-start": 1,
|
||||
"warmup-start": 36,
|
||||
"warmup-complete": 36,
|
||||
"pair-plan": 60,
|
||||
"block-start": 750,
|
||||
"block-complete": 750,
|
||||
"final-verification-start": 120,
|
||||
"pair-complete": 60,
|
||||
"cleanup-complete": 1
|
||||
},
|
||||
"measuredAppendAttempts": 46110,
|
||||
"measuredNonLateGapAttempts": 46080,
|
||||
"measuredLateGapAttempts": 30,
|
||||
"excludedWarmupAttempts": 18432
|
||||
},
|
||||
"service-b0-b1": {
|
||||
"recordTypes": {
|
||||
"run-start": 1,
|
||||
"warmup-start": 36,
|
||||
"warmup-complete": 36,
|
||||
"pair-plan": 60,
|
||||
"block-start": 750,
|
||||
"block-complete": 750,
|
||||
"final-verification-start": 120,
|
||||
"pair-complete": 60,
|
||||
"cleanup-complete": 1
|
||||
},
|
||||
"measuredAppendAttempts": 46110,
|
||||
"measuredNonLateGapAttempts": 46080,
|
||||
"measuredLateGapAttempts": 30,
|
||||
"excludedWarmupAttempts": 18432
|
||||
}
|
||||
},
|
||||
"cases": [
|
||||
{
|
||||
"history": 1000,
|
||||
"scenario": "ordered",
|
||||
"baselineMedianActiveMs": 3823.2426909999995,
|
||||
"candidateMedianActiveMs": 3554.4297090000036,
|
||||
"baselineMinMaxMs": [
|
||||
3660.1935169999997,
|
||||
3890.539190999989
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
3522.6579420000126,
|
||||
3603.657418999999
|
||||
],
|
||||
"reductionPercent": 7.0310206211284365,
|
||||
"pairedBeyondNoise": 5,
|
||||
"noisePercent": 1.4079047972212446,
|
||||
"gate": "nonregression",
|
||||
"passed": true,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 5,
|
||||
"candidate": 5
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"history": 1000,
|
||||
"scenario": "replay",
|
||||
"baselineMedianActiveMs": 3164.471254999982,
|
||||
"candidateMedianActiveMs": 3002.7086030000064,
|
||||
"baselineMinMaxMs": [
|
||||
3069.5773009999975,
|
||||
3184.937432000006
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
2943.1245840000047,
|
||||
3046.233085
|
||||
],
|
||||
"reductionPercent": 5.1118382492615355,
|
||||
"pairedBeyondNoise": 3,
|
||||
"noisePercent": 3.9822120206785225,
|
||||
"gate": "nonregression",
|
||||
"passed": true,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
256,
|
||||
256
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
256,
|
||||
256
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 4,
|
||||
"candidate": 4
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"history": 1000,
|
||||
"scenario": "late-gap",
|
||||
"baselineMedianActiveMs": 7.402558999994653,
|
||||
"candidateMedianActiveMs": 9.506674999996903,
|
||||
"baselineMinMaxMs": [
|
||||
4.523847000004025,
|
||||
8.067190000001574
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
8.462973000001512,
|
||||
11.00232299999334
|
||||
],
|
||||
"reductionPercent": -28.424170614564105,
|
||||
"pairedBeyondNoise": 0,
|
||||
"noisePercent": 15.945874769827368,
|
||||
"gate": "structural-only",
|
||||
"passed": null,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
1,
|
||||
0
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
1,
|
||||
0
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 5,
|
||||
"candidate": 7
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"history": 1000,
|
||||
"scenario": "warm-suffix",
|
||||
"baselineMedianActiveMs": 4152.802396999992,
|
||||
"candidateMedianActiveMs": 3542.40763099998,
|
||||
"baselineMinMaxMs": [
|
||||
3944.2845539999835,
|
||||
4266.957552999986
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
3468.830302999995,
|
||||
3605.3644960000674
|
||||
],
|
||||
"reductionPercent": 14.698382144090571,
|
||||
"pairedBeyondNoise": 5,
|
||||
"noisePercent": 1.8842439321434798,
|
||||
"gate": "nonregression",
|
||||
"passed": true,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 6,
|
||||
"candidate": 5
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"history": 10000,
|
||||
"scenario": "ordered",
|
||||
"baselineMedianActiveMs": 4209.649118000001,
|
||||
"candidateMedianActiveMs": 3530.6375980000594,
|
||||
"baselineMinMaxMs": [
|
||||
4049.0527909999655,
|
||||
4255.424076000025
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
3528.6497000000672,
|
||||
3597.892884000001
|
||||
],
|
||||
"reductionPercent": 16.12988401091584,
|
||||
"pairedBeyondNoise": 5,
|
||||
"noisePercent": 1.7474622955774097,
|
||||
"gate": "improvement",
|
||||
"passed": true,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 5,
|
||||
"candidate": 5
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"history": 10000,
|
||||
"scenario": "replay",
|
||||
"baselineMedianActiveMs": 3575.641203999985,
|
||||
"candidateMedianActiveMs": 3016.318643000035,
|
||||
"baselineMinMaxMs": [
|
||||
3542.3359069999424,
|
||||
3624.188894000021
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
2948.329679999937,
|
||||
3070.961685999995
|
||||
],
|
||||
"reductionPercent": 15.642580703406395,
|
||||
"pairedBeyondNoise": 5,
|
||||
"noisePercent": 1.6304743570125815,
|
||||
"gate": "improvement",
|
||||
"passed": true,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
256,
|
||||
256
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
256,
|
||||
256
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 4,
|
||||
"candidate": 4
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"history": 10000,
|
||||
"scenario": "late-gap",
|
||||
"baselineMedianActiveMs": 7.701110000023618,
|
||||
"candidateMedianActiveMs": 8.573146000009729,
|
||||
"baselineMinMaxMs": [
|
||||
7.188416999997571,
|
||||
9.012380000000121
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
6.192671999975573,
|
||||
8.915757999988273
|
||||
],
|
||||
"reductionPercent": -11.32351050671183,
|
||||
"pairedBeyondNoise": 0,
|
||||
"noisePercent": 23.08985200646279,
|
||||
"gate": "structural-only",
|
||||
"passed": null,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
1,
|
||||
0
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
1,
|
||||
0
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 5,
|
||||
"candidate": 7
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"history": 10000,
|
||||
"scenario": "warm-suffix",
|
||||
"baselineMedianActiveMs": 4614.03645500011,
|
||||
"candidateMedianActiveMs": 3592.809363000095,
|
||||
"baselineMinMaxMs": [
|
||||
4464.443957999989,
|
||||
4736.473729000078
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
3571.7655579999555,
|
||||
3615.229942000122
|
||||
],
|
||||
"reductionPercent": 22.133052089203552,
|
||||
"pairedBeyondNoise": 5,
|
||||
"noisePercent": 11.830110496052082,
|
||||
"gate": "improvement",
|
||||
"passed": true,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 6,
|
||||
"candidate": 5
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"history": 100000,
|
||||
"scenario": "ordered",
|
||||
"baselineMedianActiveMs": 8029.154580999981,
|
||||
"candidateMedianActiveMs": 3540.048427999951,
|
||||
"baselineMinMaxMs": [
|
||||
7985.540289000026,
|
||||
8157.268998000014
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
3503.1252190000378,
|
||||
3586.664618000097
|
||||
],
|
||||
"reductionPercent": 55.91007257006802,
|
||||
"pairedBeyondNoise": 5,
|
||||
"noisePercent": 1.0693214938958222,
|
||||
"gate": "improvement",
|
||||
"passed": true,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 5,
|
||||
"candidate": 5
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"history": 100000,
|
||||
"scenario": "replay",
|
||||
"baselineMedianActiveMs": 7456.020750999975,
|
||||
"candidateMedianActiveMs": 2989.0107820000267,
|
||||
"baselineMinMaxMs": [
|
||||
7350.074664000131,
|
||||
7471.52871100005
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
2938.5112860000227,
|
||||
3000.5278340000077
|
||||
],
|
||||
"reductionPercent": 59.911447649885474,
|
||||
"pairedBeyondNoise": 5,
|
||||
"noisePercent": 0.9332642395176317,
|
||||
"gate": "improvement",
|
||||
"passed": true,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
256,
|
||||
256
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
256,
|
||||
256
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 4,
|
||||
"candidate": 4
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"history": 100000,
|
||||
"scenario": "late-gap",
|
||||
"baselineMedianActiveMs": 24.45493699994404,
|
||||
"candidateMedianActiveMs": 9.347156999981962,
|
||||
"baselineMinMaxMs": [
|
||||
23.453166999970563,
|
||||
27.683694000006653
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
8.141280999989249,
|
||||
9.645160000072792
|
||||
],
|
||||
"reductionPercent": 61.77803688472658,
|
||||
"pairedBeyondNoise": 5,
|
||||
"noisePercent": 39.768871890266304,
|
||||
"gate": "structural-only",
|
||||
"passed": null,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
1,
|
||||
0
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
1,
|
||||
0
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 5,
|
||||
"candidate": 7
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"history": 100000,
|
||||
"scenario": "warm-suffix",
|
||||
"baselineMedianActiveMs": 8347.43800100009,
|
||||
"candidateMedianActiveMs": 3402.361554999836,
|
||||
"baselineMinMaxMs": [
|
||||
7836.916214999976,
|
||||
8611.93426499993
|
||||
],
|
||||
"candidateMinMaxMs": [
|
||||
3119.3996940000216,
|
||||
3627.8794139999663
|
||||
],
|
||||
"reductionPercent": 59.240648992035574,
|
||||
"pairedBeyondNoise": 5,
|
||||
"noisePercent": 0.7615089382403567,
|
||||
"gate": "improvement",
|
||||
"passed": true,
|
||||
"newDuplicateCountsPerSample": {
|
||||
"baseline": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
],
|
||||
"candidate": [
|
||||
[
|
||||
512,
|
||||
0
|
||||
]
|
||||
]
|
||||
},
|
||||
"diagnosticSqlCallsPerAttempt": {
|
||||
"baseline": 6,
|
||||
"candidate": 5
|
||||
},
|
||||
"diagnosticTransactionsPerAttempt": {
|
||||
"baseline": 1,
|
||||
"candidate": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"limitations": {
|
||||
"warmups": 2,
|
||||
"pairedOrder": "8x64 blocks interleaved; first arm alternates per pair",
|
||||
"cache": "warm; seed and projection preparation excluded",
|
||||
"databaseCpuResolutionMillis": 10,
|
||||
"databaseCpuSource": "/proc/PID/stat utime+stime clock ticks",
|
||||
"databaseCpuAggregateWorstQuantizationMillis": 80,
|
||||
"databaseCpuCaution": "Each block is tick-quantized (typically 10ms on Linux); an eight-block sum has up to eight ticks of quantization uncertainty. Do not infer sub-tick CPU precision.",
|
||||
"memoryCaution": "Heap deltas, RSS and process CPU describe shared-process intervals; heap deltas do not identify owned allocations.",
|
||||
"activeInterval": "invoke() including append receipt checks, excluding generator, CPU probes, independent verification, trace writes and inter-arm gaps",
|
||||
"pairInterval": "pair plan through final independent database verification; final pair-complete log and scenario diagnostics are outside this interval"
|
||||
},
|
||||
"excludedPriorMonolithicAA": {
|
||||
"sha256": "cfcfeb3c12812dbe48c2afe6013320922d5670a410d033d6ad9a3a2f91236ef8",
|
||||
"status": "failed unchanged noise gates; notcalibration for thisschedule; no specific warmupcause inferred"
|
||||
}
|
||||
}
|
||||
+2803
File diff suppressed because it is too large.
Load diff
+29
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"scope": "Recorded migration snapshot equality in upstream normal merge; separate from direct SOURCE\u2192APP selected-source checks",
|
||||
"before": "75dbd6497438baeac0b7060ce1c31283613dddaa",
|
||||
"after": "d8cd3be0aeed4f9fb93ecc3cf725f1d46d5a0310",
|
||||
"upstream": "ce332ecec0fa165c75261946d238a9035fd2ac71",
|
||||
"tables": {
|
||||
"public.heartbeat_runs": {
|
||||
"beforeSha256": "70def29c6c9d97093071356343b970e9f65130b7c2fba4795f2ec7ca7d31d9a3",
|
||||
"afterSha256": "70def29c6c9d97093071356343b970e9f65130b7c2fba4795f2ec7ca7d31d9a3",
|
||||
"equal": true
|
||||
},
|
||||
"public.heartbeat_run_events": {
|
||||
"beforeSha256": "ea4139388b273d3108c4e2e730d7689591402ef43eb93181a30ae4e0f988b6bd",
|
||||
"afterSha256": "ea4139388b273d3108c4e2e730d7689591402ef43eb93181a30ae4e0f988b6bd",
|
||||
"equal": true
|
||||
},
|
||||
"public.heartbeat_run_event_cursors": {
|
||||
"beforeSha256": "fa883d4daffca32538001953a52013e29633f26ce03dfc92f3662be61fae3069",
|
||||
"afterSha256": "fa883d4daffca32538001953a52013e29633f26ce03dfc92f3662be61fae3069",
|
||||
"equal": true
|
||||
},
|
||||
"public.heartbeat_run_event_source_ranges": {
|
||||
"beforeSha256": "6f2e13fdb9444a87cceba49276c2dbfe465c82d7ad85308a6b7100f360a5eb90",
|
||||
"afterSha256": "6f2e13fdb9444a87cceba49276c2dbfe465c82d7ad85308a6b7100f360a5eb90",
|
||||
"equal": true
|
||||
}
|
||||
},
|
||||
"originalManifestSha256": "75161e6233f1e425ae55ff7186d0c46cae9864042365b995f4211b751d118204"
|
||||
}
|
||||
@@ -0,0 +1,703 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { promises as fs } from "node:fs";
|
||||
import { createRequire } from "node:module";
|
||||
import { availableParallelism, loadavg, tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { monitorEventLoopDelay, performance } from "node:perf_hooks";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import type { Db } from "../packages/db/src/index.js";
|
||||
import type { AppendHeartbeatRunEventInput, AppendHeartbeatRunEventResult } from "../server/src/services/heartbeat-run-events.js";
|
||||
|
||||
// The exact bytes of this file may be copied outside APP_ROOT as .mts. Only
|
||||
// erased type imports are relative to the measurement file; runtime support is
|
||||
// always owned by the explicitly selected, frozen application checkout.
|
||||
const arg = (name: string) => process.argv.find(value => value.startsWith(`--${name}=`))?.slice(name.length + 3);
|
||||
const measurementFile = fileURLToPath(import.meta.url);
|
||||
const actualSelfSha256 = createHash("sha256").update(await fs.readFile(measurementFile)).digest("hex");
|
||||
const root = path.resolve(arg("app-root") ?? fileURLToPath(new URL("../", import.meta.url)));
|
||||
const appImport = (file: string) => import(pathToFileURL(path.join(root, file)).href);
|
||||
const serverRequire = createRequire(path.join(root, "server/package.json"));
|
||||
const gatesPath = "scripts/native-ingestion-performance-gates.mjs";
|
||||
const frozenGatesSha256 = "ded8bcbb8f05732348f6520b9c5e41b7c35bc098997eb263a9dcb174eede75fc";
|
||||
const gatesSha256 = createHash("sha256").update(await fs.readFile(path.join(root, gatesPath))).digest("hex");
|
||||
assert.equal(gatesSha256, frozenGatesSha256, "The preregistered acceptance gates must remain unchanged");
|
||||
const { evaluateServiceRepeatability, noiseExceedingPairedWins } = await appImport(gatesPath);
|
||||
const sourcePath = "server/src/services/heartbeat-run-events.ts";
|
||||
const cursorSourcePath = "server/src/services/heartbeat/native-event-cursors.ts";
|
||||
const git = (args: string[]) => execFileSync("git", args, { cwd: root, encoding: "utf8" });
|
||||
const quick = process.argv.includes("--quick");
|
||||
const phase = arg("phase") ?? "compare";
|
||||
assert.ok(["aa", "compare", "batch"].includes(phase), "--phase must be aa, compare, or batch");
|
||||
if (process.argv.includes("--require-repeatability")) assert.equal(phase, "aa", "Repeatability requires the A/A phase");
|
||||
const baselineRef = arg("baseline-ref");
|
||||
assert.ok(baselineRef, "Pass a pinned --baseline-ref=<commit>, including report validation");
|
||||
const baselineRevision = git(["rev-parse", "--verify", `${baselineRef}^{commit}`]).trim();
|
||||
const histories = (arg("histories") ?? (quick ? "1000" : "1000,10000,100000")).split(",").map(Number);
|
||||
const pairs = Number(arg("pairs") ?? 5);
|
||||
const events = Number(arg("events") ?? 32);
|
||||
const groups = Number(arg("groups") ?? 16);
|
||||
assert.deepEqual(histories, quick ? [1000] : [1000, 10000, 100000], "Retained-history matrix is frozen");
|
||||
assert.equal(pairs, 5, "The prospective schedule requires five measured pairs");
|
||||
assert.equal(events, 32, "Generator groups retain 32 attempts");
|
||||
assert.equal(groups, 16, "Each non-late-gap sample retains 16 generator groups");
|
||||
const warmups = 2, blockAttempts = 64, blocksPerArm = 8, groupsPerBlock = blockAttempts / events;
|
||||
const scenarios = quick ? ["ordered"] : ["ordered", "replay", "late-gap", "warm-suffix"];
|
||||
const workload = { histories, pairs, events, groups, scenarios, payloadTextBytes: 128, warmSourceOffset: 1000,
|
||||
seed: "native-event-ingestion-v1", retainedHistory: "vacuum-analyzed", quick,
|
||||
schedule: "interleaved-8x64-v1", blockAttempts, blocksPerArm, groupsPerBlock, warmups,
|
||||
firstArm: "alternating-per-pair", estimand: "per-arm sum of active append intervals" };
|
||||
const workloadSha256 = createHash("sha256").update(JSON.stringify(workload)).digest("hex");
|
||||
|
||||
function assertProtocolIdentity(recorded: any) {
|
||||
assert.equal(recorded.actualSelfSha256, actualSelfSha256, "Measurement bytes changed; prior evidence cannot qualify this file");
|
||||
assert.equal(recorded.workloadSha256, workloadSha256, "Workload or schedule changed; rerun A/A");
|
||||
assert.equal(createHash("sha256").update(JSON.stringify(recorded.workload)).digest("hex"), workloadSha256, "Recorded workload does not match its hash");
|
||||
assert.equal(recorded.baselineRevision, baselineRevision, "Exact pinned baseline changed; rerun A/A");
|
||||
assert.equal(recorded.gatesSha256, gatesSha256, "Acceptance gates changed; rerun A/A");
|
||||
assert.equal(recorded.candidateSourceHashes?.[gatesPath], gatesSha256, "Recorded gate source must match APP_ROOT");
|
||||
assert.ok(recorded.candidateSourceHashes?.[sourcePath], "Report must hash the actual APP_ROOT append service");
|
||||
}
|
||||
function assertServiceAcceptance(recorded: any) {
|
||||
assertProtocolIdentity(recorded);
|
||||
assert.equal(recorded.cleanupComplete, true, "Owned benchmark resources have not finished cleanup");
|
||||
assert.ok(["compare", "batch"].includes(recorded.phase), "A/A calibration is not performance acceptance");
|
||||
assert.equal(recorded.workload?.quick, false, "Qualification smoke cannot establish acceptance");
|
||||
assert.ok(recorded.workload.pairs >= 5, "Acceptance requires at least five paired samples");
|
||||
assert.ok([1000, 10000, 100000].every(history => recorded.workload.histories.includes(history)), "Acceptance requires the complete retained-history matrix");
|
||||
assert.ok(["ordered", "replay", "late-gap", "warm-suffix"].every(scenario => recorded.workload.scenarios.includes(scenario)), "Acceptance requires every fixed service scenario");
|
||||
assert.ok(recorded.comparison?.length > 0, "Acceptance requires a same-workload A/A noise comparison");
|
||||
assert.ok(recorded.comparison.every((entry: { acceptancePassed: boolean | null }) => entry.acceptancePassed !== false), "Service performance gate failed; inspect comparison and retained failures");
|
||||
}
|
||||
if (arg("validate-report")) {
|
||||
const recorded = JSON.parse(await fs.readFile(arg("validate-report")!, "utf8"));
|
||||
assertProtocolIdentity(recorded);
|
||||
assert.equal(recorded.incomplete, false, "Incomplete evidence cannot qualify a revision");
|
||||
assert.ok(Object.keys(recorded.candidateSourceHashes ?? {}).length > 0, "Report must include supporting source hashes");
|
||||
for (const [file, expected] of Object.entries(recorded.candidateSourceHashes)) {
|
||||
assert.equal(createHash("sha256").update(await fs.readFile(path.join(root, file))).digest("hex"), expected, `Measured source changed: ${file}`);
|
||||
}
|
||||
if (process.argv.includes("--require-acceptance")) {
|
||||
assertServiceAcceptance(recorded);
|
||||
}
|
||||
if (process.argv.includes("--require-repeatability")) {
|
||||
assert.equal(evaluateServiceRepeatability(recorded).passed, true, "A/A repeatability gate failed; retain evidence and diagnose before candidate measurements");
|
||||
}
|
||||
console.log(JSON.stringify({ valid: true, phase: recorded.phase, baselineRevision: recorded.baselineRevision,
|
||||
candidateRevision: recorded.candidateRevision, scope: recorded.scope }));
|
||||
process.exit(0);
|
||||
}
|
||||
if (process.argv.includes("--check-source")) {
|
||||
// Source-only smoke: import the unchanged gates and resolve support files,
|
||||
// without importing the DB client, starting PostgreSQL or creating fixtures.
|
||||
for (const file of ["packages/db/src/index.ts", "packages/db/src/work-signals.ts", sourcePath,
|
||||
"server/src/services/native-runtime/canonical.ts", "server/src/__tests__/helpers/external-test-database.ts"]) {
|
||||
await fs.access(path.join(root, file));
|
||||
}
|
||||
for (const specifier of ["drizzle-orm/postgres-js", "@paperclipai/db", "drizzle-orm"]) serverRequire.resolve(specifier);
|
||||
console.log(JSON.stringify({ sourceOnly: true, appRoot: root, measurementFile, actualSelfSha256,
|
||||
workloadSha256, baselineRevision, gatesSha256, quick, performanceAcceptanceEligible: false }));
|
||||
process.exit(0);
|
||||
}
|
||||
const baselineSource = git(["show", `${baselineRevision}:${sourcePath}`]);
|
||||
const candidateRevision = git(["rev-parse", "HEAD"]).trim();
|
||||
const { agents, applyPendingMigrations, companies, createDb, heartbeatRuns, startEmbeddedPostgresTestDatabase } =
|
||||
await appImport("packages/db/src/index.ts") as typeof import("../packages/db/src/index.js");
|
||||
const { installDatabaseWorkSignals } = await appImport("packages/db/src/work-signals.ts") as typeof import("../packages/db/src/work-signals.js");
|
||||
const { resolveExternalTestDatabaseUrl } = await appImport("server/src/__tests__/helpers/external-test-database.ts") as typeof import("../server/src/__tests__/helpers/external-test-database.js");
|
||||
const { nativeSha256 } = await appImport("server/src/services/native-runtime/canonical.ts") as typeof import("../server/src/services/native-runtime/canonical.js");
|
||||
const { drizzle } = await import(pathToFileURL(serverRequire.resolve("drizzle-orm/postgres-js")).href);
|
||||
// GNU ps reports CPU time in whole seconds. Read Linux process clock ticks
|
||||
// directly; resolve their frequency once, outside every measured interval.
|
||||
const databaseCpuClockTicks = process.platform === "linux"
|
||||
? Number(execFileSync("getconf", ["CLK_TCK"], { encoding: "utf8" }).trim()) : null;
|
||||
if (databaseCpuClockTicks !== null) assert.ok(Number.isFinite(databaseCpuClockTicks) && databaseCpuClockTicks > 0);
|
||||
const databaseCpuResolutionMillis = databaseCpuClockTicks !== null ? 1000 / databaseCpuClockTicks
|
||||
: process.platform === "darwin" ? 10 : null;
|
||||
const output = arg("output");
|
||||
assert.ok(output, "An --output=<new-report.json> path is required to retain evidence");
|
||||
const traceFile = path.resolve(arg("trace-output") ?? `${output}.blocks.jsonl`);
|
||||
assert.ok(path.resolve(output) !== traceFile, "Summary and append-only block trace must use separate paths");
|
||||
function assertOutsideApp(file: string) {
|
||||
const relative = path.relative(root, path.resolve(file));
|
||||
assert.ok(relative === ".." || relative.startsWith(`..${path.sep}`) || path.isAbsolute(relative), "Keep measurement evidence outside frozen APP_ROOT");
|
||||
}
|
||||
assertOutsideApp(traceFile); assertOutsideApp(output);
|
||||
await fs.writeFile(output, "", { flag: "wx" });
|
||||
// Exclusive creation prevents an accidental rerun from overwriting prior evidence.
|
||||
const trace = await fs.open(traceFile, "ax");
|
||||
let traceWriteMillis = 0;
|
||||
async function appendTrace(entry: Record<string, unknown>) {
|
||||
const started = performance.now();
|
||||
await trace.appendFile(`${JSON.stringify({ at: new Date().toISOString(), monotonicMs: started, ...entry })}\n`);
|
||||
traceWriteMillis += performance.now() - started;
|
||||
}
|
||||
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_EVENT_INGESTION_BENCHMARK_DATABASE_URL");
|
||||
if (externalUrl) assert.ok(["127.0.0.1", "localhost"].includes(new URL(externalUrl).hostname), "Use a disposable local database");
|
||||
|
||||
type Append = (db: Db, input: AppendHeartbeatRunEventInput) => Promise<AppendHeartbeatRunEventResult>;
|
||||
type AppendBatch = (db: Db, inputs: AppendHeartbeatRunEventInput[]) => Promise<AppendHeartbeatRunEventResult[]>;
|
||||
type EventModule = { appendHeartbeatRunEvent: Append; appendHeartbeatRunEvents?: AppendBatch };
|
||||
type Fixture = { companyId: string; agentId: string; runId: string; source: string; history: number; sourceOffset: number; next: number };
|
||||
type Query = { sql: string; params: unknown[]; elapsedMs?: number; returnedRows?: number };
|
||||
type Sample = {
|
||||
elapsedMs: number; eventsPerSecond: number; newEvents: number; duplicateEvents: number;
|
||||
appCpuMicros: number; databaseCpuMillis: number | null; heapDeltaBytes: number; rssBytes: number;
|
||||
eventLoopUtilization: number; eventLoopDelayMaxMs: number | null;
|
||||
retainedHistoryBefore: number; retainedHistoryAfter: number;
|
||||
attemptedEvents: number; nextBefore: number; nextAfter: number;
|
||||
appendStartedMonotonicMs: number; appendEndedMonotonicMs: number;
|
||||
firstCommitObservedMonotonicMs: number; lastCommitObservedMonotonicMs: number;
|
||||
verificationMillis: number; verificationEndedMonotonicMs: number;
|
||||
};
|
||||
type TimedResult = { sample: Sample; values: AppendHeartbeatRunEventInput[]; receipts: AppendHeartbeatRunEventResult[] };
|
||||
type Arm = "baseline" | "candidate";
|
||||
type CommitObservations = { first?: number; last?: number };
|
||||
const median = (values: number[]) => [...values].sort((a, b) => a - b)[Math.floor(values.length / 2)]!;
|
||||
const summary = (samples: Sample[]) => ({
|
||||
medianMs: median(samples.map(sample => sample.elapsedMs)),
|
||||
minMs: Math.min(...samples.map(sample => sample.elapsedMs)), maxMs: Math.max(...samples.map(sample => sample.elapsedMs)),
|
||||
medianEventsPerSecond: median(samples.map(sample => sample.eventsPerSecond)),
|
||||
medianAppCpuMicrosPerEvent: median(samples.map(sample => sample.appCpuMicros / (sample.newEvents + sample.duplicateEvents))),
|
||||
samples,
|
||||
});
|
||||
const baselineDir = await fs.mkdtemp(path.join(tmpdir(), "native-event-interleaved-baseline-"));
|
||||
await fs.writeFile(path.join(baselineDir, "package.json"), JSON.stringify({ type: "module" }));
|
||||
await fs.mkdir(path.join(baselineDir, "server"));
|
||||
// Keep original ESM package conditions and the same dependency instances as the
|
||||
// candidate. Only this disposable private snapshot receives a dependency link.
|
||||
await fs.symlink(path.join(root, "server/node_modules"), path.join(baselineDir, "server/node_modules"), "dir");
|
||||
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
|
||||
let db: Db | undefined;
|
||||
const companiesToRemove: string[] = [];
|
||||
const setupStarted = performance.now();
|
||||
const loadBefore = loadavg();
|
||||
const report: Record<string, unknown> = {
|
||||
scope: "Real append service / real PostgreSQL; sum of per-arm active append intervals under 8x64 interleaving. Not assembled application burst completion, browser, or hosted acceptance.",
|
||||
appRoot: root, measurementFile, actualSelfSha256, gatesSha256, traceFile,
|
||||
originalHarnessProvenance: { path: "scripts/benchmark-native-event-ingestion.ts", provenanceOnly: true,
|
||||
sha256: "48e967a5e7a8c803ff3bf4890a37ff70564ba1a618d83bcff5bb110b35ee8133" },
|
||||
performanceAcceptanceEligible: !quick,
|
||||
qualification: quick ? "Ineligible 1k ordered A/A source qualification only; full accepted series requires ROOT RELEASE" : "Full source workload; application acceptance remains separate",
|
||||
phase, baselineRevision, candidateRevision, workload, workloadSha256,
|
||||
baselineSourceSha256: createHash("sha256").update(baselineSource).digest("hex"),
|
||||
measurement: { warmups, pairedOrder: "8x64 blocks interleaved; first arm alternates per pair", cache: "warm; seed and projection preparation excluded", databaseCpuResolutionMillis,
|
||||
databaseCpuSource: databaseCpuClockTicks === null ? "ps process CPU time" : "/proc/PID/stat utime+stime clock ticks",
|
||||
databaseCpuAggregateWorstQuantizationMillis: databaseCpuResolutionMillis === null ? null : blocksPerArm * databaseCpuResolutionMillis,
|
||||
databaseCpuCaution: "Each block is tick-quantized (typically 10ms on Linux); an eight-block sum has up to eight ticks of quantization uncertainty. Do not infer sub-tick CPU precision.",
|
||||
memoryCaution: "Heap deltas, RSS and process CPU describe shared-process intervals; heap deltas do not identify owned allocations.",
|
||||
activeInterval: "invoke() including append receipt checks, excluding generator, CPU probes, independent verification, trace writes and inter-arm gaps",
|
||||
pairInterval: "pair plan through final independent database verification; final pair-complete log and scenario diagnostics are outside this interval" },
|
||||
environment: { node: process.version, platform: process.platform, architecture: process.arch, cpus: availableParallelism(), loadBefore },
|
||||
results: [], pairTimeline: [], incomplete: true, cleanupComplete: false,
|
||||
};
|
||||
let calibratedNoise: any;
|
||||
|
||||
async function save() {
|
||||
report.traceWriteMillis = traceWriteMillis;
|
||||
if (output) await fs.writeFile(output, JSON.stringify(report, null, 2));
|
||||
}
|
||||
|
||||
function backendCpuMillis(pid: number): number | null {
|
||||
if (process.platform === "win32") return null;
|
||||
if (databaseCpuClockTicks !== null) {
|
||||
const stat = readFileSync(`/proc/${pid}/stat`, "utf8");
|
||||
// Fields after the parenthesized comm begin at field 3 (state).
|
||||
const fields = stat.slice(stat.lastIndexOf(")") + 2).trim().split(/\s+/);
|
||||
return (Number(fields[11]) + Number(fields[12])) / databaseCpuClockTicks * 1000;
|
||||
}
|
||||
const value = execFileSync("ps", ["-o", "time=", "-p", String(pid)], { encoding: "utf8" }).trim();
|
||||
const parts = value.split(":").map(Number);
|
||||
return parts.length >= 2 && parts.every(Number.isFinite)
|
||||
? parts.reduce((total, part) => total * 60 + part, 0) * 1000 : null;
|
||||
}
|
||||
|
||||
function input(fixture: Fixture, sequence: number): AppendHeartbeatRunEventInput {
|
||||
const envelope = {
|
||||
schema: "paperclip.prp.event.v1", schemaVersion: 1, runId: fixture.runId,
|
||||
normalizedSessionId: fixture.runId, sourceKind: "runner", sourceInstanceId: fixture.source,
|
||||
sourceEventId: `${fixture.source}:${sequence}`, sourceSeq: sequence,
|
||||
eventType: "item.delta", priority: 1, emittedAt: "2026-11-03T00:00:00.000Z",
|
||||
turnId: "benchmark-turn", itemId: "benchmark-item",
|
||||
payload: { kind: "agentMessage", channel: "final", text: "x".repeat(workload.payloadTextBytes) },
|
||||
};
|
||||
return {
|
||||
companyId: fixture.companyId, agentId: fixture.agentId, runId: fixture.runId,
|
||||
eventType: envelope.eventType, stream: "system", level: "info", payload: { prpEvent: envelope },
|
||||
nativeSource: {
|
||||
sourceInstanceId: fixture.source, sourceEventId: envelope.sourceEventId, sourceSeq: sequence,
|
||||
protocolSchemaVersion: 1, canonicalPayload: envelope,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
// Query diagnostics run separately from unprofiled timing. Wrapping the actual
|
||||
// prepared execution captures completed query duration, including transactions.
|
||||
function diagnosticDb(target: Db, queries: Query[], transactions: Array<{ elapsedMs: number; lockAcquisitionQueryMs?: number; lockHeldThroughCommitMs?: number }>): Db {
|
||||
const recorded = installDatabaseWorkSignals(drizzle(target.$client, { schema: target._.fullSchema }), target) as Db;
|
||||
type Session = { prepareQuery: (...args: any[]) => { execute: (...args: any[]) => Promise<unknown> } };
|
||||
const decorate = (connection: Db, transaction?: { acquisitionMs?: number; acquiredAt?: number }) => {
|
||||
const session = (connection as unknown as { session: Session }).session;
|
||||
const prepare = session.prepareQuery;
|
||||
session.prepareQuery = function (this: Session, ...args: any[]) {
|
||||
const prepared = prepare.apply(this, args), execute = prepared.execute;
|
||||
const query = args[0] as Query;
|
||||
prepared.execute = async function (this: unknown, ...values: any[]) {
|
||||
const started = performance.now();
|
||||
try {
|
||||
const result = await execute.apply(this, values);
|
||||
const elapsedMs = performance.now() - started;
|
||||
queries.push({ sql: query.sql, params: query.params, elapsedMs, returnedRows: Array.isArray(result) ? result.length : undefined });
|
||||
if (transaction && /\bfor update\b/i.test(query.sql)) {
|
||||
transaction.acquisitionMs = elapsedMs;
|
||||
transaction.acquiredAt = performance.now();
|
||||
}
|
||||
return result;
|
||||
} catch (error) {
|
||||
queries.push({ sql: query.sql, params: query.params, elapsedMs: performance.now() - started });
|
||||
throw error;
|
||||
}
|
||||
};
|
||||
return prepared;
|
||||
};
|
||||
const original = connection.transaction.bind(connection);
|
||||
connection.transaction = ((callback: (tx: Db) => Promise<unknown>, options?: Parameters<Db["transaction"]>[1]) => {
|
||||
const started = performance.now(), state: { acquisitionMs?: number; acquiredAt?: number } = {};
|
||||
return original(async tx => callback(decorate(tx as unknown as Db, state)), options).finally(() => {
|
||||
transactions.push({ elapsedMs: performance.now() - started, lockAcquisitionQueryMs: state.acquisitionMs,
|
||||
lockHeldThroughCommitMs: state.acquiredAt === undefined ? undefined : performance.now() - state.acquiredAt });
|
||||
});
|
||||
}) as unknown as typeof connection.transaction;
|
||||
return connection;
|
||||
};
|
||||
return decorate(recorded);
|
||||
}
|
||||
|
||||
try {
|
||||
await appendTrace({ kind: "run-start", measurementFile, actualSelfSha256, workloadSha256, baselineRevision, gatesSha256, workload });
|
||||
// Snapshot changed supporting service modules too: a B1→B2 comparison must
|
||||
// not accidentally import B2 behavior behind its nominally B1 entry point.
|
||||
// The database schema/client intentionally remain the compatible migrated
|
||||
// superset for both arms; the benchmark is about the event owner itself.
|
||||
const changed = git(["diff", "--name-only", baselineRevision, "--", "server/src/services"])
|
||||
.trim().split("\n").filter(file => file.endsWith(".ts") && !file.endsWith(".test.ts"));
|
||||
const snapshots = new Map<string, string>([[sourcePath, baselineSource]]);
|
||||
let baselineCursorSource: string | undefined;
|
||||
try { baselineCursorSource = git(["show", `${baselineRevision}:${cursorSourcePath}`]); }
|
||||
catch { /* The pre-cursor baseline has no projection to prepare. */ }
|
||||
if (baselineCursorSource) snapshots.set(cursorSourcePath, baselineCursorSource);
|
||||
for (const file of changed) {
|
||||
if (snapshots.has(file)) continue;
|
||||
try { snapshots.set(file, git(["show", `${baselineRevision}:${file}`])); }
|
||||
catch { /* A newly added service cannot have been imported by the baseline. */ }
|
||||
}
|
||||
for (const [file, source] of snapshots) {
|
||||
const destination = path.join(baselineDir, file);
|
||||
await fs.mkdir(path.dirname(destination), { recursive: true });
|
||||
const sourceUrl = pathToFileURL(path.join(root, file));
|
||||
await fs.writeFile(destination, source.replace(/(from\s+|import\s+)["']([^"']+)["']/g, (_match, prefix: string, specifier: string) => {
|
||||
if (specifier.startsWith("node:")) return `${prefix}${JSON.stringify(specifier)}`;
|
||||
if (!specifier.startsWith(".")) return `${prefix}${JSON.stringify(specifier)}`;
|
||||
const actual = fileURLToPath(new URL(specifier, sourceUrl));
|
||||
const relative = path.relative(root, actual).replace(/\.js$/, ".ts");
|
||||
const selected = snapshots.has(relative) ? path.join(baselineDir, relative) : actual;
|
||||
return `${prefix}${JSON.stringify(pathToFileURL(selected).href)}`;
|
||||
}));
|
||||
}
|
||||
report.baselineServiceSnapshots = Object.fromEntries([...snapshots].map(([file, source]) => [file, createHash("sha256").update(source).digest("hex")]));
|
||||
const baselineFile = path.join(baselineDir, sourcePath);
|
||||
const baseline = await import(pathToFileURL(baselineFile).href) as EventModule;
|
||||
const candidate = phase === "aa" ? baseline : await appImport(sourcePath) as EventModule;
|
||||
report.projectionPriming = { baseline: Boolean(baselineCursorSource), candidate: phase === "aa" ? Boolean(baselineCursorSource) : true };
|
||||
report.effectiveCandidateRevision = phase === "aa" ? baselineRevision : candidateRevision;
|
||||
if (phase === "batch") assert.equal(typeof candidate.appendHeartbeatRunEvents, "function", "Batch phase requires appendHeartbeatRunEvents");
|
||||
report.candidateSourceSha256 = createHash("sha256").update(await fs.readFile(path.join(root, sourcePath))).digest("hex");
|
||||
report.baselineSnapshotSupport = { appRoot: root, dependencyResolution: "Original bare imports use normal ESM resolution through a private server/node_modules symlink to APP_ROOT/server/node_modules; relative unchanged support stays in APP_ROOT; DB schema/client are the APP_ROOT compatible migrated superset", privateSnapshotDirectory: baselineDir };
|
||||
const proofFiles = [...new Set([sourcePath, gatesPath, "packages/db/src/index.ts", "packages/db/src/work-signals.ts",
|
||||
"server/src/services/native-runtime/canonical.ts", "server/src/__tests__/helpers/external-test-database.ts",
|
||||
...git(["ls-files", "--", "server/src/services", "packages/db/src/schema"]).trim().split("\n"),
|
||||
...git(["ls-files", "--others", "--exclude-standard", "--", "server/src/services", "packages/db/src/schema"]).trim().split("\n"),
|
||||
])].filter(file => /\.(?:ts|mjs)$/.test(file) && !file.endsWith(".test.ts"));
|
||||
report.candidateSourceHashes = Object.fromEntries(await Promise.all(proofFiles.map(async file => [file,
|
||||
createHash("sha256").update(await fs.readFile(path.join(root, file))).digest("hex")])));
|
||||
report.candidateDirtyDiff = git(["diff", "HEAD", "--", sourcePath, "server/src/services/heartbeat", "packages/db/src/schema"]);
|
||||
if (arg("noise-report")) {
|
||||
calibratedNoise = JSON.parse(await fs.readFile(arg("noise-report")!, "utf8"));
|
||||
assertProtocolIdentity(calibratedNoise);
|
||||
assert.equal(calibratedNoise.phase, "aa"); assert.equal(calibratedNoise.incomplete, false);
|
||||
assert.equal(calibratedNoise.baselineSourceSha256, report.baselineSourceSha256, "Pinned baseline source changed after calibration");
|
||||
assert.equal(evaluateServiceRepeatability(calibratedNoise).passed, true,
|
||||
"A/A repeatability must pass before candidate measurements start");
|
||||
}
|
||||
if (!externalUrl) database = await startEmbeddedPostgresTestDatabase("native-event-ingestion-benchmark-");
|
||||
if (externalUrl) await applyPendingMigrations(externalUrl);
|
||||
db = createDb(externalUrl ?? database!.connectionString, { maxConnections: 1 });
|
||||
const [settings] = await db.$client.unsafe("select version() postgres_version, pg_backend_pid() backend_pid, current_setting('shared_buffers') shared_buffers, current_setting('work_mem') work_mem, current_setting('jit') jit");
|
||||
report.database = settings;
|
||||
|
||||
async function fixture(history: number, scenario = "ordered"): Promise<Fixture> {
|
||||
const companyId = randomUUID(), agentId = randomUUID(), runId = randomUUID(), source = randomUUID();
|
||||
companiesToRemove.push(companyId);
|
||||
await db!.insert(companies).values({ id: companyId, name: "Synthetic native event benchmark", issuePrefix: `E${companyId.replaceAll("-", "")}` });
|
||||
await db!.insert(agents).values({ id: agentId, companyId, name: "Synthetic event source", adapterType: "paperclip_runner" });
|
||||
await db!.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", runtimeMode: "native", nextEventSeq: history + 1 });
|
||||
const sourceOffset = scenario === "late-gap" ? 1 : scenario === "warm-suffix" ? workload.warmSourceOffset : 0;
|
||||
await db!.$client.unsafe(`insert into heartbeat_run_events
|
||||
(company_id, run_id, agent_id, seq, event_type, stream, level, payload, source_instance_id, source_event_id, source_seq, source_payload_sha256, protocol_schema_version)
|
||||
select $1::uuid, $2::uuid, $3::uuid, n, 'item.delta', 'system', 'info', jsonb_build_object('historicalSyntheticText', repeat('x', $5::integer)),
|
||||
$4, $4 || ':' || (n + $6::integer)::text, n + $6::integer, repeat('0', 64), 1
|
||||
from generate_series(1, $7::integer) n`, [companyId, runId, agentId, source, workload.payloadTextBytes, sourceOffset, history]);
|
||||
await db!.$client.unsafe("VACUUM ANALYZE heartbeat_run_events");
|
||||
return { companyId, agentId, runId, source, history, sourceOffset, next: history + sourceOffset + 1 };
|
||||
}
|
||||
|
||||
async function prepare(f: Fixture, arm: Arm) {
|
||||
const usesBaseline = arm === "baseline" || phase === "aa";
|
||||
if (usesBaseline && !baselineCursorSource) return;
|
||||
// Seed setup is an old-version append. Prime using the real fallback and
|
||||
// bounded rebuild before steady-state timing; cold behavior is traced below.
|
||||
const module = await import(usesBaseline
|
||||
? pathToFileURL(path.join(baselineDir, cursorSourcePath)).href
|
||||
: pathToFileURL(path.join(root, cursorSourcePath)).href) as Record<string, unknown>;
|
||||
const rebuild = module.rebuildNativeEventCursorPage as undefined | ((db: Db, binding: { companyId: string; runId: string }) => Promise<{ pending: boolean }>);
|
||||
assert.equal(typeof rebuild, "function", "Cursor owner must expose rebuildNativeEventCursorPage for seeded-history preparation");
|
||||
await db!.$client.unsafe("update heartbeat_runs set event_cursor_rebuild_pending=true where id=$1::uuid", [f.runId]);
|
||||
for (let page = 0; page <= Math.ceil((f.history + 1) / 512) + 2; page++) {
|
||||
const result = await rebuild!(db!, { companyId: f.companyId, runId: f.runId });
|
||||
if (!result.pending) return;
|
||||
}
|
||||
throw new Error("Cursor fixture failed to finish bounded rebuild");
|
||||
}
|
||||
|
||||
function emittedInputs(f: Fixture, scenario: string, repetitions = 1) {
|
||||
if (scenario === "late-gap") return [input(f, 1)];
|
||||
const newCount = scenario === "replay" ? events / 2 : events;
|
||||
const values: AppendHeartbeatRunEventInput[] = [];
|
||||
for (let group = 0; group < repetitions; group++) {
|
||||
const unique = Array.from({ length: newCount }, (_, n) => input(f, f.next + n));
|
||||
f.next += newCount;
|
||||
values.push(...unique);
|
||||
if (scenario === "replay") values.push(...unique);
|
||||
}
|
||||
return values;
|
||||
}
|
||||
|
||||
async function invoke(target: Db, arm: Arm, values: AppendHeartbeatRunEventInput[], commits?: CommitObservations) {
|
||||
const owner = arm === "baseline" ? baseline : candidate;
|
||||
const receipts: AppendHeartbeatRunEventResult[] = [];
|
||||
const receive = (received: AppendHeartbeatRunEventResult[]) => {
|
||||
if (commits && received.some(receipt => receipt.disposition === "committed")) {
|
||||
const observed = performance.now(); commits.first ??= observed; commits.last = observed;
|
||||
}
|
||||
receipts.push(...received);
|
||||
};
|
||||
if (phase === "batch" && arm === "candidate") {
|
||||
for (let offset = 0; offset < values.length; offset += 64) {
|
||||
const chunk = values.slice(offset, offset + 64);
|
||||
assert.ok(chunk.reduce((bytes, value) => bytes + Buffer.byteLength(JSON.stringify(value.nativeSource!.canonicalPayload)), 0) <= 64 * 1024);
|
||||
receive(await owner.appendHeartbeatRunEvents!(target, chunk));
|
||||
}
|
||||
}
|
||||
else for (const value of values) receive([await owner.appendHeartbeatRunEvent(target, value)]);
|
||||
assert.equal(receipts.length, values.length);
|
||||
assert.ok(receipts.every(receipt => receipt.disposition === "committed" || receipt.disposition === "duplicate"));
|
||||
const expectedCursor = Math.max(...values.map(value => value.nativeSource!.sourceSeq));
|
||||
assert.ok(receipts.every(receipt => receipt.highestContiguousSourceSeq <= expectedCursor || values.length === 1));
|
||||
return receipts;
|
||||
}
|
||||
|
||||
async function verifyDurable(f: Fixture, scenario: string, values: AppendHeartbeatRunEventInput[], receipts: AppendHeartbeatRunEventResult[]) {
|
||||
assert.equal(receipts.at(-1)!.highestContiguousSourceSeq,
|
||||
scenario === "late-gap" ? f.history + 1 : scenario === "warm-suffix" ? 0 : f.next - 1);
|
||||
const expectedRows = scenario === "late-gap" ? f.history + 1 : f.next - f.sourceOffset - 1;
|
||||
const [durable] = await db!.$client.unsafe("select count(*)::integer as events, max(seq)::bigint as last from heartbeat_run_events where run_id=$1::uuid", [f.runId]);
|
||||
assert.equal(Number(durable!.events), expectedRows, "Independent history read must account for every committed event");
|
||||
assert.equal(Number(durable!.last), expectedRows, "Run sequences must not skip or overlap");
|
||||
const expectedBySequence = new Map(values.map(value => [value.nativeSource!.sourceSeq, value]));
|
||||
const sequences = [...expectedBySequence.keys()];
|
||||
const persisted = await db!.$client.unsafe(`select source_instance_id, source_event_id, source_seq,
|
||||
source_payload_sha256, protocol_schema_version, payload from heartbeat_run_events
|
||||
where run_id=$1::uuid and source_instance_id=$2 and source_seq between $3::bigint and $4::bigint
|
||||
order by source_seq`, [f.runId, f.source, Math.min(...sequences), Math.max(...sequences)]);
|
||||
assert.equal(persisted.length, expectedBySequence.size);
|
||||
for (const row of persisted) {
|
||||
const expected = expectedBySequence.get(Number(row.source_seq))!;
|
||||
assert.equal(row.source_event_id, expected.nativeSource!.sourceEventId);
|
||||
assert.equal(row.source_instance_id, expected.nativeSource!.sourceInstanceId);
|
||||
assert.equal(row.protocol_schema_version, expected.nativeSource!.protocolSchemaVersion);
|
||||
assert.equal(row.source_payload_sha256, nativeSha256(expected.nativeSource!.canonicalPayload));
|
||||
assert.deepEqual(row.payload, expected.payload, "Durable event payload must match independently of its append receipt");
|
||||
}
|
||||
}
|
||||
|
||||
async function timed(f: Fixture, arm: Arm, scenario: string, repetitions = groups, context: Record<string, unknown> = {}): Promise<TimedResult> {
|
||||
const nextBefore = f.next, retainedHistoryBefore = f.next - f.sourceOffset - 1;
|
||||
const values = emittedInputs(f, scenario, repetitions);
|
||||
assert.equal(values.length, scenario === "late-gap" ? 1 : repetitions * events);
|
||||
const expectedNew = scenario === "replay" ? values.length / 2 : values.length;
|
||||
assert.equal(f.next, scenario === "late-gap" ? nextBefore : nextBefore + expectedNew, "Generator next must progress monotonically without gaps");
|
||||
const pgBefore = backendCpuMillis(Number(settings!.backend_pid));
|
||||
const beforeMemory = process.memoryUsage(), cpuBefore = process.cpuUsage(), eluBefore = performance.eventLoopUtilization();
|
||||
const loop = monitorEventLoopDelay({ resolution: 10 });
|
||||
loop.enable();
|
||||
const started = performance.now();
|
||||
let receipts: AppendHeartbeatRunEventResult[], ended: number;
|
||||
const commits: CommitObservations = {};
|
||||
try { receipts = await invoke(db!, arm, values, commits); ended = performance.now(); }
|
||||
catch (error) {
|
||||
const failedEnded = performance.now(); loop.disable();
|
||||
await appendTrace({ kind: "append-failed", ...context, arm, scenario, history: f.history,
|
||||
appendStartedMonotonicMs: started, appendEndedMonotonicMs: failedEnded, elapsedMs: failedEnded - started,
|
||||
attemptedEvents: values.length, nextBefore, nextAfter: f.next,
|
||||
retainedHistoryBefore, firstCommitObservedMonotonicMs: commits.first, lastCommitObservedMonotonicMs: commits.last });
|
||||
throw error;
|
||||
} finally { loop.disable(); }
|
||||
const elapsedMs = ended - started;
|
||||
const cpu = process.cpuUsage(cpuBefore), memory = process.memoryUsage();
|
||||
const eventLoopUtilization = performance.eventLoopUtilization(eluBefore).utilization;
|
||||
const pgAfter = backendCpuMillis(Number(settings!.backend_pid));
|
||||
const newEvents = receipts.filter(receipt => receipt.disposition === "committed").length;
|
||||
assert.equal(newEvents, expectedNew);
|
||||
assert.ok(commits.first !== undefined && commits.last !== undefined, "Every scenario commits at least one event");
|
||||
const verificationStarted = performance.now();
|
||||
await verifyDurable(f, scenario, values, receipts);
|
||||
const verificationEnded = performance.now();
|
||||
const sample: Sample = { elapsedMs, eventsPerSecond: values.length / elapsedMs * 1000, newEvents,
|
||||
duplicateEvents: values.length - newEvents, appCpuMicros: cpu.user + cpu.system,
|
||||
databaseCpuMillis: pgBefore === null || pgAfter === null ? null : pgAfter - pgBefore,
|
||||
heapDeltaBytes: memory.heapUsed - beforeMemory.heapUsed, rssBytes: memory.rss,
|
||||
eventLoopUtilization, eventLoopDelayMaxMs: loop.count ? loop.max / 1e6 : null,
|
||||
retainedHistoryBefore, retainedHistoryAfter: retainedHistoryBefore + newEvents,
|
||||
attemptedEvents: values.length, nextBefore, nextAfter: f.next,
|
||||
appendStartedMonotonicMs: started, appendEndedMonotonicMs: ended,
|
||||
firstCommitObservedMonotonicMs: commits.first, lastCommitObservedMonotonicMs: commits.last,
|
||||
verificationMillis: verificationEnded - verificationStarted, verificationEndedMonotonicMs: verificationEnded };
|
||||
return { sample, values, receipts };
|
||||
}
|
||||
|
||||
function aggregate(blocks: Sample[], scenario: string): Sample {
|
||||
const sum = (key: "elapsedMs" | "newEvents" | "duplicateEvents" | "appCpuMicros" | "heapDeltaBytes" | "attemptedEvents" | "verificationMillis") => blocks.reduce((total, block) => total + block[key], 0);
|
||||
assert.equal(blocks.length, scenario === "late-gap" ? 1 : blocksPerArm);
|
||||
const attemptedEvents = sum("attemptedEvents"), newEvents = sum("newEvents");
|
||||
assert.equal(attemptedEvents, scenario === "late-gap" ? 1 : 512, "Every attempted event must remain in the sample");
|
||||
assert.equal(newEvents, scenario === "replay" ? 256 : attemptedEvents);
|
||||
assert.equal(sum("duplicateEvents"), scenario === "replay" ? 256 : 0);
|
||||
for (let index = 1; index < blocks.length; index++) {
|
||||
assert.equal(blocks[index]!.retainedHistoryBefore, blocks[index - 1]!.retainedHistoryAfter, "Retained history must be continuous between blocks");
|
||||
assert.equal(blocks[index]!.nextBefore, blocks[index - 1]!.nextAfter, "Generator next must be continuous between blocks");
|
||||
}
|
||||
const first = blocks[0]!, last = blocks.at(-1)!, elapsedMs = sum("elapsedMs");
|
||||
const delays = blocks.flatMap(block => block.eventLoopDelayMaxMs === null ? [] : [block.eventLoopDelayMaxMs]);
|
||||
return { elapsedMs, eventsPerSecond: attemptedEvents / elapsedMs * 1000, attemptedEvents, newEvents,
|
||||
duplicateEvents: sum("duplicateEvents"), appCpuMicros: sum("appCpuMicros"),
|
||||
databaseCpuMillis: blocks.some(block => block.databaseCpuMillis === null) ? null : blocks.reduce((total, block) => total + block.databaseCpuMillis!, 0),
|
||||
heapDeltaBytes: sum("heapDeltaBytes"), rssBytes: Math.max(...blocks.map(block => block.rssBytes)),
|
||||
eventLoopUtilization: blocks.reduce((total, block) => total + block.eventLoopUtilization * block.elapsedMs, 0) / elapsedMs,
|
||||
eventLoopDelayMaxMs: delays.length ? Math.max(...delays) : null,
|
||||
retainedHistoryBefore: first.retainedHistoryBefore, retainedHistoryAfter: last.retainedHistoryAfter,
|
||||
nextBefore: first.nextBefore, nextAfter: last.nextAfter, appendStartedMonotonicMs: first.appendStartedMonotonicMs,
|
||||
appendEndedMonotonicMs: last.appendEndedMonotonicMs,
|
||||
firstCommitObservedMonotonicMs: first.firstCommitObservedMonotonicMs, lastCommitObservedMonotonicMs: last.lastCommitObservedMonotonicMs,
|
||||
verificationMillis: sum("verificationMillis"),
|
||||
verificationEndedMonotonicMs: last.verificationEndedMonotonicMs };
|
||||
}
|
||||
|
||||
// Qualify before entering the full benchmark. All comparisons preserve the
|
||||
// original append receipts; a failed fixture never becomes a timing sample.
|
||||
for (const history of histories) for (const scenario of scenarios) {
|
||||
const samples: Record<Arm, Sample[]> = { baseline: [], candidate: [] };
|
||||
let a: Fixture | undefined, b: Fixture | undefined;
|
||||
if (scenario !== "late-gap") {
|
||||
a = await fixture(history, scenario); b = await fixture(history, scenario);
|
||||
await prepare(a, "baseline"); await prepare(b, "candidate");
|
||||
for (let warmup = 0; warmup < warmups; warmup++) {
|
||||
for (const arm of ["baseline", "candidate"] as const) {
|
||||
await appendTrace({ kind: "warmup-start", history, scenario, warmup, arm, attempts: 512 });
|
||||
const { sample } = await timed(arm === "baseline" ? a : b, arm, scenario);
|
||||
await appendTrace({ kind: "warmup-complete", history, scenario, warmup, arm, ...sample });
|
||||
}
|
||||
}
|
||||
}
|
||||
for (let pair = 0; pair < pairs; pair++) {
|
||||
if (scenario === "late-gap") {
|
||||
a = await fixture(history, scenario); b = await fixture(history, scenario);
|
||||
await prepare(a, "baseline"); await prepare(b, "candidate");
|
||||
}
|
||||
const firstArm: Arm = pair % 2 ? "candidate" : "baseline";
|
||||
const armOrder: Arm[] = firstArm === "baseline" ? ["baseline", "candidate"] : ["candidate", "baseline"];
|
||||
const count = scenario === "late-gap" ? 1 : blocksPerArm;
|
||||
const blocks: Record<Arm, Sample[]> = { baseline: [], candidate: [] };
|
||||
const allValues: Record<Arm, AppendHeartbeatRunEventInput[]> = { baseline: [], candidate: [] };
|
||||
const allReceipts: Record<Arm, AppendHeartbeatRunEventResult[]> = { baseline: [], candidate: [] };
|
||||
const pairStarted = performance.now(), traceMillisBefore = traceWriteMillis;
|
||||
// The complete plan persists before the first append. Any missing start or
|
||||
// completion remains visible as an unstarted/interrupted block; no retry,
|
||||
// delay, outlier discard or growing-trace rewrite occurs.
|
||||
const plan = Array.from({ length: count }, (_, block) => armOrder.map(arm => ({ block, arm,
|
||||
attempts: scenario === "late-gap" ? 1 : blockAttempts }))).flat();
|
||||
await appendTrace({ kind: "pair-plan", history, scenario, pair, firstArm, pairStartedMonotonicMs: pairStarted, plan });
|
||||
for (let block = 0; block < count; block++) {
|
||||
assert.equal(a!.next - a!.sourceOffset, b!.next - b!.sourceOffset, "Corresponding blocks must start at the same retained history");
|
||||
for (const arm of armOrder) {
|
||||
const f = arm === "baseline" ? a! : b!;
|
||||
const context = { history, scenario, pair, block, arm, firstArm,
|
||||
retainedHistoryBefore: f.next - f.sourceOffset - 1, nextBefore: f.next };
|
||||
await appendTrace({ kind: "block-start", ...context });
|
||||
try {
|
||||
const measured = await timed(f, arm, scenario, scenario === "late-gap" ? 1 : groupsPerBlock, context);
|
||||
blocks[arm].push(measured.sample);
|
||||
allValues[arm].push(...measured.values); allReceipts[arm].push(...measured.receipts);
|
||||
await appendTrace({ kind: "block-complete", ...context, ...measured.sample });
|
||||
} catch (error) {
|
||||
await appendTrace({ kind: "block-failed", ...context, failedMonotonicMs: performance.now(),
|
||||
error: error instanceof Error ? { name: error.name, message: error.message, stack: error.stack } : String(error) });
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
}
|
||||
const finalVerification: Record<string, { startedMonotonicMs: number; endedMonotonicMs: number; elapsedMs: number; attemptedEvents: number; boundary: string }> = {};
|
||||
for (const arm of armOrder) {
|
||||
await appendTrace({ kind: "final-verification-start", history, scenario, pair, arm });
|
||||
const started = performance.now();
|
||||
await verifyDurable(arm === "baseline" ? a! : b!, scenario, allValues[arm], allReceipts[arm]);
|
||||
const ended = performance.now();
|
||||
finalVerification[arm] = { startedMonotonicMs: started, endedMonotonicMs: ended, elapsedMs: ended - started,
|
||||
attemptedEvents: allValues[arm].length, boundary: "All append promises have committed; independent durable count, order, identity and payload verification; no asynchronous source queue or browser drain" };
|
||||
}
|
||||
const pairEnded = performance.now();
|
||||
const activeAppendMillis = [...blocks.baseline, ...blocks.candidate].reduce((total, block) => total + block.elapsedMs, 0);
|
||||
const blockVerificationMillis = [...blocks.baseline, ...blocks.candidate].reduce((total, block) => total + block.verificationMillis, 0);
|
||||
const finalVerificationMillis = Object.values(finalVerification).reduce((total, value) => total + value.elapsedMs, 0);
|
||||
const perArm = Object.fromEntries((["baseline", "candidate"] as const).map(arm => {
|
||||
const sample = aggregate(blocks[arm], scenario); samples[arm].push(sample);
|
||||
return [arm, { activeAppendMillis: sample.elapsedMs, firstAppendStartMonotonicMs: sample.appendStartedMonotonicMs,
|
||||
firstCommitObservedMonotonicMs: sample.firstCommitObservedMonotonicMs, lastCommitObservedMonotonicMs: sample.lastCommitObservedMonotonicMs,
|
||||
firstAppendToLastCommitWallMillis: sample.lastCommitObservedMonotonicMs - sample.appendStartedMonotonicMs,
|
||||
firstToLastCommitWallMillis: sample.lastCommitObservedMonotonicMs - sample.firstCommitObservedMonotonicMs,
|
||||
commitObservationBoundary: "Host receipt resolution after commit; a batch observes its committed receipts together, not individual database commit timestamps",
|
||||
blocks: blocks[arm] }];
|
||||
}));
|
||||
const timeline = { history, scenario, pair, firstArm, wholeElapsedMs: pairEnded - pairStarted,
|
||||
startedMonotonicMs: pairStarted, endedMonotonicMs: pairEnded, perArm, finalIndependentDrainVerification: finalVerification,
|
||||
activeAppendMillis, blockVerificationMillis, finalVerificationMillis,
|
||||
bookkeepingAndGapMillis: pairEnded - pairStarted - activeAppendMillis - blockVerificationMillis - finalVerificationMillis,
|
||||
traceWriteMillisWithinPair: traceWriteMillis - traceMillisBefore };
|
||||
(report.pairTimeline as unknown[]).push(timeline);
|
||||
await appendTrace({ kind: "pair-complete", ...timeline });
|
||||
// Save only compact scenario summaries after diagnostics, never the growing
|
||||
// block trace on every block. JSONL above is the authoritative failure log.
|
||||
}
|
||||
const diagnostics: Record<string, unknown> = {};
|
||||
for (const arm of ["baseline", "candidate"] as const) {
|
||||
const f = await fixture(history, scenario); await prepare(f, arm);
|
||||
const queries: Query[] = [], transactions: Array<{ elapsedMs: number }> = [];
|
||||
const recorded = diagnosticDb(db, queries, transactions);
|
||||
const values = emittedInputs(f, scenario), receipts = await invoke(recorded, arm, values);
|
||||
const plans: unknown[] = [];
|
||||
for (const query of queries) {
|
||||
if (plans.length >= 16 || plans.some((plan: any) => plan.sql === query.sql)) continue;
|
||||
const read = /^select\b/i.test(query.sql);
|
||||
// Never reexecute a mutation merely to profile it. Read plans report
|
||||
// actual rows/buffers; mutation plans describe shape without effects.
|
||||
const [plan] = await db.$client.unsafe(`EXPLAIN (${read ? "ANALYZE, BUFFERS, " : ""}FORMAT JSON) ${query.sql}`, query.params as never[]);
|
||||
plans.push({ sql: query.sql, analyzed: read, plan: plan!["QUERY PLAN"] });
|
||||
}
|
||||
const scanNodes = (node: any): any[] => [
|
||||
...(node["Relation Name"] === "heartbeat_run_events" ? [node] : []),
|
||||
...(node.Plans ?? []).flatMap(scanNodes),
|
||||
];
|
||||
const historyScans = (plans as Array<{ analyzed: boolean; plan: Array<{ Plan: unknown }> }>).filter(plan => plan.analyzed)
|
||||
.flatMap(plan => scanNodes(plan.plan[0].Plan));
|
||||
const usesProjection = arm === "baseline" || phase === "aa" ? Boolean(baselineCursorSource) : true;
|
||||
const examinedRows = (node: any) => (node["Actual Rows"] + (node["Rows Removed by Filter"] ?? 0)
|
||||
+ (node["Rows Removed by Index Recheck"] ?? 0)) * node["Actual Loops"];
|
||||
if (usesProjection) {
|
||||
assert.ok(historyScans.every(node => node["Node Type"] !== "Seq Scan" && examinedRows(node) <= 64),
|
||||
"A ready cursor must not scan retained event history, including the optimized baseline arm");
|
||||
assert.ok(!queries.some(query => /count\(\*\)/i.test(query.sql) && /heartbeat_run_events/.test(query.sql)),
|
||||
"A ready cursor unexpectedly used the retained-history aggregate fallback");
|
||||
}
|
||||
diagnostics[arm] = { sqlCalls: queries.length, transactions: transactions.length, sqlCallsPerEvent: queries.length / values.length,
|
||||
transactionsPerEvent: transactions.length / values.length, transactionTimings: transactions,
|
||||
lockTimingBoundary: "Acquisition query duration includes network/row-read cost; held interval ends after commit. It is not a measurement of pure lock wait.",
|
||||
persistedEvents: receipts.filter(receipt => receipt.disposition === "committed").length,
|
||||
usesProjection, structuralCacheBoundPassed: usesProjection ? true : null,
|
||||
historyScanRows: historyScans.map(examinedRows), queries, plans };
|
||||
}
|
||||
const baselineStats = summary(samples.baseline), candidateStats = summary(samples.candidate);
|
||||
const relativePairedDifference = samples.baseline.map((sample, n) => (samples.candidate[n]!.elapsedMs - sample.elapsedMs) / sample.elapsedMs);
|
||||
(report.results as unknown[]).push({ history, scenario,
|
||||
latencyGate: scenario === "late-gap" ? "structural-only" : history >= 10_000 ? "improvement" : "nonregression",
|
||||
baseline: baselineStats, candidate: candidateStats, diagnostics,
|
||||
pairedCandidateWins: relativePairedDifference.filter(value => value < 0).length,
|
||||
medianLatencyChangePercent: 100 * (candidateStats.medianMs / baselineStats.medianMs - 1),
|
||||
aaNoiseEnvelopeRelative: phase === "aa" ? Math.max(...relativePairedDifference.map(Math.abs)) : undefined,
|
||||
});
|
||||
await save();
|
||||
}
|
||||
report.setupAndExecutionMillis = performance.now() - setupStarted;
|
||||
report.loadAfter = loadavg();
|
||||
report.incomplete = false;
|
||||
report.acceptance = phase === "aa" ? "Baseline repeatability only; this does not establish an improvement" : "Compare against the frozen same-workload A/A report; throughput improvement needs at least four of five paired wins beyond baseline noise. Browser, control and full-app gates remain separate.";
|
||||
if (arg("noise-report")) {
|
||||
const noise = calibratedNoise;
|
||||
report.noiseReport = arg("noise-report");
|
||||
report.comparison = (report.results as any[]).map(result => {
|
||||
const control = noise.results.find((row: any) => row.history === result.history && row.scenario === result.scenario);
|
||||
assert.ok(control);
|
||||
const improvement = (result.baseline.medianMs - result.candidate.medianMs) / result.baseline.medianMs;
|
||||
const paired = noiseExceedingPairedWins(result.baseline.samples.map((sample: Sample, index: number) =>
|
||||
(sample.elapsedMs - result.candidate.samples[index].elapsedMs) / sample.elapsedMs), control.aaNoiseEnvelopeRelative);
|
||||
const improvedBeyondNoise = paired.pairedWinRequirementPassed && improvement > control.aaNoiseEnvelopeRelative;
|
||||
const noRegressionBeyondNoise = -improvement <= control.aaNoiseEnvelopeRelative;
|
||||
return { history: result.history, scenario: result.scenario, pairedCandidateWins: result.pairedCandidateWins,
|
||||
noiseExceedingPairedWins: paired.noiseExceedingPairedWins, requiredPairedWins: paired.requiredPairedWins,
|
||||
latencyGate: result.latencyGate,
|
||||
improvementRelative: improvement, noiseEnvelopeRelative: control.aaNoiseEnvelopeRelative,
|
||||
improvedBeyondNoise, noRegressionBeyondNoise,
|
||||
acceptancePassed: result.latencyGate === "structural-only" ? null
|
||||
: result.latencyGate === "improvement" ? improvedBeyondNoise : noRegressionBeyondNoise };
|
||||
});
|
||||
}
|
||||
await save();
|
||||
} catch (error) {
|
||||
report.incomplete = true;
|
||||
report.failure = error instanceof Error ? { name: error.name, message: error.message, stack: error.stack } : String(error);
|
||||
await appendTrace({ kind: "run-failed", failure: report.failure });
|
||||
throw error;
|
||||
} finally {
|
||||
try {
|
||||
// The owned cluster is removed as a whole below. Per-company agent deletes
|
||||
// would repeatedly scan other fixtures through foreign-key checks. An
|
||||
// explicitly supplied test database retains unrelated data, so remove all
|
||||
// our event rows first, then their parents in separate phases.
|
||||
if (externalUrl) {
|
||||
for (const table of ["heartbeat_run_events", "heartbeat_runs", "agents", "companies"]) {
|
||||
for (const companyId of companiesToRemove) {
|
||||
await db?.$client.unsafe(`delete from ${table} where ${table === "companies" ? "id" : "company_id"}=$1::uuid`, [companyId]);
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
try { await db?.$client.end({ timeout: 1 }); }
|
||||
finally {
|
||||
try { await database?.cleanup(); }
|
||||
finally { await fs.rm(baselineDir, { recursive: true, force: true }); }
|
||||
}
|
||||
}
|
||||
report.cleanupComplete = true;
|
||||
await appendTrace({ kind: "cleanup-complete", incomplete: report.incomplete });
|
||||
await save();
|
||||
await trace.close();
|
||||
}
|
||||
if (phase === "aa") {
|
||||
report.repeatability = evaluateServiceRepeatability(report);
|
||||
await save();
|
||||
}
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
if (process.argv.includes("--require-acceptance")) assertServiceAcceptance(report);
|
||||
if (process.argv.includes("--require-repeatability")) {
|
||||
assert.equal(evaluateServiceRepeatability(report).passed, true, "A/A repeatability gate failed; retain evidence and diagnose before candidate measurements");
|
||||
}
|
||||
@@ -17,13 +17,70 @@ const manifest = JSON.parse(await readFile(manifestPath, "utf8")) as Array<{ bas
|
||||
assert.ok(manifest.length >= 5, "At least five actual paired application runs are required");
|
||||
const load = async (file: string) => JSON.parse(await readFile(path.resolve(path.dirname(manifestPath), file), "utf8"));
|
||||
const pairs = await Promise.all(manifest.map(async entry => ({ baseline: await load(entry.baselineFile), candidate: await load(entry.candidateFile) })));
|
||||
const names = ["paced", "burst16", "burst128", "burst512", "existing-history", "operator-stop"];
|
||||
const workloads = [
|
||||
{ name: "paced", runs: 1, events: 128, intervalMs: 25, prelude: 0, stop: false },
|
||||
{ name: "burst16", runs: 1, events: 16, intervalMs: 0, prelude: 0, stop: false },
|
||||
{ name: "burst128", runs: 10, events: 128, intervalMs: 0, prelude: 0, stop: false },
|
||||
{ name: "burst512", runs: 100, events: 512, intervalMs: 0, prelude: 0, stop: false },
|
||||
{ name: "existing-history", runs: 10, events: 128, intervalMs: 0, prelude: 1000, stop: false },
|
||||
{ name: "operator-stop", runs: 1, events: 512, intervalMs: 100, prelude: 0, stop: true },
|
||||
];
|
||||
const names = workloads.map(item => item.name);
|
||||
const median = (values: number[]) => [...values].sort((a, b) => a - b)[Math.floor(values.length / 2)]!;
|
||||
const first = pairs[0];
|
||||
const sha256 = (value: unknown) => typeof value === "string" && /^[a-f0-9]{64}$/.test(value);
|
||||
const uuid = (value: unknown) => typeof value === "string" && /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i.test(value);
|
||||
for (const pair of pairs) for (const arm of ["baseline", "candidate"] as const) {
|
||||
const report = pair[arm];
|
||||
assert.equal(report.protocolVersion, "v2-compact-markers-original-clock-stream", "Fresh browser acceptance requires the qualified v2 clock protocol");
|
||||
assert.equal(report.admissionProtocol, "sequential-provider-and-clock-readiness-before-common-epoch", "Use the qualified pre-measurement admission protocol");
|
||||
assert.equal(report.protocolVersion, "v4-remote-streaming-original-local-stop-clock-stream", "Fresh browser acceptance requires the qualified v4 mixed clock protocol");
|
||||
assert.equal(report.transport, "sandbox-remote-with-local-stop", "V4 requires remote streaming and the original local Stop transport");
|
||||
assert.deepEqual(report.transportPolicy, { streaming: "sandbox-remote", operatorStop: "local",
|
||||
realCloudStop: "Separate required qualification; fake-plugin void receipts grant no remote Stop authority" }, "Missing explicit transport and real cloud Stop qualification policy");
|
||||
assert.deepEqual(report.resourceBudgets, { providerReadinessMs: 300_000, fullInvocationMs: 900_000,
|
||||
streamingMs: 120_000, terminalDrainageMs: 180_000, stopAcknowledgementMs: 30_000, remoteLeaseCleanupMs: 30_000 },
|
||||
"Keep the prospectively registered remote setup/resource budgets and existing streaming/control timeouts");
|
||||
const remote = report.remote;
|
||||
assert.equal(remote?.transport, "sandbox-remote", "Missing normal remote runtime provenance");
|
||||
assert.equal(remote?.plugin?.pluginKey, "paperclip.fake-sandbox-provider", "Missing normal remote plugin provenance");
|
||||
assert.equal(remote?.selection, "Normal plugin install + sandbox environment + agent defaultEnvironmentId; no runtime resolver override",
|
||||
"Use normal API remote environment selection");
|
||||
assert.ok(uuid(remote.environmentId) && uuid(report.companyId), "Missing actual remote company/environment identity");
|
||||
assert.ok(remote.agentIds?.length > 0 && remote.agentIds.every(uuid), "Missing actual remote agent selection");
|
||||
assert.deepEqual(Object.keys(remote.plugin.assetHashes ?? {}).sort(), ["lease-cleanup.js", "manifest.js", "plugin.js", "worker.js"]);
|
||||
for (const [file, hash] of Object.entries(remote.plugin.assetHashes)) {
|
||||
assert.ok(sha256(hash), "Missing actual provider plugin build fingerprint");
|
||||
assert.equal(hash, first.baseline.remote.plugin.assetHashes[file], "Provider cleanup build changed across matched samples");
|
||||
}
|
||||
const local = report.local;
|
||||
assert.equal(local?.transport, "local", "Missing normal local Stop runtime provenance");
|
||||
assert.equal(local?.selection, "Normal local environment + separate control agent defaultEnvironmentId; existing test-only Codex leaf command fixture",
|
||||
"Use normal local Stop environment selection");
|
||||
assert.ok(uuid(local.environmentId) && uuid(local.controlAgentId) && local.environmentId !== remote.environmentId
|
||||
&& !remote.agentIds.includes(local.controlAgentId), "Local Stop needs a separate normal control agent and environment");
|
||||
for (const field of ["commandSha256", "loaderSha256"]) {
|
||||
assert.ok(sha256(local[field]), "Missing original local provider command provenance");
|
||||
assert.equal(local[field], first.baseline.local[field], "Local Stop fixture changed across matched samples");
|
||||
}
|
||||
assert.equal(remote.revision, report.sourceRevision);
|
||||
assert.equal(remote.buildRevision, report.compiledBuildRevision);
|
||||
assert.equal(remote.tls?.peerVerification, true, "Remote TLS peer verification is required");
|
||||
assert.equal(typeof remote.runnerPublicUrl, "string", "Missing actual remote WSS endpoint");
|
||||
const endpoint = new URL(remote.runnerPublicUrl);
|
||||
assert.equal(endpoint.protocol, "wss:", "Actual remote endpoint must use verified WSS");
|
||||
assert.equal(endpoint.hostname, "127.0.0.1", "Use the owned loopback remote fixture endpoint");
|
||||
assert.ok(Number(endpoint.port) > 0 && !endpoint.username && !endpoint.password && !endpoint.search && !endpoint.hash,
|
||||
"Use the recorded owned remote fixture endpoint without overrides");
|
||||
for (const value of [remote.tls?.caSha256, remote.tls?.serverCertificateSha256, remote.runner?.sha256,
|
||||
remote.provider?.nodeSha256, remote.provider?.wrapperSha256, remote.provider?.fixtureSha256]) {
|
||||
assert.ok(sha256(value), "Missing remote TLS, daemon or provider artifact fingerprint");
|
||||
}
|
||||
assert.equal(remote.provider.fixtureSha256, report.fixtureSha256, "Staged provider fixture changed");
|
||||
assert.ok(remote.runner.metadata?.prpTransportModes?.includes("dial_wss"), "Remote daemon must support verified WSS");
|
||||
for (const [group, field] of [["runner", "sha256"], ["provider", "nodeSha256"], ["provider", "wrapperSha256"], ["provider", "fixtureSha256"]]) {
|
||||
assert.equal(remote[group][field], first.baseline.remote[group][field], "Remote provider/daemon artifact changed across matched samples");
|
||||
}
|
||||
assert.equal(report.admissionProtocol, "two-at-a-time-provider-and-clock-readiness-before-common-epoch", "Use the qualified pre-measurement admission protocol");
|
||||
assert.equal(report.setupActivationConcurrency, 2, "Keep the qualified setup activation concurrency of two");
|
||||
assert.equal(report.setupReadinessPollingMs, 50, "Keep the qualified setup readiness observation interval");
|
||||
assert.equal(report.complete, true); assert.equal(report.qualificationOnly, false, "Smoke cannot qualify performance");
|
||||
assert.equal(report.cleanup?.complete, true, "Acceptance requires completed fixture cleanup");
|
||||
@@ -45,6 +102,11 @@ for (const pair of pairs) for (const arm of ["baseline", "candidate"] as const)
|
||||
for (const name of names) {
|
||||
const result = report.cases.find((item: any) => item.name === name);
|
||||
assert.ok(result, `Missing fixed workload ${name}`);
|
||||
const fixed = workloads.find(item => item.name === name)!;
|
||||
for (const field of ["runs", "events", "intervalMs", "prelude", "stop"] as const) assert.equal(result[field], fixed[field], `Fixed workload changed: ${name}.${field}`);
|
||||
assert.equal(result.transport, result.stop ? "local" : "sandbox-remote", "Wrong per-case transport; fake-plugin does not qualify remote Stop");
|
||||
assert.equal(result.environmentId, result.stop ? local.environmentId : remote.environmentId, "Wrong per-case normal environment");
|
||||
assert.equal(result.setupActivationConcurrency, 2, "Keep each case's qualified setup activation concurrency of two");
|
||||
assert.equal(result.persistedAfterReload, true);
|
||||
assert.equal(result.matchedVisibleClocks, result.visible?.length, "Every observed marker needs its original provider clock");
|
||||
assert.ok(result.matchedVisibleClocks >= (result.stop ? 3 : result.events), "Missing foreground render/clock observations");
|
||||
@@ -58,6 +120,177 @@ for (const pair of pairs) for (const arm of ["baseline", "candidate"] as const)
|
||||
assert.ok(result.preludeProof?.verifiedAtMs < result.epochMs, "Verify retained prelude before releasing measured emission");
|
||||
}
|
||||
assert.equal(result.durable.length, result.runs, "Every background run needs independent delta persistence proof");
|
||||
assert.equal(result.runIds?.length, result.runs, "Every remote participant needs its actual run identity");
|
||||
assert.ok(result.runIds.every(uuid), "Missing actual remote run UUID");
|
||||
assert.equal(new Set(result.runIds).size, result.runs, "Duplicate remote run identity");
|
||||
assert.deepEqual(result.durable.map((proof: any) => proof.runId).sort(), [...result.runIds].sort(), "Durable proof run set mismatch");
|
||||
const authority = result.controllerAuthority;
|
||||
assert.equal(authority?.checkedBeforeMeasuredEpoch, true, "Missing controller authority proof before the measured epoch");
|
||||
assert.equal(authority?.kind, result.stop ? "host-file-journal" : "database-controller-checkpoint", "Wrong checkpoint authority for actual transport");
|
||||
assert.equal(authority?.observedReadOnly, "on", "Remote authority proof must observe a read-only transaction");
|
||||
assert.equal(authority?.observedIsolation, "repeatable read", "Remote authority proof must observe repeatable-read isolation");
|
||||
assert.ok(Number.isSafeInteger(authority?.checkedAtMs) && authority.checkedAtMs > 0
|
||||
&& Number.isSafeInteger(authority.connectionClosedAtMs) && authority.connectionClosedAtMs > 0
|
||||
&& authority.checkedAtMs <= authority.connectionClosedAtMs && authority.connectionClosedAtMs < result.epochMs,
|
||||
"Close the actual remote authority snapshot before the measured epoch");
|
||||
assert.equal(authority.rows?.length, result.runs, "Every remote run needs its controller checkpoint and lease proof");
|
||||
assert.deepEqual(authority.rows.map((row: any) => row.run_id).sort(), [...result.runIds].sort(), "Remote authority run set mismatch");
|
||||
assert.equal(new Set(authority.rows.map((row: any) => row.issue_id)).size, result.runs, "Duplicate remote issue authority");
|
||||
const authorityByRun = new Map<string, any>();
|
||||
for (const row of authority.rows) {
|
||||
assert.ok(uuid(row.issue_id) && uuid(row.lease_id), "Missing actual task/lease UUID");
|
||||
assert.equal(row.company_id, report.companyId, "Remote authority company mismatch");
|
||||
assert.equal(row.environment_id, result.environmentId, "Remote authority environment mismatch");
|
||||
assert.equal(row.runtime_mode, "native");
|
||||
if (!result.stop) {
|
||||
assert.ok(remote.agentIds.includes(row.agent_id), "Remote run did not use its normal remote agent");
|
||||
assert.equal(row.provider, "fake-plugin");
|
||||
assert.match(row.provider_lease_id, /^fake-plugin:\/\//);
|
||||
assert.equal(row.checkpoint_part, "header", "Missing actual database controller checkpoint header");
|
||||
assert.ok(Number.isSafeInteger(row.checkpoint_header_bytes) && row.checkpoint_header_bytes > 0);
|
||||
} else {
|
||||
assert.equal(row.agent_id, local.controlAgentId, "Stop did not use its separate local control agent");
|
||||
assert.equal(row.provider, "local", "Stop requires its normal local lease");
|
||||
assert.equal(row.provider_lease_id, null); assert.equal(row.checkpoint_part, null, "Local Stop must retain host-file authority without a database header");
|
||||
assert.ok(BigInt(String(row.process_identity_seq)) > 0n);
|
||||
assert.equal(row.process_identity_source_event_id, null, "Local identity must be server-authored");
|
||||
assert.equal(row.process_identity?.targetKind, "local");
|
||||
assert.equal(row.process_identity?.processPid, row.process_pid);
|
||||
assert.equal(row.process_identity?.processGroupId, row.process_group_id);
|
||||
assert.equal(row.process_identity?.processStartedAt, row.process_started_at);
|
||||
assert.ok(uuid(row.admitted_execution_workspace_id), "Missing actual admitted local workspace binding");
|
||||
}
|
||||
authorityByRun.set(row.run_id, row);
|
||||
}
|
||||
assert.equal(result.providerClocks?.length, result.runs, "Every actual provider needs its wrapper launcher and PID");
|
||||
assert.deepEqual(result.providerClocks.map((clock: any) => clock.runId).sort(), [...result.runIds].sort());
|
||||
assert.equal(new Set(result.providerClocks.map((clock: any) => clock.providerPid)).size, result.runs);
|
||||
for (const clock of result.providerClocks) {
|
||||
assert.equal(clock.launcher, result.stop ? "host-local" : "staged-remote", "Actual wrapper launcher does not match transport");
|
||||
assert.ok(Number.isSafeInteger(clock.providerPid) && clock.providerPid > 1, "Missing actual provider PID");
|
||||
const stream = result.clockEvidence?.find((stream: any) => stream.nonce === clock.nonce);
|
||||
assert.equal(stream?.launcher, clock.launcher, "Clock hello launcher proof missing");
|
||||
assert.equal(stream?.providerPid, clock.providerPid, "Clock hello provider PID proof missing");
|
||||
assert.equal(result.durable.find((proof: any) => proof.runId === clock.runId)?.token, clock.nonce);
|
||||
}
|
||||
assert.equal(result.participants, result.runs, "Every provider GET must remain open before the common epoch");
|
||||
assert.ok(Number.isSafeInteger(result.setupReadinessStartedAtMs) && result.setupReadinessStartedAtMs > 0);
|
||||
assert.equal(result.readinessDeadlineMs, result.setupReadinessStartedAtMs + report.resourceBudgets.providerReadinessMs,
|
||||
"Keep one unchanged shared readiness deadline");
|
||||
assert.ok(Number.isSafeInteger(result.allReadyObservedAtMs) && result.allReadyObservedAtMs >= result.setupReadinessStartedAtMs
|
||||
&& result.allReadyObservedAtMs <= result.readinessDeadlineMs && result.allReadyObservedAtMs < result.epochMs,
|
||||
"All actual providers and clocks must be ready before the epoch and shared deadline");
|
||||
assert.ok(authority.checkedAtMs >= result.allReadyObservedAtMs, "Check actual controller authority after every provider is ready");
|
||||
assert.equal(result.activations?.length, result.runs, "Every actual participant needs its setup activation timing");
|
||||
assert.deepEqual(result.activations.map((entry: any) => entry.index).sort((a: number, b: number) => a - b),
|
||||
Array.from({ length: result.runs }, (_, index) => index), "Setup activation indices must cover every participant once");
|
||||
const waves = result.activationWaves;
|
||||
assert.equal(waves?.length, Math.ceil((result.runs - 1) / 2), "Background setup must use bounded waves of two");
|
||||
for (let waveIndex = 0; waveIndex < waves.length; waveIndex++) {
|
||||
const wave = waves[waveIndex], firstIndex = 1 + waveIndex * 2;
|
||||
const indices = Array.from({ length: Math.min(2, result.runs - firstIndex) }, (_, offset) => firstIndex + offset);
|
||||
assert.equal(wave.index, waveIndex); assert.deepEqual(wave.indices, indices, "Wrong setup wave participants");
|
||||
assert.deepEqual(wave.statuses, indices.map(() => "fulfilled"), "A failed or unfinished setup wave cannot qualify");
|
||||
assert.ok(Number.isSafeInteger(wave.startedAtMs) && Number.isSafeInteger(wave.settledAtMs)
|
||||
&& wave.startedAtMs >= (waveIndex ? waves[waveIndex - 1].settledAtMs : result.setupReadinessStartedAtMs)
|
||||
&& wave.settledAtMs >= wave.startedAtMs && wave.settledAtMs <= result.allReadyObservedAtMs,
|
||||
"Finish the whole setup wave before starting the next one");
|
||||
}
|
||||
for (const activation of result.activations) {
|
||||
const row = authority.rows.find((row: any) => row.issue_id === activation.issueId);
|
||||
assert.ok(row, "Setup activation task must match actual controller authority");
|
||||
const clock = result.providerClocks.find((clock: any) => clock.runId === row.run_id);
|
||||
assert.equal(activation.token, clock.nonce, "Setup activation must match its actual provider clock");
|
||||
assert.equal(activation.status, "fulfilled", "An unfinished setup activation cannot qualify");
|
||||
for (const field of ["startedAtMs", "requestSettledAtMs", "providerReadyAtMs", "clockHelloAtMs", "readinessObservedAtMs", "settledAtMs"]) {
|
||||
assert.ok(Number.isSafeInteger(activation[field]) && activation[field] > 0 && activation[field] < result.epochMs,
|
||||
"Missing actual pre-epoch activation/readiness timing");
|
||||
}
|
||||
assert.ok(activation.requestSettledAtMs >= activation.startedAtMs && activation.providerReadyAtMs >= activation.startedAtMs
|
||||
&& activation.clockHelloAtMs >= activation.startedAtMs
|
||||
&& activation.readinessObservedAtMs >= Math.max(activation.requestSettledAtMs, activation.providerReadyAtMs, activation.clockHelloAtMs)
|
||||
&& activation.settledAtMs >= activation.readinessObservedAtMs && activation.settledAtMs <= result.allReadyObservedAtMs,
|
||||
"Actual provider GET and hello must precede observed readiness");
|
||||
const stream = result.clockEvidence.find((stream: any) => stream.nonce === activation.token);
|
||||
assert.equal(stream.hello, true, "Every actual provider needs its clock hello");
|
||||
for (const field of ["providerReadyAtMs", "clockHelloAtMs", "readinessObservedAtMs"]) assert.equal(stream[field], activation[field], "Readiness metadata must match actual clock observations");
|
||||
if (activation.index === 0) assert.equal(activation.waveIndex, null);
|
||||
else {
|
||||
const waveIndex = Math.floor((activation.index - 1) / 2), wave = waves[waveIndex];
|
||||
assert.equal(activation.waveIndex, waveIndex);
|
||||
assert.ok(activation.startedAtMs >= wave.startedAtMs && activation.settledAtMs <= wave.settledAtMs,
|
||||
"Keep each activation inside its settled setup wave");
|
||||
}
|
||||
}
|
||||
assert.equal(result.settledAtMs, result.epochMs + result.drainThroughTerminalMs, "Terminal timing boundary changed");
|
||||
if (result.stop) {
|
||||
assert.equal(result.stopAcknowledgedAtMs, result.stopRequestedAtMs + result.stopThroughAcknowledgementMs, "Stop acknowledgement timing boundary changed");
|
||||
const row = authority.rows[0], journal = authority.localJournal, before = result.localProcesses;
|
||||
assert.equal(journal?.schema, "paperclip.runner.durable.control-plane-state.v1", "Missing owned local journal authority");
|
||||
assert.ok(sha256(journal.sha256) && journal.bytes > 0 && Number.isSafeInteger(journal.checkedAtMs)
|
||||
&& journal.checkedAtMs < result.epochMs, "Verify actual local file journal before the measured epoch");
|
||||
assert.notEqual(journal.checkpointEventPolicy, "control-v1", "Preserve the original local raw transport recovery policy");
|
||||
for (const [field, expected] of [["runId", row.run_id], ["runnerInstanceId", row.runner_instance_id],
|
||||
["normalizedSessionId", row.native_session_id], ["environmentLeaseId", row.admitted_execution_workspace_id]]) assert.equal(journal.identity?.[field], expected, "Local journal identity mismatch");
|
||||
assert.ok(before && before.hostname === row.process_identity.hostname && before.runnerPid === row.process_pid
|
||||
&& before.processGroupId === row.process_group_id && before.providerPid === result.providerClocks[0].providerPid,
|
||||
"Local observed process ownership mismatch");
|
||||
assert.ok(before.runnerAlive && before.groupAlive && before.providerAlive && Number.isSafeInteger(before.checkedAtMs)
|
||||
&& before.checkedAtMs < result.epochMs, "Independently observe all owned local processes before emission");
|
||||
const stopped = result.localStopProof, database = stopped?.database, after = database?.rows?.[0];
|
||||
assert.ok(stopped && Number.isSafeInteger(stopped.checkedAtMs) && stopped.checkedAtMs >= result.stopAcknowledgedAtMs
|
||||
&& stopped.checkedAtMs >= result.settledAtMs, "Verify local physical Stop after timing boundaries");
|
||||
assert.equal(database?.kind, "host-file-journal"); assert.equal(database?.observedReadOnly, "on");
|
||||
assert.equal(database?.observedIsolation, "repeatable read"); assert.equal(database?.rows?.length, 1);
|
||||
assert.equal(database?.checkedBeforeMeasuredEpoch, false);
|
||||
assert.ok(Number.isSafeInteger(database.checkedAtMs) && database.checkedAtMs >= result.stopAcknowledgedAtMs
|
||||
&& Number.isSafeInteger(database.connectionClosedAtMs) && database.connectionClosedAtMs >= database.checkedAtMs
|
||||
&& database.connectionClosedAtMs <= stopped.checkedAtMs, "Local stopped database proof predates acknowledgement");
|
||||
for (const field of ["run_id", "company_id", "environment_id", "lease_id", "agent_id", "provider", "provider_lease_id",
|
||||
"admitted_execution_workspace_id", "process_identity_seq", "process_identity_source_event_id"]) assert.equal(after?.[field], row[field], "Local stopped database identity mismatch");
|
||||
assert.equal(after.status, "cancelled"); assert.equal(after.cancellation_dispatch_state, "acknowledged");
|
||||
if (!after.process_pid && !after.process_group_id) {
|
||||
assert.ok(BigInt(String(after.process_stopped_seq)) > BigInt(String(row.process_identity_seq)), "Cleared local process identity needs its server stop receipt");
|
||||
assert.equal(after.process_stopped_source_event_id, null);
|
||||
assert.equal(after.process_stopped?.processPid, before.runnerPid);
|
||||
assert.equal(after.process_stopped?.processGroupId, before.processGroupId);
|
||||
assert.equal(after.process_stopped?.processStartedAt, row.process_identity.processStartedAt);
|
||||
} else {
|
||||
assert.equal(after.process_pid, before.runnerPid); assert.equal(after.process_group_id, before.processGroupId);
|
||||
}
|
||||
for (const field of ["runnerPid", "processGroupId", "providerPid", "hostname"]) assert.equal(stopped.processes?.[field], before[field]);
|
||||
assert.equal(stopped.processes.runnerAbsent, true); assert.equal(stopped.processes.groupAbsent, true);
|
||||
assert.equal(stopped.processes.providerAbsent, true, "Local provider remains alive after Stop");
|
||||
const lease = stopped.lease;
|
||||
assert.equal(lease?.id, row.lease_id); assert.equal(lease?.companyId, report.companyId);
|
||||
assert.equal(lease?.environmentId, local.environmentId); assert.equal(lease?.heartbeatRunId, row.run_id);
|
||||
assert.equal(lease?.provider, "local"); assert.equal(lease?.providerLeaseId, null);
|
||||
assert.ok(["released", "expired", "failed"].includes(lease.status) && lease.releasedAt
|
||||
&& lease.cleanupStatus !== "failed", "Local Stop lease cleanup remains unsettled");
|
||||
assert.equal(result.remoteLeases, undefined, "Local Stop cannot claim fake remote cleanup");
|
||||
} else {
|
||||
assert.equal(result.localStopProof, undefined, "Streaming cannot claim a local Stop proof");
|
||||
assert.equal(result.remoteLeases?.length, result.runs, "Every remote run needs post-timing lease cleanup proof");
|
||||
assert.deepEqual(result.remoteLeases.map((lease: any) => lease.runId).sort(), [...result.runIds].sort(), "Remote cleanup run set mismatch");
|
||||
assert.ok(Number.isSafeInteger(result.remoteLeasesCheckedAtMs) && result.remoteLeasesCheckedAtMs >= result.settledAtMs,
|
||||
"Verify remote leases after recorded timing boundaries");
|
||||
for (const lease of result.remoteLeases) {
|
||||
const row = authorityByRun.get(lease.runId);
|
||||
assert.equal(lease.id, row.lease_id); assert.equal(lease.companyId, report.companyId);
|
||||
assert.equal(lease.environmentId, remote.environmentId); assert.equal(lease.provider, row.provider);
|
||||
assert.equal(lease.providerLeaseId, row.provider_lease_id);
|
||||
assert.ok(["released", "expired", "failed"].includes(lease.status) && lease.releasedAt
|
||||
&& lease.cleanupStatus === "success", "Remote lease cleanup remains unsettled");
|
||||
if (lease.terminationReceipt) {
|
||||
const receipt = lease.terminationReceipt;
|
||||
assert.equal(receipt.schema, "paperclip.remote-termination.v1");
|
||||
assert.equal(receipt.companyId, report.companyId); assert.equal(receipt.runId, lease.runId);
|
||||
assert.equal(receipt.leaseId, lease.id); assert.equal(receipt.provider, lease.provider);
|
||||
assert.equal(receipt.providerLeaseId, lease.providerLeaseId);
|
||||
assert.ok(["stopped", "destroyed"].includes(receipt.state), "Invalid recorded remote termination receipt");
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const proof of result.durable) {
|
||||
assert.equal(proof.ordered, true); assert.equal(proof.uniqueSourceIdentities, proof.individualDeltas);
|
||||
assert.equal(proof.matchedSourceClocks, proof.individualDeltas, "Every durable measured delta needs an original provider clock");
|
||||
@@ -90,7 +323,11 @@ const measurements = names.flatMap(name => {
|
||||
});
|
||||
});
|
||||
const result: Record<string, any> = {
|
||||
scope: "Compiled-app deterministic-provider/browser acceptance; live-provider qualification remains separate",
|
||||
scope: "Compiled-app remote streaming and original local Stop performance; additional real cloud Stop qualification is required separately",
|
||||
transportPolicy: first.baseline.transportPolicy, additionalRealCloudStopQualificationRequired: true,
|
||||
admissionProtocol: first.baseline.admissionProtocol, setupActivationConcurrency: first.baseline.setupActivationConcurrency,
|
||||
protocolVersion: first.baseline.protocolVersion, transport: first.baseline.transport,
|
||||
resourceBudgets: first.baseline.resourceBudgets,
|
||||
phase, expectation, pairCount: pairs.length, baselineRevision: first.baseline.sourceRevision,
|
||||
candidateRevision: first.candidate.sourceRevision, measurementSha256: first.baseline.measurementSha256,
|
||||
gatesSha256: createHash("sha256").update(await readFile(new URL("./native-ingestion-performance-gates.mjs", import.meta.url))).digest("hex"),
|
||||
@@ -102,6 +339,12 @@ if (phase === "compare") {
|
||||
assert.ok(arg("noise-report"), "Comparison requires its frozen A/A --noise-report");
|
||||
const noise = JSON.parse(await readFile(arg("noise-report")!, "utf8"));
|
||||
assert.equal(noise.phase, "aa"); assert.equal(noise.complete, true);
|
||||
assert.equal(noise.protocolVersion, result.protocolVersion, "Noise report uses a different mixed protocol");
|
||||
assert.equal(noise.transport, result.transport, "Noise report uses a different transport");
|
||||
assert.deepEqual(noise.transportPolicy, result.transportPolicy, "Transport policy changed after A/A calibration");
|
||||
assert.equal(noise.admissionProtocol, result.admissionProtocol, "Admission protocol changed after A/A calibration");
|
||||
assert.equal(noise.setupActivationConcurrency, result.setupActivationConcurrency, "Setup activation concurrency changed after A/A calibration");
|
||||
assert.deepEqual(noise.resourceBudgets, result.resourceBudgets, "Resource budgets changed after A/A calibration");
|
||||
assert.equal(noise.baselineRevision, result.baselineRevision); assert.equal(noise.measurementSha256, result.measurementSha256);
|
||||
assert.equal(noise.gatesSha256, result.gatesSha256, "Acceptance gates changed after A/A calibration");
|
||||
assert.equal(noise.comparatorSha256, result.comparatorSha256, "Comparator changed after A/A calibration");
|
||||
|
||||
@@ -75,19 +75,90 @@ const workloads = [
|
||||
{ name: "operator-stop", runs: 1, events: 512, intervalMs: 100, prelude: 0, stop: true },
|
||||
];
|
||||
function browserReport(revision, latency) {
|
||||
return { protocolVersion: "v2-compact-markers-original-clock-stream", admissionProtocol: "sequential-provider-and-clock-readiness-before-common-epoch", setupReadinessPollingMs: 50, complete: true, cleanup: { complete: true, errors: [] }, qualificationOnly: false, diagnosticOnly: false, performanceAcceptanceEligible: true, browserRecording: { trace: "off", video: "off", screencast: false },
|
||||
const companyId = "00000000-0000-4000-8000-000000000001", environmentId = "00000000-0000-4000-8000-000000000002";
|
||||
const localEnvironmentId = "00000000-0000-4000-8000-000000000003", controlAgentId = "00000000-0000-4000-8000-000000000004";
|
||||
const remoteAgentId = "00000000-0000-4000-8000-000000000005";
|
||||
const id = (kind, caseIndex, index) => `00000000-0000-4000-8000-${(kind * 100000 + caseIndex * 1000 + index).toString(16).padStart(12, "0")}`;
|
||||
const transportPolicy = { streaming: "sandbox-remote", operatorStop: "local",
|
||||
realCloudStop: "Separate required qualification; fake-plugin void receipts grant no remote Stop authority" };
|
||||
return { protocolVersion: "v4-remote-streaming-original-local-stop-clock-stream", transport: "sandbox-remote-with-local-stop", transportPolicy, companyId,
|
||||
resourceBudgets: { providerReadinessMs: 300000, fullInvocationMs: 900000, streamingMs: 120000,
|
||||
terminalDrainageMs: 180000, stopAcknowledgementMs: 30000, remoteLeaseCleanupMs: 30000 },
|
||||
local: { transport: "local", environmentId: localEnvironmentId, controlAgentId,
|
||||
selection: "Normal local environment + separate control agent defaultEnvironmentId; existing test-only Codex leaf command fixture",
|
||||
commandSha256: "6".repeat(64), loaderSha256: "7".repeat(64) },
|
||||
remote: { transport: "sandbox-remote", revision, buildRevision: revision, environmentId, agentIds: [remoteAgentId],
|
||||
runnerPublicUrl: "wss://127.0.0.1:32100",
|
||||
selection: "Normal plugin install + sandbox environment + agent defaultEnvironmentId; no runtime resolver override",
|
||||
plugin: { pluginKey: "paperclip.fake-sandbox-provider", assetHashes: Object.fromEntries(["worker.js", "plugin.js", "manifest.js", "lease-cleanup.js"].map(file => [file, "5".repeat(64)])) },
|
||||
tls: { peerVerification: true, caSha256: "e".repeat(64), serverCertificateSha256: "f".repeat(64) },
|
||||
runner: { sha256: "d".repeat(64), metadata: { prpTransportModes: ["dial_wss"] } },
|
||||
provider: { nodeSha256: "b".repeat(64), wrapperSha256: "c".repeat(64), fixtureSha256: "a".repeat(64) } },
|
||||
fixtureSha256: "a".repeat(64), admissionProtocol: "two-at-a-time-provider-and-clock-readiness-before-common-epoch", setupActivationConcurrency: 2, setupReadinessPollingMs: 50, complete: true, cleanup: { complete: true, errors: [] }, qualificationOnly: false, diagnosticOnly: false, performanceAcceptanceEligible: true, browserRecording: { trace: "off", video: "off", screencast: false },
|
||||
sourceRevision: revision, compiledBuildRevision: revision, serverArtifact: "compiled", serverEntrySha256: `${revision}-compiled`,
|
||||
measurementSha256: "synthetic-fixed-fixture", compiledHotPathHashes: { append: "append", port: "port", runtime: "runtime" },
|
||||
cases: workloads.map(workload => ({ ...workload, stop: Boolean(workload.stop), persistedAfterReload: true,
|
||||
epochMs: 1000, preludeProof: workload.prelude ? { individualDeltas: workload.prelude, uniqueSourceIdentities: workload.prelude, verifiedAtMs: 999 } : undefined,
|
||||
visible: Array(workload.stop ? 3 : workload.events).fill({}), matchedVisibleClocks: workload.stop ? 3 : workload.events,
|
||||
firstVisibleAfterEmissionMs: latency, emissionToVisible: { medianMs: latency, p95Ms: latency },
|
||||
interChunkGaps: { paintCount: workload.events, count: workload.events - 1, medianMs: latency, maxMs: latency },
|
||||
intendedToVisible: { p95Ms: latency }, drainThroughTerminalMs: latency,
|
||||
stopThroughTerminalMs: latency, stopThroughAcknowledgementMs: latency,
|
||||
durable: Array.from({ length: workload.runs }, (_, index) => ({ token: String(index), ordered: true,
|
||||
uniqueSourceIdentities: workload.events, individualDeltas: workload.events, matchedSourceClocks: workload.events })),
|
||||
})) };
|
||||
cases: workloads.map((workload, caseIndex) => {
|
||||
const stop = Boolean(workload.stop), actualEnvironmentId = stop ? localEnvironmentId : environmentId;
|
||||
const launcher = stop ? "host-local" : "staged-remote", pid = 20000 + caseIndex * 1000;
|
||||
const snapshot = { checkedBeforeMeasuredEpoch: true, checkedAtMs: 998, connectionClosedAtMs: 999,
|
||||
observedReadOnly: "on", observedIsolation: "repeatable read", kind: stop ? "host-file-journal" : "database-controller-checkpoint" };
|
||||
const rows = Array.from({ length: workload.runs }, (_, index) => ({
|
||||
run_id: id(1, caseIndex, index), issue_id: id(2, caseIndex, index), company_id: companyId,
|
||||
agent_id: stop ? controlAgentId : remoteAgentId,
|
||||
lease_id: id(3, caseIndex, index), environment_id: actualEnvironmentId, runtime_mode: "native", provider: stop ? "local" : "fake-plugin",
|
||||
provider_lease_id: stop ? null : `fake-plugin://${id(1, caseIndex, index)}/fixture`,
|
||||
checkpoint_part: stop ? null : "header", checkpoint_header_bytes: stop ? null : 256,
|
||||
runner_instance_id: id(6, caseIndex, index), native_session_id: id(7, caseIndex, index), admitted_execution_workspace_id: id(8, caseIndex, index),
|
||||
...(stop ? { process_pid: pid - 1, process_group_id: pid - 1, process_started_at: "2026-11-03T00:00:00.000Z",
|
||||
process_identity_seq: "9", process_identity_source_event_id: null,
|
||||
process_identity: { targetKind: "local", hostname: "fixture-host", processPid: pid - 1, processGroupId: pid - 1,
|
||||
processStartedAt: "2026-11-03T00:00:00.000Z" } } : {}),
|
||||
}));
|
||||
const lease = { id: rows[0].lease_id, heartbeatRunId: rows[0].run_id, companyId, environmentId: localEnvironmentId,
|
||||
provider: "local", providerLeaseId: null, status: "released", releasedAt: "2026-11-03T00:00:01.000Z", cleanupStatus: null };
|
||||
const localProcesses = { runnerPid: pid - 1, processGroupId: pid - 1, providerPid: pid, hostname: "fixture-host",
|
||||
checkedAtMs: 999, runnerAlive: true, groupAlive: true, providerAlive: true };
|
||||
const activations = rows.map((row, index) => {
|
||||
const waveIndex = index ? Math.floor((index - 1) / 2) : null, startedAtMs = index ? 110 + waveIndex * 10 : 50;
|
||||
return { index, issueId: row.issue_id, token: String(index), waveIndex, startedAtMs,
|
||||
requestSettledAtMs: startedAtMs + (index ? 3 : 5), providerReadyAtMs: startedAtMs + (index ? 1 : 10),
|
||||
clockHelloAtMs: startedAtMs + (index ? 2 : 20), readinessObservedAtMs: index ? startedAtMs + 6 : 700,
|
||||
settledAtMs: index ? startedAtMs + 7 : 700, status: "fulfilled" };
|
||||
});
|
||||
const activationWaves = Array.from({ length: Math.ceil((workload.runs - 1) / 2) }, (_, index) => ({
|
||||
index, indices: Array.from({ length: Math.min(2, workload.runs - 1 - index * 2) }, (_, offset) => 1 + index * 2 + offset),
|
||||
startedAtMs: 110 + index * 10, settledAtMs: 118 + index * 10,
|
||||
statuses: Array(Math.min(2, workload.runs - 1 - index * 2)).fill("fulfilled"),
|
||||
}));
|
||||
return { ...workload, stop, transport: stop ? "local" : "sandbox-remote", environmentId: actualEnvironmentId, persistedAfterReload: true,
|
||||
setupActivationConcurrency: 2, setupReadinessStartedAtMs: 100, readinessDeadlineMs: 300100, allReadyObservedAtMs: 700,
|
||||
participants: workload.runs, activations, activationWaves,
|
||||
epochMs: 1000, preludeProof: workload.prelude ? { individualDeltas: workload.prelude, uniqueSourceIdentities: workload.prelude, verifiedAtMs: 999 } : undefined,
|
||||
settledAtMs: 1000 + latency, stopRequestedAtMs: stop ? 1000 : undefined,
|
||||
stopAcknowledgedAtMs: stop ? 1000 + latency : undefined, remoteLeasesCheckedAtMs: stop ? undefined : 1001 + latency,
|
||||
runIds: rows.map(row => row.run_id), controllerAuthority: { ...snapshot, rows,
|
||||
...(stop ? { localJournal: { schema: "paperclip.runner.durable.control-plane-state.v1", sha256: "8".repeat(64), bytes: 512, checkedAtMs: 999,
|
||||
identity: { runId: rows[0].run_id, runnerInstanceId: rows[0].runner_instance_id, normalizedSessionId: rows[0].native_session_id, environmentLeaseId: rows[0].admitted_execution_workspace_id } } } : {}) },
|
||||
providerClocks: rows.map((row, index) => ({ runId: row.run_id, nonce: String(index), launcher, providerPid: pid + index })),
|
||||
clockEvidence: rows.map((row, index) => ({ nonce: String(index), launcher, providerPid: pid + index,
|
||||
hello: true, providerReadyAtMs: activations[index].providerReadyAtMs, clockHelloAtMs: activations[index].clockHelloAtMs,
|
||||
readinessObservedAtMs: activations[index].readinessObservedAtMs })),
|
||||
...(stop ? { localProcesses, localStopProof: { checkedAtMs: 1001 + latency, lease,
|
||||
processes: { ...localProcesses, runnerAbsent: true, groupAbsent: true, providerAbsent: true },
|
||||
database: { ...snapshot, checkedBeforeMeasuredEpoch: false, checkedAtMs: 1000 + latency, connectionClosedAtMs: 1000 + latency,
|
||||
rows: [{ ...rows[0], process_pid: null, process_group_id: null, process_started_at: null, status: "cancelled", cancellation_dispatch_state: "acknowledged",
|
||||
process_stopped_seq: "64", process_stopped_source_event_id: null, process_stopped: { processPid: pid - 1, processGroupId: pid - 1, processStartedAt: rows[0].process_started_at } }] } } }
|
||||
: { remoteLeases: rows.map(row => ({ id: row.lease_id, runId: row.run_id, companyId, environmentId, provider: "fake-plugin", providerLeaseId: row.provider_lease_id,
|
||||
status: "released", cleanupStatus: "success", releasedAt: "2026-11-03T00:00:00.000Z" })) }),
|
||||
visible: Array(stop ? 3 : workload.events).fill({}), matchedVisibleClocks: stop ? 3 : workload.events,
|
||||
firstVisibleAfterEmissionMs: latency, emissionToVisible: { medianMs: latency, p95Ms: latency },
|
||||
interChunkGaps: { paintCount: workload.events, count: workload.events - 1, medianMs: latency, maxMs: latency },
|
||||
intendedToVisible: { p95Ms: latency }, drainThroughTerminalMs: latency,
|
||||
stopThroughTerminalMs: latency, stopThroughAcknowledgementMs: latency,
|
||||
durable: rows.map((row, index) => ({ runId: row.run_id, token: String(index), ordered: true,
|
||||
uniqueSourceIdentities: stop ? 3 : workload.events, individualDeltas: stop ? 3 : workload.events, matchedSourceClocks: stop ? 3 : workload.events })),
|
||||
};
|
||||
}) };
|
||||
}
|
||||
|
||||
test("browser CLI rejects the three-pairs-above-noise counterexample and ineligible evidence", async () => {
|
||||
@@ -117,14 +188,102 @@ test("browser CLI rejects the three-pairs-above-noise counterexample and ineligi
|
||||
assert.ok(result.gates.filter(gate => gate.improvementRequired).every(gate => gate.noiseExceedingPairedWins === 3));
|
||||
const good = run([`--pairs=${await manifest("four-wins", [0.30, 0.25, 0.20, 0.15, 0.01])}`, `--noise-report=${noise}`, "--require-acceptance"]);
|
||||
assert.equal(good.status, 0, good.stderr);
|
||||
const wide = run([`--pairs=${await manifest("int64-ordered-receipt", Array(5).fill(0.30), false, report => {
|
||||
const stop = report.cases[5];
|
||||
stop.controllerAuthority.rows[0].process_identity_seq = "9223372036854775806";
|
||||
stop.localStopProof.database.rows[0].process_identity_seq = "9223372036854775806";
|
||||
stop.localStopProof.database.rows[0].process_stopped_seq = "9223372036854775807";
|
||||
})}`, `--noise-report=${noise}`, "--require-acceptance"]);
|
||||
assert.equal(wide.status, 0, wide.stderr);
|
||||
for (const [name, change, message] of [
|
||||
["smoke", report => { report.qualificationOnly = true; }, /Smoke cannot qualify/],
|
||||
["source", report => { report.serverArtifact = "source-qualification"; }, /compiled server/],
|
||||
["profile", report => { report.diagnosticOnly = true; }, /Diagnostic profiles/],
|
||||
["profile-ineligible", report => { report.performanceAcceptanceEligible = false; }, /Profile-only evidence/],
|
||||
["missing-retained-prelude", report => { report.cases.find(row => row.prelude).preludeProof.individualDeltas--; }, /stored prelude deltas/],
|
||||
["legacy-protocol", report => { delete report.protocolVersion; }, /v2 clock protocol/],
|
||||
["legacy-protocol", report => { delete report.protocolVersion; }, /v4 mixed clock protocol/],
|
||||
["old-local-protocol", report => { report.protocolVersion = "v2-compact-markers-original-clock-stream"; report.transport = "local"; }, /v4 mixed clock protocol/],
|
||||
["old-remote-protocol", report => { report.protocolVersion = "v3-supported-remote-original-clock-stream"; }, /v4 mixed clock protocol/],
|
||||
["wrong-transport", report => { report.transport = "local"; }, /original local Stop transport/],
|
||||
["missing-transport-policy", report => { delete report.transportPolicy; }, /explicit transport/],
|
||||
["fake-remote-stop", report => { report.cases[5].transport = "sandbox-remote"; }, /Wrong per-case transport/],
|
||||
["local-loaded-workload", report => { report.cases[3].transport = "local"; }, /Wrong per-case transport/],
|
||||
["wrong-case-environment", report => { report.cases[5].environmentId = report.remote.environmentId; }, /normal environment/],
|
||||
["missing-local-selection", report => { delete report.local; }, /local Stop runtime provenance/],
|
||||
["shared-control-agent", report => { report.local.controlAgentId = report.remote.agentIds[0]; }, /separate normal control agent/],
|
||||
["changed-local-command", report => { report.local.commandSha256 = "9".repeat(64); }, /Local Stop fixture changed/],
|
||||
["changed-provider-cleanup", report => { report.remote.plugin.assetHashes["lease-cleanup.js"] = "9".repeat(64); }, /Provider cleanup build changed/],
|
||||
["unregistered-resource-budget", report => { report.resourceBudgets.providerReadinessMs = 600000; }, /registered remote setup\/resource budgets/],
|
||||
["missing-normal-remote", report => { delete report.remote; }, /normal remote runtime provenance/],
|
||||
["missing-plugin", report => { delete report.remote.plugin; }, /plugin provenance/],
|
||||
["resolver-override", report => { report.remote.selection = "forced external controller flag"; }, /normal API remote environment selection/],
|
||||
["tls-unverified", report => { report.remote.tls.peerVerification = false; }, /TLS peer verification/],
|
||||
["plaintext-remote", report => { report.remote.runnerPublicUrl = "ws://127.0.0.1:32100"; }, /endpoint must use verified WSS/],
|
||||
["foreign-remote", report => { report.remote.runnerPublicUrl = "wss://example.com:32100"; }, /owned loopback/],
|
||||
["different-provider-artifact", report => { report.remote.provider.nodeSha256 = "9".repeat(64); }, /artifact changed/],
|
||||
["fixed-workload-changed", report => { report.cases[3].runs = 10; }, /Fixed workload changed/],
|
||||
["missing-controller-authority", report => { delete report.cases[3].controllerAuthority; }, /controller authority proof/],
|
||||
["authority-writable", report => { report.cases[3].controllerAuthority.observedReadOnly = "off"; }, /read-only transaction/],
|
||||
["authority-wrong-isolation", report => { report.cases[3].controllerAuthority.observedIsolation = "read committed"; }, /repeatable-read isolation/],
|
||||
["authority-late", report => { report.cases[3].controllerAuthority.checkedAtMs = 1000; }, /before the measured epoch/],
|
||||
["authority-fractional-closed-time", report => { report.cases[3].controllerAuthority.connectionClosedAtMs = 999.5; }, /before the measured epoch/],
|
||||
["authority-missing-run", report => { report.cases[3].controllerAuthority.rows.pop(); }, /Every remote run/],
|
||||
["authority-wrong-run", report => { report.cases[3].controllerAuthority.rows[99].run_id = report.cases[3].controllerAuthority.rows[0].run_id; }, /run set mismatch/],
|
||||
["authority-wrong-company", report => { report.cases[3].controllerAuthority.rows[99].company_id = "other"; }, /company mismatch/],
|
||||
["authority-wrong-environment", report => { report.cases[3].controllerAuthority.rows[99].environment_id = "other"; }, /environment mismatch/],
|
||||
["authority-wrong-provider", report => { report.cases[3].controllerAuthority.rows[99].provider = "fake"; }, /fake-plugin/],
|
||||
["authority-missing-checkpoint", report => { report.cases[3].controllerAuthority.rows[99].checkpoint_part = null; }, /checkpoint header/],
|
||||
["local-wrong-checkpoint-authority", report => { report.cases[5].controllerAuthority.kind = "database-controller-checkpoint"; }, /Wrong checkpoint authority/],
|
||||
["local-fake-database-header", report => { report.cases[5].controllerAuthority.rows[0].checkpoint_part = "header"; }, /host-file authority/],
|
||||
["missing-local-journal", report => { delete report.cases[5].controllerAuthority.localJournal; }, /owned local journal authority/],
|
||||
["local-journal-wrong-run", report => { report.cases[5].controllerAuthority.localJournal.identity.runId = "other"; }, /Local journal identity mismatch/],
|
||||
["local-journal-wrong-workspace", report => { report.cases[5].controllerAuthority.rows[0].admitted_execution_workspace_id = report.remote.environmentId; }, /Local journal identity mismatch/],
|
||||
["local-journal-bookkeeping-alias", report => { report.cases[5].controllerAuthority.localJournal.identity.environmentLeaseId = report.cases[5].controllerAuthority.rows[0].lease_id; }, /Local journal identity mismatch/],
|
||||
["local-missing-admitted-workspace", report => { delete report.cases[5].controllerAuthority.rows[0].admitted_execution_workspace_id; }, /admitted local workspace binding/],
|
||||
["local-journal-wrong-policy", report => { report.cases[5].controllerAuthority.localJournal.checkpointEventPolicy = "control-v1"; }, /original local raw transport/],
|
||||
["missing-wrapper-provenance", report => { delete report.cases[3].providerClocks; }, /wrapper launcher and PID/],
|
||||
["remote-used-local-wrapper", report => { report.cases[3].providerClocks[99].launcher = "host-local"; }, /wrapper launcher/],
|
||||
["local-used-remote-wrapper", report => { report.cases[5].providerClocks[0].launcher = "staged-remote"; }, /wrapper launcher/],
|
||||
["missing-hello-pid", report => { delete report.cases[3].clockEvidence[99].providerPid; }, /hello provider PID/],
|
||||
["missing-local-identity-event", report => { delete report.cases[5].controllerAuthority.rows[0].process_identity; }, /targetKind|local/],
|
||||
["provider-authored-local-identity", report => { report.cases[5].controllerAuthority.rows[0].process_identity_source_event_id = "provider-event"; }, /server-authored/],
|
||||
["missing-local-physical-proof", report => { delete report.cases[5].localStopProof; }, /physical Stop/],
|
||||
["local-provider-still-alive", report => { report.cases[5].localStopProof.processes.providerAbsent = false; }, /remains alive/],
|
||||
["local-group-still-alive", report => { report.cases[5].localStopProof.processes.groupAbsent = false; }, /false.*true|false !== true/],
|
||||
["local-database-ack-missing", report => { report.cases[5].localStopProof.database.rows[0].cancellation_dispatch_state = "pending"; }, /acknowledged/],
|
||||
["local-cleared-identity-no-receipt", report => { delete report.cases[5].localStopProof.database.rows[0].process_stopped_seq; }, /BigInt/],
|
||||
["local-bigint-lexicographic-false-pass", report => {
|
||||
report.cases[5].controllerAuthority.rows[0].process_identity_seq = "90071992547409920";
|
||||
report.cases[5].localStopProof.database.rows[0].process_identity_seq = "90071992547409920";
|
||||
report.cases[5].localStopProof.database.rows[0].process_stopped_seq = "9007199254740993";
|
||||
}, /server stop receipt/],
|
||||
["local-provider-stop-receipt", report => { report.cases[5].localStopProof.database.rows[0].process_stopped_source_event_id = "provider"; }, /null/],
|
||||
["local-lease-cleanup-failed", report => { report.cases[5].localStopProof.lease.cleanupStatus = "failed"; }, /Local Stop lease cleanup/],
|
||||
["local-lease-wrong-binding", report => { report.cases[5].localStopProof.lease.heartbeatRunId = "other"; }, /other/],
|
||||
["missing-lease-cleanup", report => { delete report.cases[3].remoteLeases; }, /lease cleanup proof/],
|
||||
["lease-pending", report => { report.cases[3].remoteLeases[99].status = "pending_cleanup"; }, /cleanup remains unsettled/],
|
||||
["lease-wrong-binding", report => { report.cases[3].remoteLeases[99].providerLeaseId = "other"; }, /fake-plugin/],
|
||||
["lease-check-before-timing", report => { report.cases[3].remoteLeasesCheckedAtMs = report.cases[3].settledAtMs - 1; }, /after recorded timing boundaries/],
|
||||
["stop-timing-restamped", report => { report.cases[5].stopAcknowledgedAtMs++; }, /timing boundary changed/],
|
||||
["unqualified-admission", report => { delete report.admissionProtocol; }, /admission protocol/],
|
||||
["old-sequential-admission", report => { report.admissionProtocol = "sequential-provider-and-clock-readiness-before-common-epoch"; }, /admission protocol/],
|
||||
["missing-activation-concurrency", report => { delete report.setupActivationConcurrency; }, /activation concurrency of two/],
|
||||
["excessive-activation-concurrency", report => { report.setupActivationConcurrency = 10; }, /activation concurrency of two/],
|
||||
["serial-activation-concurrency", report => { report.setupActivationConcurrency = 1; }, /activation concurrency of two/],
|
||||
["wrong-case-activation-concurrency", report => { report.cases[3].setupActivationConcurrency = 1; }, /case's qualified setup activation/],
|
||||
["extended-shared-readiness-deadline", report => { report.cases[3].readinessDeadlineMs++; }, /unchanged shared readiness deadline/],
|
||||
["authority-before-all-ready", report => { report.cases[3].controllerAuthority.checkedAtMs = 699; }, /authority after every provider/],
|
||||
["missing-live-provider-get", report => { report.cases[3].participants--; }, /provider GET must remain open/],
|
||||
["missing-activation-record", report => { report.cases[3].activations.pop(); }, /activation timing/],
|
||||
["wrong-activation-task", report => { report.cases[3].activations[99].issueId = "other"; }, /task must match actual controller/],
|
||||
["wrong-activation-token", report => { report.cases[3].activations[99].token = "other"; }, /actual provider clock/],
|
||||
["unfinished-activation", report => { report.cases[3].activations[99].status = "pending"; }, /unfinished setup activation/],
|
||||
["unfinished-wave", report => { report.cases[3].activationWaves[49].statuses = ["rejected"]; }, /failed or unfinished setup wave/],
|
||||
["overlapping-waves", report => { report.cases[3].activationWaves[1].startedAtMs = 110; }, /whole setup wave before/],
|
||||
["oversized-wave", report => { report.cases[3].activationWaves[0].indices.push(3); }, /Wrong setup wave participants/],
|
||||
["late-readiness-observation", report => { report.cases[3].activations[99].readinessObservedAtMs = 1001; }, /pre-epoch activation/],
|
||||
["missing-provider-clock-hello", report => { report.cases[3].clockEvidence[99].hello = false; }, /actual provider needs its clock hello/],
|
||||
["restamped-provider-readiness", report => { report.cases[3].clockEvidence[99].readinessObservedAtMs++; }, /actual clock observations/],
|
||||
["unqualified-readiness-observation", report => { delete report.setupReadinessPollingMs; }, /readiness observation interval/],
|
||||
["unfinished-cleanup", report => { report.cleanup.complete = false; }, /completed fixture cleanup/],
|
||||
["cleanup-error", report => { report.cleanup.errors.push({ step: "restore", message: "timeout" }); }, /cleanup errors/],
|
||||
@@ -138,5 +297,29 @@ test("browser CLI rejects the three-pairs-above-noise counterexample and ineligi
|
||||
const rejected = run([`--pairs=${await manifest(name, Array(5).fill(0.30), false, change)}`, `--noise-report=${noise}`, "--require-acceptance"]);
|
||||
assert.notEqual(rejected.status, 0); assert.match(rejected.stderr, message);
|
||||
}
|
||||
for (const [label, value] of [["null", null], ["missing", undefined], ["invalid", "not-a-sequence"], ["fractional", 1.5]]) {
|
||||
for (const field of ["identity", "stopped"]) {
|
||||
const rejected = run([`--pairs=${await manifest(`bigint-${field}-${label}`, Array(5).fill(0.30), false, report => {
|
||||
const stop = report.cases[5];
|
||||
if (field === "identity") {
|
||||
stop.controllerAuthority.rows[0].process_identity_seq = value;
|
||||
stop.localStopProof.database.rows[0].process_identity_seq = value;
|
||||
} else stop.localStopProof.database.rows[0].process_stopped_seq = value;
|
||||
})}`, `--noise-report=${noise}`, "--require-acceptance"]);
|
||||
assert.notEqual(rejected.status, 0); assert.match(rejected.stderr, /BigInt/);
|
||||
}
|
||||
}
|
||||
for (const [name, change, message] of [
|
||||
["old-noise-protocol", report => { report.protocolVersion = "v2-compact-markers-original-clock-stream"; }, /different mixed protocol/],
|
||||
["local-noise-transport", report => { report.transport = "local"; }, /different transport/],
|
||||
["changed-noise-budget", report => { report.resourceBudgets.providerReadinessMs++; }, /budgets changed after A\/A/],
|
||||
["changed-noise-transport-policy", report => { report.transportPolicy.operatorStop = "sandbox-remote"; }, /Transport policy changed after A\/A/],
|
||||
["changed-noise-activation-concurrency", report => { report.setupActivationConcurrency = 10; }, /concurrency changed after A\/A/],
|
||||
]) {
|
||||
const changed = JSON.parse(await readFile(noise, "utf8")); change(changed);
|
||||
const file = await save(name + ".json", changed);
|
||||
const rejected = run([`--pairs=${await manifest(name, Array(5).fill(0.30))}`, `--noise-report=${file}`, "--require-acceptance"]);
|
||||
assert.notEqual(rejected.status, 0); assert.match(rejected.stderr, message);
|
||||
}
|
||||
} finally { await rm(directory, { recursive: true, force: true }); }
|
||||
});
|
||||
@@ -1,5 +1,7 @@
|
||||
# Native ingestion experiment helpers
|
||||
|
||||
For fresh cursor results and the current interleaved source reproduction, see [the new source report](../../doc/benchmarks/2026-11-03-native-event-ingestion-interleaved.md). The frozen runner below preserves the earlier experiment. Current application performance requires the separately qualified H4c driver and fresh comparisons; its gates remain open.
|
||||
|
||||
These helpers reproduce the experiment registered on November 3, 2026. They
|
||||
retain its immutable application revisions, harness revision, workload hashes,
|
||||
acceptance checks and failure behavior. They are not a benchmark against an
|
||||
@@ -22,7 +24,7 @@ Packaging changes: the diagnostic runner imports its renamed adjacent
|
||||
`diagnostic-preload.mjs`; its dry-run message uses the committed path; the
|
||||
acceptance runner rejects both the original and packaged preload filenames in
|
||||
`NODE_OPTIONS`. Application, protocol, compiled-anchor and gate hashes are
|
||||
unchanged. The runtime comparison helper and diagnostic preload are exact copies.
|
||||
unchanged. The diagnostic preload remains an exact original copy. The runtime comparison helper later gained the optional explicit CI base described below; its SHA above records its original-copy provenance.
|
||||
|
||||
The frozen experiment revisions are:
|
||||
|
||||
@@ -151,12 +153,7 @@ python3 "$TOOLS_CHECKOUT/scripts/native-ingestion/verify-delivery-runtime.py" \
|
||||
--delivery=3b77c1ca29467fbd742e9b6df4885bccef0acf08
|
||||
```
|
||||
|
||||
For B2, supply its frozen and delivery commits instead. This is a read-only Git
|
||||
source/input comparison with the two documented test-typecheck script exceptions
|
||||
and genuine CI helpers pinned to `8ab89f41e499032e3e55d261f457a3463d754a1d`.
|
||||
It does not compare installed dependencies or compiled artifacts; those require
|
||||
their separate build/equivalence receipts. Later delivery revisions must remain
|
||||
explicit full commit IDs.
|
||||
For B2, supply its frozen and delivery commits instead. This read-only Git source/input comparison preserves the two documented test-typecheck script exceptions. CI helpers and those two scripts must match the immutable `--ci-base` commit, which defaults to `8ab89f41e499032e3e55d261f457a3463d754a1d` for historical reproduction. Later upstream helpers require an explicit full commit ID; the current evidence-only914→d05 comparison uses the tested frozen914 commit. The result reports both the selected-base match and the independent historical8ab match. Installed dependencies and compiled artifacts need separate equivalence receipts. Later delivery revisions must remain explicit full commit IDs.
|
||||
|
||||
## Evidence retention
|
||||
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawn, execFileSync } from 'node:child_process';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { readFile, writeFile, mkdir, open } from 'node:fs/promises';
|
||||
import net from 'node:net';
|
||||
import path from 'node:path';
|
||||
|
||||
// APPLICATION ONLY, prospective draft: execution still requires root release after H4c full qualification.
|
||||
const option = name => process.argv.find(value => value.startsWith(`--${name}=`))?.slice(name.length + 3);
|
||||
const base = '/home/ubuntu/native-ingestion';
|
||||
const harnessRoot = `${base}/harness-h4c`;
|
||||
const harnessRevision = '7d7e7a88d562a95a1d407b76474763b666c181f5';
|
||||
const trees = { B0: `${base}/ce33-baseline`, B1: `${base}/ce33-cursors`, B2: `${base}/ce33-batching` };
|
||||
const revisions = { B0: 'a36567f9fbf805ad72207d54f539516e3121741e', B1: 'd8cd3be0aeed4f9fb93ecc3cf725f1d46d5a0310', B2: '914129afb141d81c0d6d8c5f0ea11c20bd69a3da' };
|
||||
const browserHash = '5db4129bbdc1fa38775121aff5d0e68a8a11762519e41cfdf36eacd90ba7256b';
|
||||
const comparatorHash = '01e0de114c886c255aaa7062a63f3e48788d8a83ce50921d672b2d661baef43b';
|
||||
const lockHash = '54a81b2ed3f1f85eaa085c034560ac248e384db34fbc7aba8c5a3846c29b3ce8';
|
||||
const out = option('output-root') ?? `${base}/evidence/ce33/browser`;
|
||||
const hash = bytes => createHash('sha256').update(bytes).digest('hex');
|
||||
const getJson = async file => JSON.parse(await readFile(file, 'utf8'));
|
||||
const saveJson = async (file, value) => writeFile(file, JSON.stringify(value, null, 2) + '\n');
|
||||
let commandIndex = 0; const commands = [];
|
||||
let release;
|
||||
const stages = ['browser-aa-b0','browser-b0-b1','browser-aa-b1','browser-b1-b2','browser-b0-b2'];
|
||||
|
||||
|
||||
async function checkFrozen() {
|
||||
for (const key of ['DATABASE_URL','PAPERCLIP_RUNNER_STATE_DIR','PAPERCLIP_INGESTION_SERVER_COMMAND',
|
||||
'PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_DIR','PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_FS',
|
||||
'PAPERCLIP_NATIVE_INGESTION_DIAGNOSTIC_INSPECTOR','PAPERCLIP_INGESTION_BROWSER_CAPTURE','NODE_OPTIONS'])
|
||||
assert.ok(!process.env[key], `Acceptance rejects ambient ${key}`);
|
||||
const qualifiedBytes = await readFile(`${base}/ce33-builds-qualified.json`);
|
||||
if (release) assert.equal(hash(qualifiedBytes), release.qualifiedInventorySha256, 'Qualified build inventory changed after root release');
|
||||
const qualified = JSON.parse(qualifiedBytes);
|
||||
for (const [label, tree] of Object.entries(trees)) {
|
||||
assert.equal(execFileSync('git',['rev-parse','HEAD'],{cwd:tree,encoding:'utf8'}).trim(),revisions[label]);
|
||||
assert.equal(execFileSync('git',['status','--short'],{cwd:tree,encoding:'utf8'}).trim(),'');
|
||||
assert.equal((await getJson(path.join(tree,'server/dist/build-info.json'))).commit,revisions[label]);
|
||||
assert.equal(qualified.results.find(row=>row.variant===label).revision,revisions[label]);
|
||||
}
|
||||
assert.equal(execFileSync('git',['rev-parse','HEAD'],{cwd:harnessRoot,encoding:'utf8'}).trim(),harnessRevision);
|
||||
assert.equal(execFileSync('git',['status','--short'],{cwd:harnessRoot,encoding:'utf8'}).trim(),'');
|
||||
const protocol = ['native-event-ingestion.journey.ts','playwright-native-ingestion.config.ts','fixtures/in-feed-codex.mjs',
|
||||
'fixtures/native-ingestion-bin/codex','fixtures/composer-stop-codex-loader.mjs','fixtures/native-ingestion-bootstrap.mjs',
|
||||
'fixtures/native-ingestion-remote-server.mjs','fixtures/native-ingestion-protocol-qualification.mjs'];
|
||||
assert.equal(hash(Buffer.concat(await Promise.all(protocol.map(file=>readFile(path.join(harnessRoot,'tests/e2e',file)))))),browserHash);
|
||||
assert.equal(hash(await readFile(path.join(harnessRoot,'scripts/compare-native-ingestion-browser.ts'))),comparatorHash);
|
||||
assert.equal(hash(await readFile(path.join(harnessRoot,'pnpm-lock.yaml'))),lockHash);
|
||||
}
|
||||
|
||||
async function execute(step) {
|
||||
const index = ++commandIndex, startedAt = new Date().toISOString();
|
||||
const log = path.join(out, `${step.stage}.log`);
|
||||
const command = { index, stage: step.stage, cwd: step.cwd, executable: process.execPath, args: step.args,
|
||||
environmentOverrides: step.env ?? {}, startedAt, log };
|
||||
commands.push(command); await saveJson(path.join(out, `${option('stage')}-commands.json`), commands);
|
||||
console.log(`START ${step.stage} ${startedAt}; log=${log}`);
|
||||
const handle = await open(log, 'wx');
|
||||
const stream = handle.createWriteStream();
|
||||
const child = spawn(process.execPath, step.args, { cwd: step.cwd, env: { ...process.env, ...step.env }, stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
command.childPid = child.pid;
|
||||
await saveJson(path.join(out, `${option('stage')}-commands.json`), commands);
|
||||
const interrupt = () => child.kill('SIGINT'), terminate = () => child.kill('SIGTERM');
|
||||
process.once('SIGINT', interrupt); process.once('SIGTERM', terminate);
|
||||
const watchdogMillis = step.stage.startsWith('service-') ? 30 * 60_000 : 20 * 60_000;
|
||||
const watchdog = setTimeout(() => { command.watchdogExpired = true; child.kill('SIGTERM'); }, watchdogMillis);
|
||||
child.stdout.pipe(stream, { end: false }); child.stderr.pipe(stream, { end: false });
|
||||
const exit = await new Promise((resolve, reject) => { child.once('error', reject); child.once('close', (code, signal) => resolve({ code, signal })); });
|
||||
clearTimeout(watchdog); process.off('SIGINT', interrupt); process.off('SIGTERM', terminate);
|
||||
await new Promise(resolve => stream.end(resolve));
|
||||
Object.assign(command, { finishedAt: new Date().toISOString(), ...exit });
|
||||
await saveJson(path.join(out, `${option('stage')}-commands.json`), commands);
|
||||
console.log(`END ${step.stage} code=${exit.code} signal=${exit.signal ?? 'none'}`);
|
||||
assert.equal(command.watchdogExpired, undefined, 'Resource watchdog expired; run is incomplete, not acceptance');
|
||||
assert.equal(exit.code, 0, `Stage ${step.stage} failed; preserve ${log} and diagnose before continuing`);
|
||||
return command;
|
||||
}
|
||||
|
||||
async function freePort() {
|
||||
const server = net.createServer();
|
||||
await new Promise((resolve, reject) => { server.once('error', reject); server.listen(0, '127.0.0.1', resolve); });
|
||||
const port = server.address().port;
|
||||
await new Promise(resolve => server.close(resolve));
|
||||
return port;
|
||||
}
|
||||
|
||||
async function extractEvidence(directory) {
|
||||
const report = await getJson(path.join(directory, 'playwright.json'));
|
||||
const attachments = [];
|
||||
const visit = node => {
|
||||
if (Array.isArray(node)) for (const child of node) visit(child);
|
||||
else if (node && typeof node === 'object') {
|
||||
if (Array.isArray(node.attachments)) attachments.push(...node.attachments.filter(value => value.name === 'native-ingestion-evidence'));
|
||||
for (const [key, child] of Object.entries(node)) if (key !== 'attachments') visit(child);
|
||||
}
|
||||
};
|
||||
visit(report); assert.equal(attachments.length, 1, 'Exactly one complete native-ingestion evidence attachment is required');
|
||||
const attachment = attachments[0];
|
||||
const evidence = attachment.body ? JSON.parse(Buffer.from(attachment.body, 'base64').toString()) : await getJson(attachment.path);
|
||||
assert.equal(evidence.complete, true);
|
||||
assert.equal(evidence.transport, 'sandbox-remote-with-local-stop');
|
||||
assert.equal(evidence.cleanup?.complete, true);
|
||||
assert.deepEqual(evidence.cleanup?.errors, []); assert.equal(evidence.measurementSha256, browserHash);
|
||||
assert.equal(evidence.protocolVersion, 'v4-remote-streaming-original-local-stop-clock-stream');
|
||||
assert.equal(evidence.harnessProvenance?.sourceRevision, harnessRevision);
|
||||
assert.equal(evidence.harnessProvenance?.sourceStatus, '');
|
||||
const file = path.join(directory, 'native-ingestion-evidence.json'); await saveJson(file, evidence);
|
||||
return { file, evidence };
|
||||
}
|
||||
|
||||
async function browserRun(stage, tree, smoke) {
|
||||
await checkFrozen();
|
||||
const directory = path.join(out, stage); await mkdir(directory, { recursive: false });
|
||||
const port = await freePort();
|
||||
await execute({ stage, cwd: harnessRoot, args: ['node_modules/@playwright/test/cli.js', 'test', '--config', 'tests/e2e/playwright-native-ingestion.config.ts'],
|
||||
env: { PAPERCLIP_RUNNER_BINARY: path.join(trees[tree], 'packages/paperclip-runner/runner/target/release/paperclip-runnerd'),
|
||||
PAPERCLIP_RUNNER_REMOTE_BINARY_PATH: path.join(trees[tree], 'packages/paperclip-runner/runner/target/release/paperclip-runnerd'), PAPERCLIP_INGESTION_TRANSPORT: 'sandbox-remote-with-local-stop', PAPERCLIP_E2E_PORT: String(port), PAPERCLIP_INGESTION_SMOKE: smoke ? '1' : '0', PAPERCLIP_INGESTION_VARIANT: `${stage}-${tree}`,
|
||||
PAPERCLIP_INGESTION_APP_ROOT: trees[tree], PAPERCLIP_INGESTION_EVIDENCE_DIR: directory,
|
||||
PAPERCLIP_INGESTION_DRIVER_DEPENDENCIES: 'Independent H4c locked offline ignore-scripts driver install; measured applications own independent normal locked installs/builds' } });
|
||||
const { file, evidence } = await extractEvidence(directory);
|
||||
assert.equal(evidence.sourceRevision, revisions[tree]); assert.equal(evidence.compiledBuildRevision, revisions[tree]);
|
||||
assert.equal(evidence.serverArtifact, 'compiled'); assert.equal(evidence.qualificationOnly, smoke);
|
||||
assert.equal(evidence.diagnosticOnly, false); assert.equal(evidence.performanceAcceptanceEligible, true);
|
||||
assert.equal(evidence.browserRecording?.trace, 'off'); assert.equal(evidence.browserRecording?.video, 'off');
|
||||
assert.equal(evidence.browserRecording?.screencast, false);
|
||||
if (!smoke) {
|
||||
const loaded = evidence.cases.find(item => item.name === 'burst512');
|
||||
assert.equal(loaded.runs, 100); assert.equal(loaded.participants, 100); assert.equal(loaded.durable.length, 100);
|
||||
}
|
||||
// An invocation is complete only when its own HTTP server has released the port.
|
||||
const listener = net.createServer();
|
||||
await new Promise((resolve, reject) => { listener.once('error', reject); listener.listen(port, '127.0.0.1', resolve); });
|
||||
await new Promise(resolve => listener.close(resolve));
|
||||
return file;
|
||||
}
|
||||
|
||||
async function browserSeries(stage, baseline, candidate, aa = false, expectation = 'improvement') {
|
||||
const manifest = [], file = path.join(out, `${stage}-pairs.json`);
|
||||
const noisePath = path.join(out, `browser-aa-${baseline.toLowerCase()}.json`);
|
||||
const checkNoise = async () => {
|
||||
assert.match(release.noiseReportSha256, /^[a-f0-9]{64}$/, 'Candidate release must freeze the reviewed A/A bytes');
|
||||
const bytes = await readFile(noisePath);
|
||||
assert.equal(hash(bytes), release.noiseReportSha256, 'Reviewed A/A noise report changed after release');
|
||||
const noise = JSON.parse(bytes);
|
||||
assert.equal(noise.complete, true); assert.equal(noise.phase, 'aa');
|
||||
};
|
||||
if (!aa) await checkNoise();
|
||||
for (let pair = 0; pair < 5; pair++) {
|
||||
const entry = {};
|
||||
const arms = pair % 2 ? [['candidate', candidate], ['baseline', baseline]] : [['baseline', baseline], ['candidate', candidate]];
|
||||
for (const [arm, tree] of arms) entry[`${arm}File`] = await browserRun(`${stage}-pair${pair + 1}-${arm}`, tree, false);
|
||||
manifest.push(entry); await saveJson(file, manifest);
|
||||
}
|
||||
if (!aa) await checkNoise();
|
||||
await execute({ stage: `${stage}-comparison`, cwd: harnessRoot, args: ['scripts/compare-native-ingestion-browser.ts', `--phase=${aa ? 'aa' : 'compare'}`,
|
||||
`--expect=${expectation}`, `--pairs=${file}`, `--output=${out}/${stage}.json`,
|
||||
...(aa ? [] : [`--noise-report=${noisePath}`, '--require-acceptance'])] });
|
||||
}
|
||||
|
||||
const stage = option('stage');
|
||||
if (!process.argv.includes('--execute')) {
|
||||
console.log(JSON.stringify({draft:true, stages, revisions, harnessRevision, browserHash, comparatorHash,
|
||||
note:'No execution in dry run. Root must release after latest full H4c qualification. Separate A/A suitability review before each candidate series.'},null,2));
|
||||
process.exit(0);
|
||||
}
|
||||
assert.ok(stages.includes(stage), 'Select exactly one browser stage');
|
||||
await checkFrozen();
|
||||
await mkdir(out,{recursive:true});
|
||||
// The owner supplies a prospectively frozen root release and full qualification evidence.
|
||||
const releaseBytes = await readFile(option('release'));
|
||||
release = JSON.parse(releaseBytes);
|
||||
assert.match(release.qualifiedInventorySha256, /^[a-f0-9]{64}$/);
|
||||
assert.equal(release.orchestratorSha256, hash(await readFile(new URL(import.meta.url))), 'Orchestrator changed after release');
|
||||
await checkFrozen();
|
||||
assert.equal(release.stage,stage); assert.equal(release.harnessRevision,harnessRevision);
|
||||
assert.deepEqual(release.revisions,revisions); assert.equal(release.protocolSha256,browserHash);
|
||||
assert.equal(release.comparatorSha256,comparatorHash); assert.equal(release.authorized,true);
|
||||
await saveJson(path.join(out, `${stage}-release-record.json`), {
|
||||
...release, releaseFileSha256: hash(releaseBytes), recordedAt: new Date().toISOString(),
|
||||
});
|
||||
if(stage==='browser-aa-b0') await browserSeries(stage,'B0','B0',true);
|
||||
else if(stage==='browser-aa-b1') await browserSeries(stage,'B1','B1',true);
|
||||
else if(stage==='browser-b0-b1') await browserSeries(stage,'B0','B1',false,'nonregression');
|
||||
else if(stage==='browser-b1-b2') await browserSeries(stage,'B1','B2');
|
||||
else if(stage==='browser-b0-b2') await browserSeries(stage,'B0','B2');
|
||||
@@ -6,16 +6,19 @@ import json
|
||||
import re
|
||||
import subprocess
|
||||
|
||||
CI_BASE = "8ab89f41e499032e3e55d261f457a3463d754a1d"
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--repo", required=True)
|
||||
parser.add_argument("--frozen", required=True)
|
||||
parser.add_argument("--delivery", required=True)
|
||||
parser.add_argument("--ci-base", default=CI_BASE,
|
||||
help="Immutable commit whose CI helpers and two test scripts must match (default: %(default)s)")
|
||||
args = parser.parse_args()
|
||||
for ref in [args.frozen, args.delivery]:
|
||||
for ref in [args.frozen, args.delivery, args.ci_base]:
|
||||
if not re.fullmatch(r"[a-f0-9]{40}", ref):
|
||||
parser.error("Use immutable full commit IDs")
|
||||
|
||||
CI_BASE = "8ab89f41e499032e3e55d261f457a3463d754a1d"
|
||||
SCRIPT_EXCEPTIONS = {"test:e2e:runner:typecheck", "test:lifecycle-baseline:typecheck"}
|
||||
CI_ONLY_SERVER_FILES = {"server/scripts/typecheck-layers.mjs"}
|
||||
|
||||
@@ -52,12 +55,23 @@ for name, expanded in [("approvedScope", False), ("expandedServerBuildInputs", T
|
||||
assert frozen == delivery, (name, frozen, delivery)
|
||||
output[name] = {"equal": True, "frozen": frozen, "delivery": delivery}
|
||||
|
||||
approved_package = json.loads(git("show", CI_BASE + ":package.json"))
|
||||
delivery_package = json.loads(git("show", args.delivery + ":package.json"))
|
||||
for key in SCRIPT_EXCEPTIONS:
|
||||
assert delivery_package["scripts"][key] == approved_package["scripts"][key], key
|
||||
for path in ["scripts/typecheck-low-memory.mjs", "scripts/typecheck-test-projects.mjs", *CI_ONLY_SERVER_FILES]:
|
||||
assert git("show", args.delivery + ":" + path) == git("show", CI_BASE + ":" + path), path
|
||||
output["realCiHelpersMatchApproved8ab"] = True
|
||||
|
||||
def ci_helper_differences(base):
|
||||
approved_package = json.loads(git("show", base + ":package.json"))
|
||||
differences = ["package.json:scripts." + key for key in sorted(SCRIPT_EXCEPTIONS)
|
||||
if delivery_package["scripts"][key] != approved_package["scripts"][key]]
|
||||
for path in ["scripts/typecheck-low-memory.mjs", "scripts/typecheck-test-projects.mjs", *sorted(CI_ONLY_SERVER_FILES)]:
|
||||
if git("show", args.delivery + ":" + path) != git("show", base + ":" + path):
|
||||
differences.append(path)
|
||||
return differences
|
||||
|
||||
differences = ci_helper_differences(args.ci_base)
|
||||
assert not differences, ("ciHelpers", args.ci_base, differences)
|
||||
output["ciBase"] = args.ci_base
|
||||
output["realCiHelpersMatchSelectedBase"] = True
|
||||
historical_differences = differences if args.ci_base == CI_BASE else ci_helper_differences(CI_BASE)
|
||||
output["realCiHelpersMatchApproved8ab"] = not historical_differences
|
||||
output["ciHelperDifferencesFromApproved8ab"] = historical_differences
|
||||
output["allChangedPaths"] = git("diff", "--name-only", args.frozen, args.delivery).decode().splitlines()
|
||||
print(json.dumps(output, indent=2))
|
||||
@@ -89,8 +89,13 @@ function openIngestionClock(plan, gate) {
|
||||
closed = true;
|
||||
try { request.end(); } catch (error) { fail(error); }
|
||||
};
|
||||
const launcher = process.env.PAPERCLIP_INGESTION_FIXTURE_LAUNCHER;
|
||||
const providerPid = Number(process.env.PAPERCLIP_INGESTION_FIXTURE_PID);
|
||||
if (!['host-local', 'staged-remote'].includes(launcher) || providerPid !== process.pid) {
|
||||
throw new Error('Ingestion fixture requires its actual wrapper launcher and process identity');
|
||||
}
|
||||
write({ type: 'hello', nonce: plan.nonce, markerCode: plan.markerCode,
|
||||
events: plan.events, intervalMs: plan.intervalMs });
|
||||
events: plan.events, intervalMs: plan.intervalMs, launcher, providerPid });
|
||||
return { write, close, done };
|
||||
}
|
||||
|
||||
|
||||
@@ -1,2 +1,4 @@
|
||||
#!/bin/sh
|
||||
export PAPERCLIP_INGESTION_FIXTURE_LAUNCHER=host-local
|
||||
export PAPERCLIP_INGESTION_FIXTURE_PID=$$
|
||||
exec node "$(dirname "$0")/../in-feed-codex.mjs" --ingestion-stream-fixture "$@"
|
||||
@@ -7,6 +7,7 @@ import { createInterface } from 'node:readline';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const fixture = fileURLToPath(new URL('./in-feed-codex.mjs', import.meta.url));
|
||||
const wrapper = fileURLToPath(new URL('./native-ingestion-bin/codex', import.meta.url));
|
||||
const output = process.argv[2];
|
||||
assert.ok(output, 'Pass an owned output JSON path');
|
||||
const reports = [];
|
||||
@@ -63,7 +64,7 @@ async function run({ name, count, intervalMs = 0, stopAfter, clockStatus = 200,
|
||||
await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
|
||||
const gate = `http://127.0.0.1:${server.address().port}/native-ingestion-gate/${key}`;
|
||||
const send = value => child.stdin.write(JSON.stringify(value) + '\n');
|
||||
child = spawn(process.execPath, [fixture, '--ingestion-stream-fixture'], { stdio: ['pipe', 'pipe', 'pipe'] });
|
||||
child = spawn(wrapper, [], { stdio: ['pipe', 'pipe', 'pipe'] });
|
||||
child.stderr.on('data', value => { stderr += value; });
|
||||
const watchdog = setTimeout(() => fail(new Error(`${name}: qualification watchdog`)), 10_000);
|
||||
createInterface({ input: child.stdout }).on('line', line => {
|
||||
@@ -104,7 +105,8 @@ async function run({ name, count, intervalMs = 0, stopAfter, clockStatus = 200,
|
||||
if (failure) throw failure;
|
||||
assert.equal(exit.code, clockStatus === 200 ? 0 : 1, stderr);
|
||||
assert.equal(clockPosts, 1); assert.equal(gateGets, 1);
|
||||
assert.deepEqual(hello, { type: 'hello', nonce, markerCode, events: count, intervalMs });
|
||||
assert.deepEqual(hello, { type: 'hello', nonce, markerCode, events: count, intervalMs,
|
||||
launcher: 'host-local', providerPid: child.pid });
|
||||
assert.equal(emissions.length, deltas.length);
|
||||
assert.ok(deltas.length > 0);
|
||||
for (let index = 0; index < emissions.length; index++) {
|
||||
@@ -127,6 +129,7 @@ async function run({ name, count, intervalMs = 0, stopAfter, clockStatus = 200,
|
||||
}
|
||||
if (clockStatus !== 200) assert.match(stderr, /Ingestion fixture clock .*HTTP 503/);
|
||||
reports.push({ name, exit, count, emitted: deltas.length, clockPosts, clockRecords: clockRecords.length,
|
||||
launcher: hello.launcher, providerPid: hello.providerPid,
|
||||
completedTextCharacters: completedItem?.text.length, terminal: terminal.status,
|
||||
uploadHeldUntilAfterTerminal: true, stopAcknowledgedBeforeUploadResponse: stopAfter ? true : undefined,
|
||||
explicitClockError: clockStatus !== 200 ? stderr.trim() : undefined });
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
// Test-only assembly of the normal remote sandbox path. The sandbox plugin
|
||||
// executes separate OS processes on this same host; TLS and runtime authority
|
||||
// follow the production direct-outbound contract.
|
||||
import assert from 'node:assert/strict';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { lstat, mkdir, readFile, realpath, writeFile } from 'node:fs/promises';
|
||||
import { createConnection } from 'node:net';
|
||||
import path from 'node:path';
|
||||
import { createServer } from 'node:tls';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
|
||||
assert.equal(process.env.NODE_ENV, 'test');
|
||||
assert.equal(process.env.PAPERCLIP_INGESTION_TRANSPORT, 'sandbox-remote-with-local-stop');
|
||||
assert.ok(process.env.PAPERCLIP_HOME && process.env.PAPERCLIP_CONFIG);
|
||||
assert.ok(!process.env.DATABASE_URL, 'Remote fixture requires its disposable embedded database');
|
||||
assert.ok(!process.env.PAPERCLIP_RUNNER_STATE_DIR, 'H4 uses the normal owned instance journal directory');
|
||||
assert.ok(path.resolve(process.env.PAPERCLIP_CONFIG).startsWith(`${path.resolve(process.env.PAPERCLIP_HOME)}${path.sep}`));
|
||||
const ownedHome = await realpath(process.env.PAPERCLIP_HOME);
|
||||
const ownedConfiguration = await realpath(process.env.PAPERCLIP_CONFIG);
|
||||
assert.ok(ownedConfiguration.startsWith(`${ownedHome}${path.sep}`));
|
||||
const configuration = JSON.parse(await readFile(ownedConfiguration, 'utf8'));
|
||||
assert.equal(configuration.database.mode, 'embedded-postgres');
|
||||
const databaseDirectory = path.resolve(configuration.database.embeddedPostgresDataDir);
|
||||
assert.ok((await realpath(path.dirname(databaseDirectory))).startsWith(`${ownedHome}${path.sep}`));
|
||||
try {
|
||||
const info = await lstat(databaseDirectory);
|
||||
assert.ok(info.isDirectory() && !info.isSymbolicLink());
|
||||
} catch (error) {
|
||||
if (error.code !== 'ENOENT') throw error;
|
||||
await mkdir(databaseDirectory, { mode: 0o700 });
|
||||
}
|
||||
assert.ok((await realpath(databaseDirectory)).startsWith(`${ownedHome}${path.sep}`));
|
||||
assert.ok(process.argv.includes('--compiled-entry=server/dist/index.js'));
|
||||
const appRoot = process.cwd();
|
||||
const revision = execFileSync('git', ['rev-parse', 'HEAD'], { cwd: appRoot, encoding: 'utf8' }).trim();
|
||||
const build = JSON.parse(await readFile(path.join(appRoot, 'server/dist/build-info.json'), 'utf8'));
|
||||
assert.equal(build.commit, revision, 'Remote fixture requires the matching compiled application');
|
||||
const directory = path.join(process.env.PAPERCLIP_HOME, 'ingestion-remote-fixture');
|
||||
await mkdir(directory, { recursive: true, mode: 0o700 });
|
||||
const key = path.join(directory, 'loopback-key.pem'), cert = path.join(directory, 'loopback-server.pem');
|
||||
const caKey = path.join(directory, 'loopback-ca-key.pem'), ca = path.join(directory, 'loopback-ca.pem');
|
||||
execFileSync('openssl', ['req', '-x509', '-newkey', 'rsa:2048', '-nodes', '-sha256', '-days', '2',
|
||||
'-keyout', caKey, '-out', ca, '-subj', '/CN=Paperclip ingestion fixture CA',
|
||||
'-addext', 'basicConstraints=critical,CA:TRUE', '-addext', 'keyUsage=critical,keyCertSign,cRLSign'], { stdio: 'ignore' });
|
||||
const request = path.join(directory, 'loopback-server.csr'), extensions = path.join(directory, 'loopback-server.ext');
|
||||
await writeFile(extensions, 'basicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth\nsubjectAltName=IP:127.0.0.1,DNS:localhost\n', { mode: 0o600 });
|
||||
execFileSync('openssl', ['req', '-new', '-newkey', 'rsa:2048', '-nodes', '-sha256',
|
||||
'-keyout', key, '-out', request, '-subj', '/CN=127.0.0.1'], { stdio: 'ignore' });
|
||||
execFileSync('openssl', ['x509', '-req', '-in', request, '-CA', ca, '-CAkey', caKey, '-CAcreateserial',
|
||||
'-out', cert, '-days', '2', '-sha256', '-extfile', extensions], { stdio: 'ignore' });
|
||||
execFileSync('openssl', ['verify', '-CAfile', ca, cert], { stdio: 'ignore' });
|
||||
const fixture = path.resolve(import.meta.dirname, 'in-feed-codex.mjs');
|
||||
const wrapper = path.join(directory, 'codex');
|
||||
const quote = value => `'${value.replaceAll("'", "'\\''")}'`;
|
||||
await writeFile(wrapper, `#!/bin/sh\nexport PAPERCLIP_INGESTION_FIXTURE_LAUNCHER=staged-remote\nexport PAPERCLIP_INGESTION_FIXTURE_PID=$$\nexec ${quote(process.execPath)} ${quote(fixture)} --ingestion-stream-fixture "$@"\n`, { mode: 0o700 });
|
||||
const runner = process.env.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH
|
||||
?? path.join(appRoot, 'packages/paperclip-runner/runner/target/release/paperclip-runnerd');
|
||||
const runnerMetadata = JSON.parse(execFileSync(runner, ['--build-metadata'], { encoding: 'utf8' }));
|
||||
const sockets = new Set();
|
||||
const proxy = createServer({ key: await readFile(key), cert: await readFile(cert) }, socket => {
|
||||
const upstream = createConnection({ host: '127.0.0.1', port: Number(process.env.PORT) });
|
||||
for (const connection of [socket, upstream]) {
|
||||
sockets.add(connection); connection.on('close', () => sockets.delete(connection));
|
||||
connection.on('error', () => { socket.destroy(); upstream.destroy(); });
|
||||
}
|
||||
socket.pipe(upstream).pipe(socket);
|
||||
});
|
||||
await new Promise((resolve, reject) => { proxy.once('error', reject); proxy.listen(0, '127.0.0.1', resolve); });
|
||||
const address = proxy.address();
|
||||
assert.ok(address && typeof address !== 'string');
|
||||
process.env.PAPERCLIP_RUNNER_PUBLIC_URL = `wss://127.0.0.1:${address.port}`;
|
||||
process.env.PAPERCLIP_RUNNER_CA_BUNDLE_PATH = ca;
|
||||
process.env.PAPERCLIP_RUNNER_REMOTE_BINARY_PATH = runner;
|
||||
process.env.PAPERCLIP_RUNNER_REMOTE_CODEX_PATH = wrapper;
|
||||
const hash = async file => createHash('sha256').update(await readFile(file)).digest('hex');
|
||||
const provenance = { transport: 'sandbox-remote', provisioning: 'First-party fake-plugin; separate OS processes on this same physical host',
|
||||
revision, buildRevision: build.commit, runnerPublicUrl: process.env.PAPERCLIP_RUNNER_PUBLIC_URL,
|
||||
tls: { peerVerification: true, caSha256: await hash(ca), serverCertificateSha256: await hash(cert), subjectAlternativeNames: ['127.0.0.1', 'localhost'] },
|
||||
runner: { path: runner, sha256: await hash(runner), metadata: runnerMetadata },
|
||||
provider: { wrapper, wrapperSha256: await hash(wrapper), node: process.execPath, nodeVersion: process.version, nodeSha256: await hash(process.execPath),
|
||||
fixture, fixtureSha256: await hash(fixture) } };
|
||||
await writeFile(path.join(process.env.PAPERCLIP_HOME, 'ingestion-remote-provenance.json'), JSON.stringify(provenance, null, 2) + '\n');
|
||||
const { startServer } = await import(pathToFileURL(path.join(appRoot, 'server/dist/index.js')).href);
|
||||
try { await startServer(); }
|
||||
catch (error) { for (const socket of sockets) socket.destroy(); proxy.close(); throw error; }
|
||||
@@ -1,7 +1,9 @@
|
||||
import { createServer, type ServerResponse } from "node:http";
|
||||
import { randomUUID, createHash } from "node:crypto";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { createRequire } from "node:module";
|
||||
import { lstat, readFile, readdir, realpath } from "node:fs/promises";
|
||||
import { hostname } from "node:os";
|
||||
import path from "node:path";
|
||||
import { expect, test } from "@playwright/test";
|
||||
import { json } from "./agent-chat.shared";
|
||||
@@ -11,11 +13,20 @@ type Emission = { nonce: string; markerCode: string; index: number; intendedMs:
|
||||
type Visible = SeenMarker & Emission;
|
||||
type ClockStream = { nonce: string; markerCode: string; opened: boolean; hello: boolean; ended: boolean; aborted: boolean;
|
||||
partialBytes: number; records: Map<number, Emission>; error?: string;
|
||||
expectedLauncher: "host-local" | "staged-remote"; launcher?: string; providerPid?: number;
|
||||
providerReadyAtMs?: number; clockHelloAtMs?: number; readinessObservedAtMs?: number };
|
||||
type ClockCase = { events: number; intervalMs: number; epochMs?: number; streams: Map<string, ClockStream>; errors: string[] };
|
||||
type Case = { name: string; runs: number; events: number; intervalMs: number; prelude: number };
|
||||
type SetupActivation = { index: number; issueId: string; token: string; waveIndex: number | null; startedAtMs: number;
|
||||
requestSettledAtMs?: number; providerReadyAtMs?: number; clockHelloAtMs?: number; readinessObservedAtMs?: number;
|
||||
settledAtMs?: number; status: "pending" | "fulfilled" | "rejected"; failure?: string };
|
||||
type SetupWave = { index: number; indices: number[]; startedAtMs: number; settledAtMs?: number;
|
||||
statuses?: Array<"fulfilled" | "rejected"> };
|
||||
const smoke = process.env.PAPERCLIP_INGESTION_SMOKE === "1";
|
||||
const cases: Case[] = smoke ? [{ name: "qualification", runs: 1, events: 16, intervalMs: 25, prelude: 0 }] : [
|
||||
const transport = process.env.PAPERCLIP_INGESTION_TRANSPORT ?? "sandbox-remote-with-local-stop";
|
||||
const providerReadinessMs = 300_000;
|
||||
const setupActivationConcurrency = 2;
|
||||
const cases: Case[] = smoke ? [{ name: "qualification", runs: 3, events: 16, intervalMs: 25, prelude: 0 }] : [
|
||||
{ name: "paced", runs: 1, events: 128, intervalMs: 25, prelude: 0 },
|
||||
{ name: "burst16", runs: 1, events: 16, intervalMs: 0, prelude: 0 },
|
||||
{ name: "burst128", runs: 10, events: 128, intervalMs: 0, prelude: 0 },
|
||||
@@ -28,6 +39,116 @@ const stats = (values: number[]) => {
|
||||
...(values.length >= 100 ? { p95Ms: sorted[Math.ceil(sorted.length * 0.95) - 1] } : {}) };
|
||||
};
|
||||
|
||||
async function runtimeAuthority(input: {
|
||||
appRoot: string; companyId: string; environmentId: string; issueIds: string[]; transport: "local" | "sandbox-remote";
|
||||
afterStop?: boolean;
|
||||
}) {
|
||||
const configuration = process.env.PAPERCLIP_E2E_SERVER_CONFIG;
|
||||
if (!configuration) throw new Error("Missing owned server configuration");
|
||||
const config = JSON.parse(await readFile(configuration, "utf8"));
|
||||
expect(config.database.mode).toBe("embedded-postgres");
|
||||
const dataDirectory = await realpath(config.database.embeddedPostgresDataDir);
|
||||
const ownedHome = await realpath(path.resolve(path.dirname(configuration), "../.."));
|
||||
expect(dataDirectory.startsWith(`${ownedHome}${path.sep}`)).toBe(true);
|
||||
const owner = (await readFile(path.join(dataDirectory, "postmaster.pid"), "utf8")).split(/\r?\n/);
|
||||
expect(await realpath(owner[1])).toBe(dataDirectory);
|
||||
const port = Number(owner[3]);
|
||||
expect(Number.isSafeInteger(port) && port > 0 && port <= 65535).toBe(true);
|
||||
const require = createRequire(path.join(input.appRoot, "packages/db/package.json"));
|
||||
const postgres = require("postgres");
|
||||
const sql = postgres({ host: "127.0.0.1", port, database: "paperclip", username: "paperclip", password: "paperclip",
|
||||
max: 1, connect_timeout: 10, connection: { application_name: "native_ingestion_remote_authority_proof" } });
|
||||
let proof;
|
||||
try {
|
||||
proof = await sql.begin("isolation level repeatable read read only", async (tx: any) => {
|
||||
const settings = await tx`select current_setting('transaction_read_only') as read_only,
|
||||
current_setting('transaction_isolation') as isolation`;
|
||||
expect(settings[0].read_only).toBe("on");
|
||||
expect(settings[0].isolation).toBe("repeatable read");
|
||||
const rows = await tx`select r.id as run_id, r.company_id, r.agent_id, r.native_issue_id as issue_id, r.runtime_mode,
|
||||
r.native_session_id, r.runner_instance_id, r.status, r.process_pid, r.process_group_id, r.process_started_at,
|
||||
r.runner_profile_json->'nativeExecutionInput'->'binding'->>'executionWorkspaceId' as admitted_execution_workspace_id,
|
||||
r.result_json->'nativeCancellation'->>'dispatchState' as cancellation_dispatch_state,
|
||||
i.seq as process_identity_seq, i.payload as process_identity, i.source_event_id as process_identity_source_event_id,
|
||||
s.seq as process_stopped_seq, s.payload as process_stopped, s.source_event_id as process_stopped_source_event_id,
|
||||
l.id as lease_id, l.environment_id, l.provider, l.provider_lease_id,
|
||||
c.part as checkpoint_part, octet_length(c.material::text) as checkpoint_header_bytes
|
||||
from heartbeat_runs r
|
||||
join environment_leases l on l.heartbeat_run_id = r.id and l.company_id = r.company_id
|
||||
left join native_runner_checkpoints c on c.run_id = r.id and c.company_id = r.company_id
|
||||
and c.issue_id = r.native_issue_id and c.part = 'header'
|
||||
left join lateral (select seq, payload, source_event_id from heartbeat_run_events e where e.run_id = r.id
|
||||
and e.company_id = r.company_id and e.event_type = 'native.process_identity_recorded'
|
||||
and e.source_event_id is null order by seq desc limit 1) i on true
|
||||
left join lateral (select seq, payload, source_event_id from heartbeat_run_events e where e.run_id = r.id
|
||||
and e.company_id = r.company_id and e.event_type = 'native.local_process_stopped'
|
||||
and e.source_event_id is null order by seq desc limit 1) s on true
|
||||
where r.company_id = ${input.companyId} and r.native_issue_id = any(${input.issueIds}::uuid[])`;
|
||||
expect(rows).toHaveLength(input.issueIds.length);
|
||||
expect(new Set(rows.map((row: any) => row.issue_id)).size).toBe(input.issueIds.length);
|
||||
for (const row of rows) {
|
||||
expect(row.company_id).toBe(input.companyId);
|
||||
expect(row.runtime_mode).toBe("native");
|
||||
expect(row.environment_id).toBe(input.environmentId);
|
||||
if (input.transport === "sandbox-remote") {
|
||||
expect(row.provider).toBe("fake-plugin");
|
||||
expect(row.provider_lease_id).toMatch(/^fake-plugin:\/\//);
|
||||
expect(row.checkpoint_part).toBe("header");
|
||||
expect(Number(row.checkpoint_header_bytes)).toBeGreaterThan(0);
|
||||
} else {
|
||||
expect(row.provider).toBe("local"); expect(row.provider_lease_id).toBeNull();
|
||||
expect(row.checkpoint_part, "Local raw transport keeps its existing host-file journal authority").toBeNull();
|
||||
expect(row.process_identity?.targetKind).toBe("local"); expect(row.process_identity?.hostname).toBe(hostname());
|
||||
expect(row.process_identity_source_event_id).toBeNull();
|
||||
expect(BigInt(String(row.process_identity_seq))).toBeGreaterThan(0n);
|
||||
expect(row.admitted_execution_workspace_id).toMatch(/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/i);
|
||||
if (row.process_started_at) expect(row.process_identity.processStartedAt).toBe(new Date(row.process_started_at).toISOString());
|
||||
}
|
||||
}
|
||||
return { kind: input.transport === "local" ? "host-file-journal" : "database-controller-checkpoint",
|
||||
checkedBeforeMeasuredEpoch: !input.afterStop, checkedAtMs: Date.now(), transaction: "READ ONLY REPEATABLE READ",
|
||||
observedReadOnly: settings[0].read_only, observedIsolation: settings[0].isolation, rows };
|
||||
});
|
||||
} finally { await sql.end({ timeout: 5 }); }
|
||||
const connectionClosedAtMs = Date.now();
|
||||
let localJournal;
|
||||
if (input.transport === "local" && !input.afterStop) {
|
||||
expect(input.issueIds).toHaveLength(1);
|
||||
const base = await realpath(path.join(path.dirname(configuration), "runtime/paperclip-runner/durable-sessions"));
|
||||
expect(base.startsWith(`${ownedHome}${path.sep}`)).toBe(true);
|
||||
const directories = (await readdir(base, { withFileTypes: true })).filter(entry => entry.isDirectory() && /^[a-f0-9]{64}$/.test(entry.name));
|
||||
expect(directories.length, "Bound the owned fixture journal metadata search").toBeLessThanOrEqual(256);
|
||||
for (const directory of directories) {
|
||||
const file = path.join(base, directory.name, "control-plane/control-plane-state.json");
|
||||
const metadata = await lstat(file).catch((error: NodeJS.ErrnoException) => {
|
||||
if (error.code === "ENOENT") return null; throw error;
|
||||
});
|
||||
if (!metadata) continue;
|
||||
expect(metadata.isFile() && !metadata.isSymbolicLink() && metadata.size <= 16 * 1024 * 1024).toBe(true);
|
||||
expect((await realpath(file)).startsWith(`${base}${path.sep}`)).toBe(true);
|
||||
const bytes = await readFile(file), state = JSON.parse(bytes.toString("utf8"));
|
||||
if (state.identity?.runId !== proof!.rows[0].run_id) continue;
|
||||
expect(localJournal, "Exactly one owned local journal must match the actual run").toBeUndefined();
|
||||
expect(state.schema).toBe("paperclip.runner.durable.control-plane-state.v1");
|
||||
expect(state.checkpointEventPolicy).not.toBe("control-v1");
|
||||
expect(state.identity.runnerInstanceId).toBe(proof!.rows[0].runner_instance_id);
|
||||
expect(state.identity.normalizedSessionId).toBe(proof!.rows[0].native_session_id);
|
||||
expect(state.identity.environmentLeaseId).toBe(proof!.rows[0].admitted_execution_workspace_id);
|
||||
localJournal = { file, sha256: createHash("sha256").update(bytes).digest("hex"), bytes: bytes.length,
|
||||
schema: state.schema, identity: Object.fromEntries(["runnerInstanceId", "environmentLeaseId", "runId", "normalizedSessionId", "turnId", "itemId"]
|
||||
.map(key => [key, state.identity[key]])), checkpointEventPolicy: state.checkpointEventPolicy, checkedAtMs: Date.now() };
|
||||
}
|
||||
expect(localJournal, "Missing actual owned local raw journal").toBeTruthy();
|
||||
}
|
||||
return { ...proof, connectionClosedAtMs, localJournal };
|
||||
}
|
||||
|
||||
const processAbsent = (pid: number) => {
|
||||
expect(Number.isSafeInteger(pid) && Math.abs(pid) > 1).toBe(true);
|
||||
try { process.kill(pid, 0); return false; }
|
||||
catch (error) { if ((error as NodeJS.ErrnoException).code === "ESRCH") return true; throw error; }
|
||||
};
|
||||
|
||||
test("native ingestion reaches the shipped UI promptly and survives reload", async ({ page, request }, info) => {
|
||||
const pending = new Map<string, Map<string, ServerResponse>>();
|
||||
const clocks = new Map<string, ClockCase>();
|
||||
@@ -62,7 +183,10 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
const value = JSON.parse(text);
|
||||
if (value.nonce !== participant || value.markerCode !== stream.markerCode) throw new Error("Clock identity mismatch");
|
||||
if (!stream.hello) {
|
||||
if (value.type !== "hello" || value.events !== state.events || value.intervalMs !== state.intervalMs) throw new Error("Clock hello mismatch");
|
||||
if (value.type !== "hello" || value.events !== state.events || value.intervalMs !== state.intervalMs
|
||||
|| value.launcher !== stream.expectedLauncher || !Number.isSafeInteger(value.providerPid) || value.providerPid <= 1
|
||||
|| Object.keys(value).sort().join(",") !== "events,intervalMs,launcher,markerCode,nonce,providerPid,type") throw new Error("Clock hello mismatch");
|
||||
stream.launcher = value.launcher; stream.providerPid = value.providerPid;
|
||||
stream.hello = true; stream.clockHelloAtMs = Date.now(); return;
|
||||
}
|
||||
if (value.type !== "emission" || !Number.isSafeInteger(value.index) || value.index < 0 || value.index >= state.events
|
||||
@@ -147,8 +271,13 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
const browserRecording = { trace: captureMode(info.project.use.trace), video: captureMode(info.project.use.video),
|
||||
screencast: captureMode(info.project.use.trace) !== "off" || captureMode(info.project.use.video) !== "off" };
|
||||
const report: Record<string, unknown> = {
|
||||
protocolVersion: "v2-compact-markers-original-clock-stream",
|
||||
admissionProtocol: "sequential-provider-and-clock-readiness-before-common-epoch",
|
||||
protocolVersion: "v4-remote-streaming-original-local-stop-clock-stream", transport,
|
||||
transportPolicy: { streaming: "sandbox-remote", operatorStop: "local",
|
||||
realCloudStop: "Separate required qualification; fake-plugin void receipts grant no remote Stop authority" },
|
||||
resourceBudgets: { providerReadinessMs, fullInvocationMs: info.timeout, streamingMs: 120_000,
|
||||
terminalDrainageMs: 180_000, stopAcknowledgementMs: 30_000, remoteLeaseCleanupMs: 30_000 },
|
||||
admissionProtocol: "two-at-a-time-provider-and-clock-readiness-before-common-epoch",
|
||||
setupActivationConcurrency,
|
||||
setupReadinessPollingMs: 50,
|
||||
dependency: "Deterministic Codex app-server fixture; real application/runtime/port/PostgreSQL/live WebSocket/shipped UI",
|
||||
variant: process.env.PAPERCLIP_INGESTION_VARIANT ?? "unlabelled", qualificationOnly: smoke,
|
||||
@@ -177,7 +306,8 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
}
|
||||
report.fixtureSha256 = createHash("sha256").update(await readFile(path.resolve(import.meta.dirname, "fixtures/in-feed-codex.mjs"))).digest("hex");
|
||||
const protocolFiles = ["native-event-ingestion.journey.ts", "playwright-native-ingestion.config.ts", "fixtures/in-feed-codex.mjs",
|
||||
"fixtures/native-ingestion-bin/codex", "fixtures/composer-stop-codex-loader.mjs", "fixtures/native-ingestion-bootstrap.mjs"];
|
||||
"fixtures/native-ingestion-bin/codex", "fixtures/composer-stop-codex-loader.mjs", "fixtures/native-ingestion-bootstrap.mjs",
|
||||
"fixtures/native-ingestion-remote-server.mjs", "fixtures/native-ingestion-protocol-qualification.mjs"];
|
||||
const protocolSources = await Promise.all(protocolFiles.map(async file => ({ file, bytes: await readFile(path.resolve(import.meta.dirname, file)) })));
|
||||
report.measurementSha256 = createHash("sha256").update(Buffer.concat(protocolSources.map(source => source.bytes))).digest("hex");
|
||||
const harnessRoot = path.resolve(import.meta.dirname, "../..");
|
||||
@@ -185,7 +315,8 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
sourceRevision: execFileSync("git", ["rev-parse", "HEAD"], { cwd: harnessRoot, encoding: "utf8" }).trim(),
|
||||
sourceStatus: execFileSync("git", ["status", "--porcelain", "--", "tests/e2e"], { cwd: harnessRoot, encoding: "utf8" }).trim(),
|
||||
sourceHashes: Object.fromEntries(protocolSources.map(source => [source.file, createHash("sha256").update(source.bytes).digest("hex")])),
|
||||
driverDependencySource: process.env.PAPERCLIP_INGESTION_DRIVER_DEPENDENCIES ?? "Checkout tooling dependencies",
|
||||
driverDependencySource: process.env.PAPERCLIP_INGESTION_DRIVER_DEPENDENCIES
|
||||
?? "Independent locked H4 driver install; every measured app owns separate dependencies and compiled output",
|
||||
driverLockSha256: createHash("sha256").update(await readFile(path.join(harnessRoot, "pnpm-lock.yaml"))).digest("hex"),
|
||||
scope: "Separate browser harness and driver-only tooling; every measured app uses its own frozen clean source and matching compiled stamp",
|
||||
};
|
||||
@@ -193,11 +324,37 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
await json(await request.patch("/api/instance/settings/experimental", { data: { enableNativeRunner: true, enableWorktreeRunExecution: true } }));
|
||||
await json(await request.post(`/api/companies/${company.id}/user-secret-definitions`, { data: { key: "OPENAI_API_KEY", name: "Fixture credential" } }));
|
||||
await json(await request.post(`/api/companies/${company.id}/me/user-secrets`, { data: { definitionKey: "OPENAI_API_KEY", value: "not-a-real-model-key" } }));
|
||||
const agents = [];
|
||||
let remoteEnvironment: { id: string };
|
||||
{
|
||||
const pluginPath = path.join(appRoot, "packages/plugins/paperclip-plugin-fake-sandbox");
|
||||
const plugin = await json(await request.post("/api/plugins/install", { timeout: 120_000,
|
||||
data: { packageName: pluginPath, isLocalPath: true } }));
|
||||
expect(plugin.pluginKey).toBe("paperclip.fake-sandbox-provider");
|
||||
const serverConfig = process.env.PAPERCLIP_E2E_SERVER_CONFIG;
|
||||
if (!serverConfig) throw new Error("Missing owned server configuration for remote fixture provenance");
|
||||
const remoteProvenance = JSON.parse(await readFile(path.resolve(path.dirname(serverConfig), "../..", "ingestion-remote-provenance.json"), "utf8"));
|
||||
remoteEnvironment = await json(await request.post(`/api/companies/${company.id}/environments`, { data: {
|
||||
name: "Native ingestion remote fixture", driver: "sandbox",
|
||||
config: { provider: "fake-plugin", image: "fake:latest", reuseLease: false, runnerLifecycleMode: "per_turn" },
|
||||
envVars: { PATH: `${path.dirname(remoteProvenance.provider.node)}:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin` },
|
||||
} }));
|
||||
report.remote = { ...remoteProvenance, plugin: { id: plugin.id, pluginKey: plugin.pluginKey, path: pluginPath },
|
||||
environmentId: remoteEnvironment!.id,
|
||||
selection: "Normal plugin install + sandbox environment + agent defaultEnvironmentId; no runtime resolver override" };
|
||||
(report.remote as any).plugin.assetHashes = Object.fromEntries(await Promise.all(
|
||||
["worker.js", "plugin.js", "manifest.js", "lease-cleanup.js"].map(async file => [file,
|
||||
createHash("sha256").update(await readFile(path.join(pluginPath, "dist", file))).digest("hex")])));
|
||||
}
|
||||
const localEnvironments = await json(await request.get(`/api/companies/${company.id}/environments?driver=local`));
|
||||
expect(localEnvironments).toHaveLength(1);
|
||||
const localEnvironment = localEnvironments[0];
|
||||
expect(localEnvironment.driver).toBe("local");
|
||||
const agents: Array<{ id: string; name: string }> = [];
|
||||
const agentCount = Math.ceil(Math.max(...cases.map(item => item.runs)) / 50);
|
||||
for (let index = 0; index < agentCount; index++) {
|
||||
const agent = await json(await request.post(`/api/companies/${company.id}/agents`, { timeout: 120_000, data: {
|
||||
name: `Ingestion ${index + 1}`, role: "engineer", adapterType: "paperclip_runner",
|
||||
defaultEnvironmentId: remoteEnvironment.id,
|
||||
adapterConfig: { provider: "codex", model: "gpt-5.1-codex-mini", env: {
|
||||
OPENAI_API_KEY: { type: "user_secret_ref", key: "OPENAI_API_KEY", version: "latest", required: true },
|
||||
} }, runtimeConfig: { heartbeat: { enabled: false, wakeOnDemand: true, maxConcurrentRuns: 50 } },
|
||||
@@ -206,30 +363,53 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
{ timeout: 60_000 }).toBe("ready");
|
||||
agents.push(agent);
|
||||
}
|
||||
(report.remote as any).agentIds = agents.map(agent => agent.id);
|
||||
const controlAgent = await json(await request.post(`/api/companies/${company.id}/agents`, { timeout: 120_000, data: {
|
||||
name: "Ingestion local Stop control", role: "engineer", adapterType: "paperclip_runner",
|
||||
defaultEnvironmentId: localEnvironment.id,
|
||||
adapterConfig: { provider: "codex", model: "gpt-5.1-codex-mini", env: {
|
||||
OPENAI_API_KEY: { type: "user_secret_ref", key: "OPENAI_API_KEY", version: "latest", required: true },
|
||||
} }, runtimeConfig: { heartbeat: { enabled: false, wakeOnDemand: true, maxConcurrentRuns: 50 } },
|
||||
} }));
|
||||
await expect.poll(async () => (await json(await request.get(`/api/agents/${controlAgent.id}`))).lifecycleState,
|
||||
{ timeout: 60_000 }).toBe("ready");
|
||||
report.local = { transport: "local", environmentId: localEnvironment.id, controlAgentId: controlAgent.id,
|
||||
selection: "Normal local environment + separate control agent defaultEnvironmentId; existing test-only Codex leaf command fixture",
|
||||
commandSha256: (report.harnessProvenance as any).sourceHashes["fixtures/native-ingestion-bin/codex"],
|
||||
loaderSha256: (report.harnessProvenance as any).sourceHashes["fixtures/composer-stop-codex-loader.mjs"] };
|
||||
|
||||
async function perform(workload: Case, stop = false) {
|
||||
const caseTransport = stop ? "local" : "sandbox-remote";
|
||||
const caseEnvironment = stop ? localEnvironment : remoteEnvironment;
|
||||
const caseAgents = stop ? [controlAgent] : agents;
|
||||
const key = randomUUID().replaceAll("-", ""), nonce = randomUUID().replaceAll("-", "");
|
||||
const markerCode = nextMarkerCode();
|
||||
pending.set(key, new Map());
|
||||
const state: ClockCase = { events: workload.events, intervalMs: workload.intervalMs, streams: new Map(), errors: [] };
|
||||
clocks.set(key, state);
|
||||
const register = (token: string, code: string) => state.streams.set(token, { nonce: token, markerCode: code,
|
||||
opened: false, hello: false, ended: false, aborted: false, partialBytes: 0, records: new Map() });
|
||||
opened: false, hello: false, ended: false, aborted: false, partialBytes: 0, records: new Map(),
|
||||
expectedLauncher: stop ? "host-local" : "staged-remote" });
|
||||
register(nonce, markerCode);
|
||||
const compactOutput = Array.from({ length: workload.events }, (_, index) => `[${markerCode}${index.toString(36)}] `).join("");
|
||||
const preludeOutput = workload.prelude ? `${".".repeat(Math.max(0, workload.prelude - 1))} PRELUDE_${nonce}_READY ` : "";
|
||||
expect([...compactOutput].length, "Full measured completion must fit the unchanged Rust text bound").toBeLessThan(4000);
|
||||
expect([...preludeOutput + compactOutput].length, "Prelude and measured text must remain below that bound").toBeLessThan(4000);
|
||||
const active: Record<string, unknown> = { ...workload, stop, nonce, markerCode, phase: "setup", key };
|
||||
const active: Record<string, unknown> = { ...workload, stop, transport: caseTransport,
|
||||
environmentId: caseEnvironment.id, nonce, markerCode, phase: "setup", key };
|
||||
report.activeCase = active;
|
||||
const prompt = (token: string, code: string, prelude: number) => `Measure native streaming for ${token.slice(0, 8)}.\n\nINGESTION_PERF nonce=${token} events=${workload.events} intervalMs=${workload.intervalMs} prelude=${prelude} markerCode=${code} gate=${base}${key}`;
|
||||
const before = new Set((await json(await request.get(`/api/companies/${company.id}/issues`))).map((issue: { id: string }) => issue.id));
|
||||
await page.goto(`/${company.issuePrefix}/agents/${agents[0].id}/overview`);
|
||||
await page.goto(`/${company.issuePrefix}/agents/${caseAgents[0].id}/overview`);
|
||||
await page.getByRole("link", { name: "Tasks", exact: true }).click();
|
||||
await page.getByRole("button", { name: "New Task", exact: true }).last().click();
|
||||
const dialog = page.getByRole("dialog");
|
||||
await dialog.getByRole("textbox", { name: "editable markdown" }).fill(prompt(nonce, markerCode, workload.prelude));
|
||||
await expect(dialog.getByRole("button", { name: "Select assignee" })).toContainText(agents[0].name);
|
||||
await dialog.getByRole("button", { name: "Select assignee", exact: true }).click();
|
||||
await page.getByRole("listbox", { name: "Assignees", exact: true }).getByRole("option")
|
||||
.filter({ has: page.getByText(caseAgents[0].name, { exact: true }) }).click();
|
||||
await expect(dialog.getByRole("button", { name: "Select assignee" })).toContainText(caseAgents[0].name);
|
||||
const foregroundActivationStartedAtMs = Date.now();
|
||||
await dialog.getByRole("button", { name: "Create task", exact: true }).click();
|
||||
let issue: any;
|
||||
await expect.poll(async () => {
|
||||
@@ -238,28 +418,54 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
}).toBe(true);
|
||||
const issueIds = [issue.id];
|
||||
active.issueIds = issueIds;
|
||||
const activations: SetupActivation[] = [{ index: 0, issueId: issue.id, token: nonce, waveIndex: null,
|
||||
startedAtMs: foregroundActivationStartedAtMs, requestSettledAtMs: Date.now(), status: "pending" }];
|
||||
const activationWaves: SetupWave[] = [];
|
||||
active.activations = activations; active.activationWaves = activationWaves;
|
||||
active.setupActivationConcurrency = setupActivationConcurrency;
|
||||
const issuedTokens = new Map<string, string>([[issue.id, nonce]]);
|
||||
for (let index = 1; index < workload.runs; index++) {
|
||||
const backgroundToken = randomUUID().replaceAll("-", ""), backgroundCode = nextMarkerCode();
|
||||
register(backgroundToken, backgroundCode);
|
||||
const background = await json(await request.post(`/api/companies/${company.id}/issues`, { data: {
|
||||
title: `Ingestion background ${key} ${index}`, description: prompt(backgroundToken, backgroundCode, 0),
|
||||
status: "backlog", assigneeAgentId: agents[Math.floor(index / 50)].id,
|
||||
status: "backlog", assigneeAgentId: caseAgents[Math.floor(index / 50)].id,
|
||||
} }));
|
||||
issueIds.push(background.id);
|
||||
issuedTokens.set(background.id, backgroundToken);
|
||||
}
|
||||
const readinessDeadlineMs = Date.now() + 180_000;
|
||||
const setupReadinessStartedAtMs = Date.now();
|
||||
const readinessDeadlineMs = setupReadinessStartedAtMs + providerReadinessMs;
|
||||
active.setupReadinessStartedAtMs = setupReadinessStartedAtMs; active.readinessDeadlineMs = readinessDeadlineMs;
|
||||
active.phase = "activation-and-provider-readiness";
|
||||
for (let index = 1; index < issueIds.length; index++) {
|
||||
const id = issueIds[index], token = issuedTokens.get(id)!;
|
||||
active.activation = { index, issueId: id, token, startedAtMs: Date.now(), readinessDeadlineMs };
|
||||
await json(await request.patch(`/api/issues/${id}`, { data: { status: "todo" } }));
|
||||
// PATCH schedules wakeup asynchronously. Wait for this actual provider,
|
||||
// while the common emission gate remains closed, before scheduling more.
|
||||
await expect.poll(() => pending.get(key)!.has(token) && state.streams.get(token)!.hello,
|
||||
{ timeout: Math.max(1, readinessDeadlineMs - Date.now()), intervals: [50] }).toBe(true);
|
||||
state.streams.get(token)!.readinessObservedAtMs = Date.now();
|
||||
for (let firstIndex = 1; firstIndex < issueIds.length; firstIndex += setupActivationConcurrency) {
|
||||
expect(Date.now(), "Do not activate a later wave after the shared readiness deadline").toBeLessThanOrEqual(readinessDeadlineMs);
|
||||
const indices = Array.from({ length: Math.min(setupActivationConcurrency, issueIds.length - firstIndex) }, (_, offset) => firstIndex + offset);
|
||||
const wave: SetupWave = { index: activationWaves.length, indices, startedAtMs: Date.now() };
|
||||
activationWaves.push(wave); active.activationWave = wave;
|
||||
const settled = await Promise.allSettled(indices.map(async index => {
|
||||
const id = issueIds[index], token = issuedTokens.get(id)!;
|
||||
const activation: SetupActivation = { index, issueId: id, token, waveIndex: wave.index,
|
||||
startedAtMs: Date.now(), status: "pending" };
|
||||
activations.push(activation);
|
||||
try {
|
||||
await json(await request.patch(`/api/issues/${id}`, { data: { status: "todo" } }));
|
||||
activation.requestSettledAtMs = Date.now();
|
||||
// Keep the gate closed until both actual providers in this wave have
|
||||
// opened their GET and clock hello; never overlap a later wave.
|
||||
await expect.poll(() => pending.get(key)!.has(token) && state.streams.get(token)!.hello,
|
||||
{ timeout: Math.max(1, readinessDeadlineMs - Date.now()), intervals: [50] }).toBe(true);
|
||||
const stream = state.streams.get(token)!;
|
||||
stream.readinessObservedAtMs = Date.now();
|
||||
Object.assign(activation, { providerReadyAtMs: stream.providerReadyAtMs, clockHelloAtMs: stream.clockHelloAtMs,
|
||||
readinessObservedAtMs: stream.readinessObservedAtMs, status: "fulfilled" });
|
||||
} catch (error) {
|
||||
activation.status = "rejected"; activation.failure = String(error); throw error;
|
||||
} finally { activation.settledAtMs = Date.now(); }
|
||||
}));
|
||||
wave.settledAtMs = Date.now(); wave.statuses = settled.map(result => result.status);
|
||||
const failure = settled.find(result => result.status === "rejected");
|
||||
if (failure?.status === "rejected") throw failure.reason;
|
||||
}
|
||||
const route = `/${company.issuePrefix}/issues/${issue.identifier}`;
|
||||
await page.goto(route);
|
||||
@@ -268,6 +474,11 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
clocks: [...state.streams.values()].filter(stream => stream.hello).length }),
|
||||
{ timeout: Math.max(1, readinessDeadlineMs - Date.now()), intervals: [50] }).toEqual({ providers: workload.runs, clocks: workload.runs });
|
||||
expect(Date.now(), "All participants must be ready within the shared activation/readiness budget").toBeLessThanOrEqual(readinessDeadlineMs);
|
||||
const allReadyObservedAtMs = Date.now(), foregroundStream = state.streams.get(nonce)!;
|
||||
foregroundStream.readinessObservedAtMs = allReadyObservedAtMs;
|
||||
Object.assign(activations[0], { providerReadyAtMs: foregroundStream.providerReadyAtMs, clockHelloAtMs: foregroundStream.clockHelloAtMs,
|
||||
readinessObservedAtMs: allReadyObservedAtMs, settledAtMs: allReadyObservedAtMs, status: "fulfilled" });
|
||||
active.allReadyObservedAtMs = allReadyObservedAtMs;
|
||||
let preludeProof: Record<string, unknown> | undefined;
|
||||
if (workload.prelude) {
|
||||
await expect(page.getByText(`PRELUDE_${nonce}_READY`, { exact: false }).first()).toBeVisible({ timeout: 120_000 });
|
||||
@@ -293,9 +504,28 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
active.preludeProof = preludeProof;
|
||||
expect(count).toBe(workload.prelude); expect(identities.size).toBe(count);
|
||||
}
|
||||
const controllerAuthority = await runtimeAuthority({
|
||||
appRoot, companyId: company.id, environmentId: caseEnvironment.id, issueIds, transport: caseTransport,
|
||||
});
|
||||
active.controllerAuthority = controllerAuthority;
|
||||
const providerClocks = controllerAuthority.rows.map((row: any) => {
|
||||
const token = issuedTokens.get(row.issue_id)!, stream = state.streams.get(token)!;
|
||||
return { runId: row.run_id, nonce: token, launcher: stream.launcher, providerPid: stream.providerPid };
|
||||
});
|
||||
let localProcesses;
|
||||
if (stop) {
|
||||
const row = controllerAuthority.rows[0], identity = row.process_identity;
|
||||
expect(identity.processPid).toBe(row.process_pid); expect(identity.processGroupId).toBe(row.process_group_id);
|
||||
const providerPid = providerClocks[0].providerPid!;
|
||||
localProcesses = { runnerPid: row.process_pid, processGroupId: row.process_group_id, providerPid,
|
||||
hostname: hostname(), checkedAtMs: Date.now(), runnerAlive: !processAbsent(row.process_pid),
|
||||
groupAlive: !processAbsent(-row.process_group_id), providerAlive: !processAbsent(providerPid) };
|
||||
expect(localProcesses.runnerAlive && localProcesses.groupAlive && localProcesses.providerAlive).toBe(true);
|
||||
}
|
||||
const participants = [...pending.get(key)!.keys()];
|
||||
expect(participants, "Every actual provider GET must still be open after the authority snapshot").toHaveLength(workload.runs);
|
||||
const epochMs = Date.now() + 1000;
|
||||
state.epochMs = epochMs; active.epochMs = epochMs; active.phase = "streaming";
|
||||
const participants = [...pending.get(key)!.keys()];
|
||||
active.participants = participants;
|
||||
for (const response of pending.get(key)!.values()) response.writeHead(200, { "Content-Type": "application/json" }).end(JSON.stringify({ epochMs }));
|
||||
const observed = () => page.evaluate(code => (window as unknown as { nativeIngestionVisible: SeenMarker[] }).nativeIngestionVisible.filter(item => item.markerCode === code), markerCode);
|
||||
@@ -324,6 +554,52 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
(await json(await request.get(`/api/heartbeat-runs/${runs[0].id}`))).resultJson?.nativeCancellation?.dispatchState,
|
||||
{ timeout: 30_000 }).toBe("acknowledged");
|
||||
const acknowledgedAt = stop ? Date.now() : undefined;
|
||||
let remoteLeases: unknown[] | undefined;
|
||||
let remoteLeasesCheckedAtMs: number | undefined;
|
||||
if (!stop) {
|
||||
let leases: any[] = [];
|
||||
await expect.poll(async () => {
|
||||
leases = await json(await request.get(`/api/environments/${remoteEnvironment.id}/leases`));
|
||||
return runs.every(run => leases.some(lease => lease.heartbeatRunId === run.id && lease.releasedAt
|
||||
&& ["released", "expired", "failed"].includes(lease.status) && lease.cleanupStatus === "success"));
|
||||
}, { timeout: 30_000 }).toBe(true);
|
||||
remoteLeasesCheckedAtMs = Date.now();
|
||||
remoteLeases = runs.map(run => {
|
||||
const lease = leases.find((candidate: any) => candidate.heartbeatRunId === run.id);
|
||||
expect(lease, `Missing normal sandbox lease for ${run.id}`).toBeTruthy();
|
||||
expect(lease.environmentId).toBe(remoteEnvironment.id);
|
||||
expect(lease.providerLeaseId).toMatch(/^fake-plugin:\/\//);
|
||||
return { id: lease.id, runId: run.id, companyId: lease.companyId, environmentId: lease.environmentId, provider: lease.provider,
|
||||
providerLeaseId: lease.providerLeaseId, status: lease.status, releasedAt: lease.releasedAt, cleanupStatus: lease.cleanupStatus,
|
||||
terminationReceipt: lease.metadata?.remoteExecutionTermination };
|
||||
});
|
||||
}
|
||||
let localStopProof;
|
||||
if (stop) {
|
||||
let leases: any[] = [];
|
||||
await expect.poll(async () => {
|
||||
leases = await json(await request.get(`/api/environments/${localEnvironment.id}/leases`));
|
||||
return leases.some(lease => lease.heartbeatRunId === runs[0].id && lease.provider === "local"
|
||||
&& lease.releasedAt && lease.cleanupStatus !== "failed");
|
||||
}, { timeout: 30_000 }).toBe(true);
|
||||
const processIdentity = localProcesses!;
|
||||
await expect.poll(() => processAbsent(processIdentity.runnerPid) && processAbsent(-processIdentity.processGroupId)
|
||||
&& processAbsent(processIdentity.providerPid), { timeout: 30_000 }).toBe(true);
|
||||
const database = await runtimeAuthority({ appRoot, companyId: company.id, environmentId: localEnvironment.id,
|
||||
issueIds, transport: "local", afterStop: true });
|
||||
const row = database.rows[0];
|
||||
expect(row.status).toBe("cancelled"); expect(row.cancellation_dispatch_state).toBe("acknowledged");
|
||||
if (!row.process_pid && !row.process_group_id) {
|
||||
expect(BigInt(String(row.process_stopped_seq))).toBeGreaterThan(BigInt(String(row.process_identity_seq)));
|
||||
expect(row.process_stopped?.processPid).toBe(processIdentity.runnerPid);
|
||||
expect(row.process_stopped?.processGroupId).toBe(processIdentity.processGroupId);
|
||||
}
|
||||
localStopProof = { checkedAtMs: Date.now(), database, processes: { ...processIdentity,
|
||||
runnerAbsent: processAbsent(processIdentity.runnerPid), groupAbsent: processAbsent(-processIdentity.processGroupId),
|
||||
providerAbsent: processAbsent(processIdentity.providerPid) },
|
||||
lease: Object.fromEntries(["id", "companyId", "environmentId", "heartbeatRunId", "provider", "providerLeaseId", "status", "releasedAt", "cleanupStatus"]
|
||||
.map(key => [key, leases.find(lease => lease.heartbeatRunId === runs[0].id)[key]])) };
|
||||
}
|
||||
active.phase = "clock-reconciliation";
|
||||
await expect.poll(() => [...state.streams.values()].every(stream => !stream.error && (stop
|
||||
? seenMarkers.every(marker => state.streams.get(nonce)!.records.has(marker.index))
|
||||
@@ -341,7 +617,12 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
const { count, medianMs, maxMs } = stats(paintTimes.slice(1).map((at, index) => at - paintTimes[index]));
|
||||
const interChunkGaps = { paintCount: paintTimes.length, count, medianMs, maxMs };
|
||||
const clockEvidence: Array<Record<string, unknown>> = [];
|
||||
const result = { ...workload, stop, nonce, markerCode, epochMs, preludeProof, clockEvidence, interChunkGaps, runIds: runs.map(run => run.id), participants: participants.length, visible, matchedVisibleClocks: visible.length,
|
||||
const result = { ...workload, stop, transport: caseTransport, environmentId: caseEnvironment.id,
|
||||
setupActivationConcurrency, setupReadinessStartedAtMs, readinessDeadlineMs, allReadyObservedAtMs, activations, activationWaves,
|
||||
nonce, markerCode, epochMs, settledAtMs: settledAt, stopRequestedAtMs: stopRequestedAt,
|
||||
stopAcknowledgedAtMs: acknowledgedAt, preludeProof, controllerAuthority, clockEvidence, interChunkGaps,
|
||||
providerClocks, localProcesses, localStopProof,
|
||||
remoteLeases, remoteLeasesCheckedAtMs, runIds: runs.map(run => run.id), participants: participants.length, visible, matchedVisibleClocks: visible.length,
|
||||
firstVisibleAfterEmissionMs: visible[0].visibleMs - visible[0].emittedMs,
|
||||
emissionToVisible: stats(visible.map(item => item.visibleMs - item.emittedMs)),
|
||||
intendedToVisible: stats(visible.map(item => item.visibleMs - item.intendedMs)),
|
||||
@@ -399,6 +680,7 @@ test("native ingestion reaches the shipped UI promptly and survives reload", asy
|
||||
expect(state.errors).toEqual([]);
|
||||
expect([...state.streams.values()].every(stream => !stream.error)).toBe(true);
|
||||
clockEvidence.push(...[...state.streams.values()].map(stream => ({ nonce: stream.nonce, markerCode: stream.markerCode,
|
||||
launcher: stream.launcher, providerPid: stream.providerPid,
|
||||
hello: stream.hello, ended: stream.ended, aborted: stream.aborted, error: stream.error, partialBytes: stream.partialBytes,
|
||||
providerReadyAtMs: stream.providerReadyAtMs, clockHelloAtMs: stream.clockHelloAtMs, readinessObservedAtMs: stream.readinessObservedAtMs,
|
||||
emissions: [...stream.records.values()].sort((a, b) => a.index - b.index) })));
|
||||
|
||||
@@ -11,7 +11,14 @@ process.env.PAPERCLIP_INGESTION_BROWSER_RECORDING = browserRecording ? "on" : "o
|
||||
const appRoot = path.resolve(process.env.PAPERCLIP_INGESTION_APP_ROOT ?? path.resolve(import.meta.dirname, "../.."));
|
||||
const quote = (value: string) => `'${value.replaceAll("'", "'\\''")}'`;
|
||||
const bootstrap = path.resolve(import.meta.dirname, "fixtures/native-ingestion-bootstrap.mjs");
|
||||
const compiledCommand = `node --import ./server/node_modules/tsx/dist/loader.mjs ${quote(bootstrap)} && node --import ./server/node_modules/tsx/dist/loader.mjs server/dist/index.js`;
|
||||
const transport = process.env.PAPERCLIP_INGESTION_TRANSPORT ?? "sandbox-remote-with-local-stop";
|
||||
if (transport !== "sandbox-remote-with-local-stop") throw new Error("H4 requires remote streaming with the original local Stop workload");
|
||||
if (process.env.DATABASE_URL) {
|
||||
throw new Error("Remote ingestion fixture requires a new disposable embedded database");
|
||||
}
|
||||
const remoteServer = path.resolve(import.meta.dirname, "fixtures/native-ingestion-remote-server.mjs");
|
||||
const compiledEntry = `${quote(remoteServer)} --compiled-entry=server/dist/index.js`;
|
||||
const compiledCommand = `node --import ./server/node_modules/tsx/dist/loader.mjs ${quote(bootstrap)} && node --import ./server/node_modules/tsx/dist/loader.mjs ${compiledEntry}`;
|
||||
const command = process.env.PAPERCLIP_INGESTION_SERVER_COMMAND ?? compiledCommand;
|
||||
const autoStartingOnboard = /\bonboard\b[^;&]*(?:--yes|-y)\b/.test(command);
|
||||
if (command.includes("server/dist/index.js") && autoStartingOnboard) {
|
||||
@@ -26,7 +33,7 @@ const providerHome = fs.mkdtempSync(path.join(os.tmpdir(), "ingestion-provider-"
|
||||
export default defineConfig({
|
||||
...base,
|
||||
testMatch: "native-event-ingestion.journey.ts",
|
||||
timeout: 600_000,
|
||||
timeout: 900_000,
|
||||
...(evidence ? { outputDir: path.join(evidence, "test-results"),
|
||||
reporter: [["list"], ["json", { outputFile: path.join(evidence, "playwright.json") }]] as ReporterDescription[] } : {}),
|
||||
use: { ...base.use, trace: browserRecording ? "on" : "off", video: "off", actionTimeout: 20_000 },
|
||||
@@ -37,6 +44,7 @@ export default defineConfig({
|
||||
timeout: 240_000,
|
||||
env: {
|
||||
...server?.env,
|
||||
PAPERCLIP_INGESTION_TRANSPORT: transport,
|
||||
PAPERCLIP_STOP_CODEX_COMMAND: path.resolve(import.meta.dirname, "fixtures/native-ingestion-bin/codex"),
|
||||
CODEX_HOME: providerHome,
|
||||
OPENAI_API_KEY: "ingestion-fixture-key",
|
||||
|
||||
Reference in new issue
Block a user