diff --git a/doc/benchmarks/2026-11-03-native-event-ingestion-cloud-stop.md b/doc/benchmarks/2026-11-03-native-event-ingestion-cloud-stop.md index fd33e909ff7..bd46c23cee0 100644 --- a/doc/benchmarks/2026-11-03-native-event-ingestion-cloud-stop.md +++ b/doc/benchmarks/2026-11-03-native-event-ingestion-cloud-stop.md @@ -1,13 +1,17 @@ # Real cloud Stop acceptance (2026-11-03) -The normal task UI's Stop control cancelled a native run in a real Daytona sandbox. The production-built controller served `137501f595fd5ee13d395e26b4147ca8bc4d4eb6`; the provider was an explicitly declared deterministic Codex fixture executing a real `sleep 60`. This adds cloud termination coverage to the separate [live-LLM journey](2026-11-03-native-event-ingestion-live.md). +Fresh qualification passed on the production-built application at `914129afb141d81c0d6d8c5f0ea11c20bd69a3da`, after integrating the current upstream Stop finalizer and recovery changes. The ordinary task UI stopped a real Daytona sandbox running the current Runner and a deterministic Codex fixture with a real `sleep 60`. This is functional cloud lifecycle acceptance; the separate [live-provider journey](2026-11-03-native-event-ingestion-live.md) used a real model. -The [recorded proof](native-event-ingestion-2026-11-03/cloud-stop/qualification.json) contains the original native acknowledgement, shared cancellation acknowledgement with `provider_termination_receipt`, matching stopped-provider receipt, and the exact lease's successful release. Independent SDK observations confirmed child retirement, absence of its forbidden effect, and the same labelled sandbox's stopped state. Reload and direct PostgreSQL checks verified the retained transcript, canonical event hashes, cursor projection, and absence of a successor or accepted result. +The [current proof](native-event-ingestion-2026-11-03/cloud-stop-ce33-914/qualification.json) records native acknowledgement, shared acknowledgement with `provider_termination_receipt`, the exact lease's successful release, and the matching stopped-provider receipt. Independent SDK observations confirmed the young process's birth identity before the UI click, child retirement with no forbidden effect, and the labelled sandbox's stopped state. Reload and direct PostgreSQL checks verified the cancelled run, 82 global rows, all 16 canonical hashes, current cursor marker, and no successor or accepted result. -This is one functional sample. Its acknowledgement timings establish no latency distribution, throughput improvement, or deployed-production gain. The database checks retain the existing sparse-source/cancellation policy; they do not equate every ordinary streaming wire ACK with database durability. +All six task-owned sandboxes, including the historical attempts, were independently confirmed absent. The [cleanup and credential scan](native-event-ingestion-2026-11-03/cloud-stop-ce33-914/cleanup-and-scan.json), [executed helper hashes](native-event-ingestion-2026-11-03/cloud-stop-ce33-914/executed-helpers.json), [toolchain](native-event-ingestion-2026-11-03/cloud-stop-ce33-914/toolchain.json), and [manifest](native-event-ingestion-2026-11-03/cloud-stop-ce33-914/manifest.json) make the result inspectable. The current journey executed the corrected public cleanup helper whose SHA starts `753ea9b8`. See its [environment and reproduction notes](native-event-ingestion-2026-11-03/cloud-stop-ce33-914/README.md). -All five created sandboxes were independently confirmed absent. The [attempt ledger](native-event-ingestion-2026-11-03/cloud-stop/attempts.json) preserves four earlier harness failures and their cleanup. The controller, database and credential-bearing temporary data were removed; retained and decompressed artifacts were scanned for the actual credentials. The [reproduction recipe](native-event-ingestion-2026-11-03/cloud-stop/README.md) declares live versus simulated dependencies, resource bounds, immutable image, SDK, runtime hashes, and the public helper's credential, resource-cap and clock-guard adaptations. The [manifest](native-event-ingestion-2026-11-03/cloud-stop/manifest.json) records exact artifact hashes. +This is one functional sample, with no throughput, latency-distribution, deployed-production or live-LLM claim. Sparse source history and the existing raw-output recovery policy remain in force: ordinary wire ACKs do not universally establish database durability. -![Actual streaming and real command before Stop](native-event-ingestion-2026-11-03/cloud-stop/before-stop.png) +![Current real sandbox streaming and command before Stop](native-event-ingestion-2026-11-03/cloud-stop-ce33-914/before-stop.png) -![Stopped run and persisted text after reload](native-event-ingestion-2026-11-03/cloud-stop/after-stop-reload.png) +![Current stopped run and persisted text after reload](native-event-ingestion-2026-11-03/cloud-stop-ce33-914/after-stop-reload.png) + +## Historical qualification + +The earlier controller at `137501f595fd5ee13d395e26b4147ca8bc4d4eb6` also passed. Its [proof](native-event-ingestion-2026-11-03/cloud-stop/qualification.json), [four preceding harness failures](native-event-ingestion-2026-11-03/cloud-stop/attempts.json), [recipe and current cleanup helpers](native-event-ingestion-2026-11-03/cloud-stop/README.md), and [artifact manifest](native-event-ingestion-2026-11-03/cloud-stop/manifest.json) remain available. They are historical evidence; the fresh journey above qualifies the newer runtime. diff --git a/doc/benchmarks/2026-11-03-native-event-ingestion-live.md b/doc/benchmarks/2026-11-03-native-event-ingestion-live.md index abb26287d41..1f71a13cc9d 100644 --- a/doc/benchmarks/2026-11-03-native-event-ingestion-live.md +++ b/doc/benchmarks/2026-11-03-native-event-ingestion-live.md @@ -1,74 +1,21 @@ # Native ingestion live-provider verification (2026-11-03) -The ordinary native task flow and Stop journey passed against the compiled -application at `b9a66be242395a4bab671d512643e338fbac77fe`. This is functional -integration acceptance, separate from paired performance measurements. +The ordinary task, tool, completed result, reload and Stop journeys passed on production-built application `914129afb141d81c0d6d8c5f0ea11c20bd69a3da`. OpenCode and `openrouter/deepseek/deepseek-v4-flash-0731` were real dependencies. The disposable environment used macOS arm64, Node26.4 and PostgreSQL18.1. This is functional integration evidence, separate from paired performance acceptance. -The environment was macOS arm64, Node 26.4.0, embedded PostgreSQL 18.1, -OpenCode and the real provider model -`openrouter/deepseek/deepseek-v4-flash-0731`. The database and company were -isolated disposable fixtures. Server/UI build hashes and exact identifiers are -retained in [launch provenance](native-event-ingestion-2026-11-03/live/live-launch.json). +Original retained browser frames show text growing from a short partial phrase to numbered sentences before later ordinary deltas and the accepted result. The initial helper's chosen observation happened during `workspace_finalizing`; it proves only pre-terminal visibility. The [retained progressive evidence](native-event-ingestion-2026-11-03/live-ce33-914/retained-progressive-streaming-proof.json) supplies earlier unmodified frames, matching source identities, later emission/commit timestamps and the original selector's missed live interstitial. The earlier UI run response omitted native phase; this evidence does not invent a provider-active phase value. -| Journey | Observed and independently checked outcome | -| --- | --- | -| Normal visible task creation | Ordinary assistant text rendered while the run was active; native `list_task_documents` completed; the result was accepted. | -| Reload | The completed transcript and result remained present. The run stored 33 `item.delta` events; an ordinary assistant delta was independently matched while active. | -| Stop | A second task executed native `bash` with `sleep 60`; clicking Stop produced canonical cancellation acknowledgement. Reload retained the stopped transcript without a second run. | -| Persistence | Direct PostgreSQL reads verified 126/69 global rows, company/run/source identities, all 101 canonical payload hashes and projection markers. Physical process settlement was independently recorded. | -| Cleanup | Owned server and PostgreSQL stopped, both ports independently closed, temporary credential-bearing directories removed. | +The success run `688a6c34-f658-48ce-8ce6-61aeb2f72094` completed native `list_task_documents`, accepted its result and retained the transcript after reload. The Stop run `51ad4429-6d34-4df9-bdb9-a54b91842d5a` was stopped while a positively identified real Runner/provider/shell/sleep process chain was alive. All four captured OS birth identities were independently absent after settlement, with a durable `native.local_process_stopped` receipt and no successor after reload. The 372ms acknowledgement is one functional observation, not a performance comparison. -Success run: `e51fe504-e971-4ba1-a146-d91b5b8c92ae`. -Stop run: `9807b34f-4325-462d-bcd2-4addf66696e3`. -The stopped source's cursor remains at 24 before missing raw source sequences -25/26 and retained sequence 27. This preserves the sparse-source contract. A raw -wire acknowledgement does not universally imply ordinary transcript durability. +Read-only PostgreSQL verification checked 122/71 global rows, 99 canonical payload hashes, current projection markers, and the stopped source's correct cursor26 with pending range `[29,29]`. The owned controller, database, monitors and credential-bearing data were removed; exact ports independently closed and retained/decompressed evidence passed credential scanning. Ordinary wire ACKs retain their existing recovery policy and are not universally database receipts. -The observed Stop acknowledgement interval was 481 ms. It is one functional -sample; no performance comparison or tail-latency claim follows from it. +Inspect the [qualification](native-event-ingestion-2026-11-03/live-ce33-914/qualification.json), [independent database proof](native-event-ingestion-2026-11-03/live-ce33-914/independent-database-proof.json), [process proof](native-event-ingestion-2026-11-03/live-ce33-914/independent-birth-settlement-proof.json), [execution receipts](native-event-ingestion-2026-11-03/live-ce33-914/execution-receipts.json), and [manifest](native-event-ingestion-2026-11-03/live-ce33-914/manifest.json). The [README](native-event-ingestion-2026-11-03/live-ce33-914/README.md) distinguishes exact executed helpers, retained-trace review, future observer corrections and their offline checks. Future helper corrections were not retroactively executed in this journey. [Real cloud Stop](2026-11-03-native-event-ingestion-cloud-stop.md) is independently qualified. -## Inspect the evidence +![Original short partial response](native-event-ingestion-2026-11-03/live-ce33-914/partial-one.jpeg) -- [Browser journey outcomes and identifiers](native-event-ingestion-2026-11-03/live/live-provider-result.json) -- [Independent PostgreSQL proof](native-event-ingestion-2026-11-03/live/independent-database-proof.json) -- [Independent lifecycle and OS settlement proof](native-event-ingestion-2026-11-03/live/independent-lifecycle-process-proof.json) -- [Build receipt](native-event-ingestion-2026-11-03/live/build-receipt.json) and [command exit/helper hash receipts](native-event-ingestion-2026-11-03/live/execution-receipts.json) -- [Cleanup receipt](native-event-ingestion-2026-11-03/live/cleanup.json) -- [Credential scan](native-event-ingestion-2026-11-03/live/artifact-secret-scan.json) and [file SHA256 manifest](native-event-ingestion-2026-11-03/live/manifest.json) +![Original response after more streamed text](native-event-ingestion-2026-11-03/live-ce33-914/partial-two.jpeg) -The original artifact scan inspected 3,710 retained files and decompressed trace -entries and found zero exact qualification-credential matches. These selected -public files were scanned again before copying. No credential is committed. -The screenshots below supply the reviewable visual evidence. The video and -full trace remain optional local supplements. +![Stopped live-provider run after reload](native-event-ingestion-2026-11-03/live-ce33-914/after-live-stop.png) -![Ordinary assistant text during an active native run](native-event-ingestion-2026-11-03/live/live-streaming.png) +## Historical qualification -![Accepted native result](native-event-ingestion-2026-11-03/live/live-result.png) - -![Persisted result after reload](native-event-ingestion-2026-11-03/live/live-reloaded.png) - -![Active native tool before Stop](native-event-ingestion-2026-11-03/live/before-live-stop.png) - -![Stopped native run after reload](native-event-ingestion-2026-11-03/live/after-live-stop.png) - -## Reproduce - -[The live-provider helpers and instructions](../../scripts/native-ingestion/README-live.md) -document the tested helper origins and follow-up review fixes. The journey used -the original helper bytes recorded in the B9 source tag and command receipts; -current helpers repair setup/observation false failures, with a separate nine-case -[red/green receipt](native-event-ingestion-2026-11-03/live/helper-review-regressions.json). The independent database -verifier is a reconstructed utility, now exercised against this live journey's -raw PostgreSQL rows. Its [exact command record](native-event-ingestion-2026-11-03/live/independent-database-verifier-command.json) -and aggregate proof are included. Its separate real-PostgreSQL qualification -also rejected persisted hash/cursor corruption. The owned live database was -removed after verification. The earlier 6afeba4 journey remains historical; -its different 84-digest proof did not use this newer verifier. - -The earlier browser attempt failed observer qualification: it watched a tool -progress marker only in completed transcript bubbles and missed its live -presentation. The successful helper requires a matching ordinary delta visibly -rendered while the run is still active. The failed attempt remains excluded. -This work does not establish deployed production performance or a 10,000-agent -qualification. +The earlier B9 journey remains available in its [result](native-event-ingestion-2026-11-03/live/live-provider-result.json), [101-hash PostgreSQL proof](native-event-ingestion-2026-11-03/live/independent-database-proof.json), [process proof](native-event-ingestion-2026-11-03/live/independent-lifecycle-process-proof.json), and [manifest](native-event-ingestion-2026-11-03/live/manifest.json). The [live helper recipe](../../scripts/native-ingestion/README-live.md) records its original execution and subsequent setup/observer fixes. These historical results do not substitute for the newer runtime's qualification above. No journey establishes deployed performance or a 10,000-agent qualification. diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/README.md b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/README.md new file mode 100644 index 00000000000..26cb8e16308 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/README.md @@ -0,0 +1,17 @@ +# Fresh common-base real Daytona Stop qualification + +The normal production build at `914129afb141d81c0d6d8c5f0ea11c20bd69a3da` passed a fresh real-Daytona Stop journey. This replaces the earlier 137 proof for the updated runtime: the upstream native Stop finalizer, recovery path and run-log wrapper changed, so source-equivalence alone was insufficient. The earlier proof and its failed harness attempts remain historical evidence. + +This current epoch used exactly one new sandbox and succeeded on its first journey. The provider was an explicit deterministic Codex fixture, with a real `sleep 60` process inside real Daytona, real native runnerd, normal authenticated ingress and the ordinary task UI Stop button. The separate fresh local live-LLM journey uses the same immutable Mac build. This cloud result is functional acceptance, not performance or live-LLM evidence. + +`qualification.json` records actual normalized assistant text, visible streaming, a young independently inspected process, the exact UI cancel response, native acknowledgement, shared acknowledgement with `provider_termination_receipt`, the released successful terminal lease and matching `paperclip.remote-termination.v1` receipt, independent stopped provider state, child retirement with no forbidden effect, reload/persisted text, no successor and no accepted result. `database-proof.json` independently verifies 82 retained global rows, 16 canonical payload hashes and the current cursor/range projection in a read-only repeatable-read PostgreSQL snapshot. Gaps remain subject to the existing raw-transport policy; ordinary wire ACKs are not universally database-durable. + +The new Linux runner was built from Rust tree `eac015472eb88b49aa92e6f30def348aa34b4911`, binary SHA256 `6873b6325850a0f9eeedb43eca83413ac2379e7e804dee0ef614aa87c57859c8`. Both Mac and Linux actual toolchains were 1.97.1; actual verbose records and override facts are in `toolchain.json`. The old `bf628` target/artifact was not reused. The immutable image supplied OS/Node only; the current daemon and self-contained fixture were staged through supported overrides. + +`executed-helpers.json` pins the public helpers actually used, including `run.mjs` SHA256 `753ea9b830bbaa9be6fa8510fe2b7bdcc1099248939f7aa28d51d29fb854e4e9`. They were copied unchanged from the owning PR's `cloud-stop/helpers/`. The source-driven offline cleanup regressions prove all observed resource IDs reach the ledger before cap failure or a first SDK lookup error, and cleanup checks exact ownership before deletion. The helper has one resource per epoch, no automatic retry, a 30-minute provider TTL and a host watchdog. The task-wide bound was six resources, with one active at a time; no seventh was created. + +Reproduce with the owning `cloud-stop/README.md` commands and its unchanged `helpers/` files against a fresh, independently installed and normally built exact revision. Bind the provider key privately through the credential manager/environment; never put it in arguments, logs or files. The tested controller was macOS arm64/Node 26.4.0; SDK 0.203.0 and the immutable image are pinned. Do not weaken the normal visible Stop, exact target/receipt, independent provider-state or cleanup gates. + +Provider process age uses only the provider clock: 478 ms at the awaited pre-click snapshot. The remote RPC completed before the UI click, but remote and controller clocks are independent; no cross-host interval is inferred. Receipt timestamps are one functional observation and must not be used as a performance comparison. Full traces/videos remain in private task evidence. Compact screenshots are included. + +Normal environment deletion returned HTTP success; exact-owned SDK deletion then established final provider absence. `cleanup-and-scan.json` records all six task-owned provider IDs freshly absent, exact credential scans over retained/decompressed evidence, and local controller/database/data cleanup. Only 22 read-only directories inside the recorded stopped data root were made owner-writable, without following symlinks, before removing that root. Shared application source/build outputs remain intact. diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/after-stop-reload.png b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/after-stop-reload.png new file mode 100644 index 00000000000..cdee6063847 Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/after-stop-reload.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/before-stop.png b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/before-stop.png new file mode 100644 index 00000000000..0a8e30c9c9b Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/before-stop.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/cleanup-and-scan.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/cleanup-and-scan.json new file mode 100644 index 00000000000..1d2b4f960c7 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/cleanup-and-scan.json @@ -0,0 +1,68 @@ +{ + "status": "passed", + "allTaskOwnedResourcesAndCredentialScan": { + "status": "passed", + "checkedAt": "2026-11-04T00:30:33.752Z", + "providerApiOperations": "get only; no resource mutations", + "taskOwnedProviderCount": 6, + "providerStates": [ + { + "id": "61ea5024-e513-4234-8925-22109c3e9649", + "absent": true, + "errorName": "DaytonaNotFoundError", + "status": 404 + }, + { + "id": "3f713f74-c3d8-49e3-9e84-807192c6ea0e", + "absent": true, + "errorName": "DaytonaNotFoundError", + "status": 404 + }, + { + "id": "d61aec89-e2c4-4cc7-aaf3-6b7d2bf04c5e", + "absent": true, + "errorName": "DaytonaNotFoundError", + "status": 404 + }, + { + "id": "cbf9f558-49f2-4b84-bbe1-ea651502a6bb", + "absent": true, + "errorName": "DaytonaNotFoundError", + "status": 404 + }, + { + "id": "19e2daa7-e0fa-4427-988b-de136dd1b2b0", + "absent": true, + "errorName": "DaytonaNotFoundError", + "status": 404 + }, + { + "id": "79fd034e-b161-4ce0-87b0-a54c430be3f4", + "absent": true, + "errorName": "DaytonaNotFoundError", + "status": 404 + } + ], + "exactCredentialKinds": [ + "Daytona API key", + "1Password service-account token" + ], + "knownPreviewTokenValues": 0, + "scannedFilesAndDecompressedEntries": 660, + "matches": [] + }, + "localCleanup": { + "status": "passed", + "checkedAt": "2026-11-04T00:27:05.886Z", + "ownedControllerProcessGone": true, + "ownedPostgresProcessGone": true, + "ownedHttpPortClosed": true, + "ownedDatabasePortClosed": true, + "postmasterPidFileGone": true, + "credentialBearingDataRemoved": true, + "ownedDirectoriesVisited": 141, + "ownedReadonlyDirectoriesMadeWritable": 22, + "followedSymlinks": false, + "cleanupScope": "Only the recorded stopped qualification data directory; shared application outputs remain intact" + } +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/database-proof.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/database-proof.json new file mode 100644 index 00000000000..4ae59f2ce87 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/database-proof.json @@ -0,0 +1,35 @@ +{ + "verified": true, + "reconstructedVerifier": true, + "checkedAt": "2026-11-04T00:25:32.811Z", + "database": "Supplied disposable loopback PostgreSQL; read-only repeatable-read snapshot", + "limitation": "Cursor reconstruction requires the complete retained fixture history; no production or raw wire ACK durability claim", + "proofs": [ + { + "runId": "12cc7b1f-b698-4465-b9bb-dc3258328afb", + "companyId": "1211ed47-48bf-4607-91e3-8680d05c840d", + "status": "cancelled", + "nextEventSeq": 83, + "eventCursorProjectionSeq": 82, + "eventCursorRebuildPending": false, + "globalEvents": 82, + "canonicalPayloadDigestsVerified": 16, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "d4246355-c2ee-40a2-b297-5c4f65faf5c5", + "events": 16, + "contiguousSourceSeq": 15, + "pendingRanges": [ + { + "startSeq": 19, + "endSeq": 19 + } + ] + } + ] + } + ] +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/executed-helpers.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/executed-helpers.json new file mode 100644 index 00000000000..c959ae4f423 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/executed-helpers.json @@ -0,0 +1,37 @@ +{ + "status": "passed", + "publicHelperFilesUsedWithoutModification": { + "cloud-ui.mjs": "c43c112546709fca064842c529d27bc78363ffdd2ca4258830dd17c85affb6cc", + "launch.mjs": "b6124d665002da53628d3483452cbf30aa8b1acf5b759134c0298d13a5cd4aac", + "provider.mjs": "b62bd845e63a2736a76bcf834c2a2bd281db46079d5e5c786c7e996f9a9ef269", + "run.mjs": "753ea9b830bbaa9be6fa8510fe2b7bdcc1099248939f7aa28d51d29fb854e4e9", + "verify-failure-cleanup.mjs": "3ae73e3146ddb65051568e7b25e7b9f87fdd35ee6a649f9bc29b434ee8086d83", + "watchdog.mjs": "cbe0770ef67217a609b13ba5a557ee57dc9440180658314ff38919a81648fd2a" + }, + "mainPublicHelperSha256": "753ea9b830bbaa9be6fa8510fe2b7bdcc1099248939f7aa28d51d29fb854e4e9", + "offlineActualSourceCleanupCases": [ + { + "case": "cap1_with2_observed_owned_resources", + "status": "passed", + "allIdsLedgeredBeforeFailure": true, + "bothExactOwnedResourcesCleaned": true + }, + { + "case": "first_sdk_lookup_throws_with2_observed_owned_resources", + "status": "passed", + "allIdsLedgeredBeforeFailure": true, + "bothExactOwnedResourcesCleaned": true + } + ], + "offlineCloudCalls": 0, + "credentialBoundary": "A private credential-manager bootstrap supplied only DAYTONA_API_KEY in the journey child environment; service-account token was not inherited. The controller was launched with inherited credentials stripped. Normal encrypted company-secret API and reference were used.", + "resourceBounds": { + "currentEpochMaximum": 1, + "currentEpochActual": 1, + "maximumTaskTotal": 6, + "taskActual": 6, + "maximumConcurrent": 1, + "ttlMinutes": 30, + "automaticRetry": false + } +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/manifest.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/manifest.json new file mode 100644 index 00000000000..ad8ded56445 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/manifest.json @@ -0,0 +1,73 @@ +{ + "status": "ready_for_root_review_and_publication", + "testedAppRevision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "actualExecutedPublicHelperSha256": "753ea9b830bbaa9be6fa8510fe2b7bdcc1099248939f7aa28d51d29fb854e4e9", + "newLinuxRunnerSha256": "6873b6325850a0f9eeedb43eca83413ac2379e7e804dee0ef614aa87c57859c8", + "files": [ + { + "path": "README.md", + "bytes": 4151, + "sha256": "68aed06dd4ec51a18eec6489cadb9c78016583e43e64f52c0765cc78b235fead" + }, + { + "path": "after-stop-reload.png", + "bytes": 143426, + "sha256": "82706973e619d190416fc9bdd41728ca9eafb5d14c0cedeb6b409f1f6a6de252" + }, + { + "path": "before-stop.png", + "bytes": 202012, + "sha256": "d61feac74c943a88cbbda6f0f4882b62f51e57a9a3ae2c86bf3038ab73c5123a" + }, + { + "path": "cleanup-and-scan.json", + "bytes": 1998, + "sha256": "7742a55d584790adad7c2e4ccd724f4108e57420535eef13420b281e6738512d" + }, + { + "path": "database-proof.json", + "bytes": 1122, + "sha256": "97ec00424c34b9dbe40b4c78711370d01683709dcaa1a6e849b2f5be778f32a3" + }, + { + "path": "executed-helpers.json", + "bytes": 1665, + "sha256": "fef09f42ecade5751f2e92dfb703c98d2132893ea4e38a6e602e6d9246d41200" + }, + { + "path": "normalized-marker-db.json", + "bytes": 387, + "sha256": "14364a2df301cb83caecd125e3c3194b0ed72d0bcca17b89a717a69d5cacf666" + }, + { + "path": "qualification.json", + "bytes": 10214, + "sha256": "ef68dccda1ad35168ccb65ecc238f8dfa672d51037d7b41eae53ceccac7a2e93" + }, + { + "path": "toolchain.json", + "bytes": 3650, + "sha256": "3c327299b6fc284d15338703dac27f72871cc0d957645b3be95b737b1424ffa9" + } + ], + "totalBytes": 368625, + "checks": { + "functionalCloudStop": "passed", + "directPostgres": "82rows16canonicalhashes/currentprojectionpassed", + "freshProviderAbsenceCount": 6, + "providerAbsenceCheckedAt": "2026-11-04T00:30:33.752Z", + "credentialScanEntries": 660, + "credentialMatches": 0, + "privateMetadataCheck": "passed", + "localCleanup": "passed", + "immutableAppSourceBuildAfterJourney": "passed" + }, + "exclusions": [ + "privatecredentialbootstrapandOPmetadata", + "rawAPIpayloads", + "fulltrace/videoarchives", + "launchconfigs/localdatapaths", + "LinuxbinaryandprivateVMbuildpaths" + ], + "publication": "No repository copy, upload or PR edit by child; root owns publication." +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/normalized-marker-db.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/normalized-marker-db.json new file mode 100644 index 00000000000..602d0eb3121 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/normalized-marker-db.json @@ -0,0 +1,14 @@ +{ + "status": "passed", + "source": "independent read-only PostgreSQL", + "events": [ + { + "seq": "64", + "event_type": "item.delta", + "source_event_id": "d4246355-c2ee-40a2-b297-5c4f65faf5c5:12cc7b1f-b698-4465-b9bb-dc3258328afb:12", + "source_seq": "12", + "kind": "agentMessage", + "text": "CLOUD-STOP-STREAM: real sandbox process is starting. " + } + ] +} \ No newline at end of file diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/qualification.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/qualification.json new file mode 100644 index 00000000000..5a00f9f3a6d --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/qualification.json @@ -0,0 +1,261 @@ +{ + "schema": "paperclip.real-cloud-stop-qualification.public.v2", + "status": "passed", + "scope": "Fresh latest-common-base real-Daytona Stop qualification; one resource, one successful journey. Deterministic Codex provider fixture with a real sleep60 process, current real runnerd, normal production-built Paperclip and actual provider lifecycle. Not a performance comparison or live-LLM claim.", + "testedRevision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "productionBuildCommit": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "platform": { + "node": "v26.4.0", + "os": "darwin", + "architecture": "arm64" + }, + "runnerRustTree": "eac015472eb88b49aa92e6f30def348aa34b4911", + "hashes": { + "server": "7db3ab75d39e5d6a467f961a267991cc55781439572cc37e45b4599908d1e23b", + "ui": "847e43ecc2469a5b54dcf89e0be4a71ba750f9d123e9d893ed90a3b3f693959e", + "linuxRunner": "6873b6325850a0f9eeedb43eca83413ac2379e7e804dee0ef614aa87c57859c8", + "provider": "b62bd845e63a2736a76bcf834c2a2bd281db46079d5e5c786c7e996f9a9ef269" + }, + "runtime": { + "node": "/opt/paperclip-runner/provider-pack/node_modules/node/bin/node", + "nodeVersion": "v24.21.0", + "nodeSha256": "7fde7b8afa198da66257f42ee2001d874c7355631e6d1579a5fb5ef1f246df4c", + "runnerSha256": "6873b6325850a0f9eeedb43eca83413ac2379e7e804dee0ef614aa87c57859c8", + "fixtureSha256": "b62bd845e63a2736a76bcf834c2a2bd281db46079d5e5c786c7e996f9a9ef269", + "runnerMetadata": { + "binaryContractVersion": 2, + "binaryName": "paperclip-runnerd", + "durableSessionCapabilities": [ + "unlimited_runtime", + "connection_lease_renewal" + ], + "externalProviderCapabilities": [ + "openai_dot_mcp", + "grok_bot_mcp", + "muse_external_v1" + ], + "harnessDriverVersion": 1, + "nativeExecutionVersion": 1, + "packageName": "@paperclipai/paperclip-runner", + "packageVersion": "0.0.0", + "providerPacks": [], + "prp": { + "maximumVersion": 2, + "minimumVersion": 1, + "name": "paperclip.runner" + }, + "prpTransportModes": [ + "dial_ws_loopback", + "dial_wss", + "listen_ws" + ], + "schema": "paperclip-runner/runnerd-build-metadata/v1" + }, + "fixtureState": "awaiting-observer" + }, + "dependencies": { + "provider": "deterministic Codex app-server fixture with real sleep60; no model call", + "sandbox": "real Daytona", + "sdk": "0.203.0", + "image": "ghcr.io/paperclipai/paperclip-daytona-runner@sha256:bff4c3f291087a0eeae37e4c20dd51857b92833eaf73ba3aca4157f37de1e109", + "controller": "normal production-built local Paperclip", + "actualLlmJourney": "separate real OpenCode/DeepSeek acceptance" + }, + "issueId": "edc4ce6f-641c-4198-ad18-b37ccf03ec2d", + "runId": "12cc7b1f-b698-4465-b9bb-dc3258328afb", + "leaseId": "73a0f737-9830-44b2-9055-19278957ef4e", + "providerLeaseId": "79fd034e-b161-4ce0-87b0-a54c430be3f4", + "checks": { + "normalApiBindings": true, + "currentDaemonAndFixture": true, + "visibleStreamingBeforeTerminal": true, + "realSleepObserved": true, + "nativeAcknowledgement": true, + "sharedProviderReceiptAcknowledgement": true, + "independentChildRetirementNoEffect": true, + "exactLeaseProviderReceipt": true, + "independentProviderStopped": true, + "reloadAndNoSuccessorOrResult": true + }, + "visibleSourceMarker": { + "eventType": "item.delta", + "kind": "agentMessage", + "text": "CLOUD-STOP-STREAM: real sandbox process is starting. ", + "sourceEventId": "d4246355-c2ee-40a2-b297-5c4f65faf5c5:12cc7b1f-b698-4465-b9bb-dc3258328afb:12", + "sourceSeq": 12 + }, + "uiStop": { + "method": "POST", + "path": "/api/heartbeat-runs/12cc7b1f-b698-4465-b9bb-dc3258328afb/cancel", + "httpStatus": 200, + "requestedAtControllerMs": 1793751885311, + "responseRunId": "12cc7b1f-b698-4465-b9bb-dc3258328afb", + "responseStatus": "cancelled", + "nativeCancellation": { + "runId": "12cc7b1f-b698-4465-b9bb-dc3258328afb", + "scope": "run", + "schema": "paperclip.native-cancellation.v1", + "effects": [ + "release_run_resources" + ], + "issueId": "edc4ce6f-641c-4198-ad18-b37ccf03ec2d", + "intentId": "native-cancellation:1cedca68-9e0d-4ab6-9c6d-df2192ee1be3", + "companyId": "1211ed47-48bf-4607-91e3-8680d05c840d", + "decisionId": null, + "dispatched": true, + "reasonCode": "cancellation_run_only", + "recordedAt": "2026-11-04T00:24:45.363Z", + "dispatchState": "acknowledged", + "intentAuditId": "2bc9850f-2235-4845-8175-5f046fdf4c17", + "acknowledgedAt": "2026-11-04T00:24:45.754Z", + "acknowledgementAuditId": "2b197246-165f-4fab-a18a-70b7f6478416", + "priorCoordinatorDecisionId": null + }, + "executionCancellation": { + "proof": "provider_termination_receipt", + "state": "acknowledged", + "requestedAt": "2026-11-04T00:24:45.351Z", + "acknowledgedAt": "2026-11-04T00:24:50.626Z" + }, + "timingScope": "One functional observation; no latency distribution or performance claim" + }, + "processProof": { + "beforeStop": { + "remoteObservedAtMs": 1793751885334, + "remoteStartedAtMs": 1793751884856, + "remoteProcessAgeAtObservationMs": 478, + "child": { + "pid": 1216, + "ppid": 1161, + "pgid": 1216, + "startTicks": "2214887048", + "bootId": "89a86dc0-653a-4053-ae37-0701d477ebb5" + }, + "groupProcesses": [ + { + "pid": 1216, + "ppid": 1161, + "pgid": 1216, + "startTicks": "2214887048", + "command": "/bin/sh -c sleep 60; printf forbidden > \"$1\" cloud-stop-owned-child /home/daytona/paperclip-workspace/.cloud-stop-forbidden-effect " + }, + { + "pid": 1217, + "ppid": 1216, + "pgid": 1216, + "startTicks": "2214887048", + "command": "sleep 60 " + } + ], + "effectPresent": false + }, + "retirement": { + "remoteObservedAtMs": 1793751885626, + "phase": "interrupted-child-retired", + "childLive": false, + "groupProcesses": [], + "effectPresent": false + }, + "causalOrdering": "Independent SDK snapshot promise resolved before ordinary UI Stop click.", + "clockLimitation": "Provider and controller/browser clocks are independent; no cross-host process age or delivery interval is inferred." + }, + "terminalLease": { + "id": "73a0f737-9830-44b2-9055-19278957ef4e", + "companyId": "1211ed47-48bf-4607-91e3-8680d05c840d", + "environmentId": "0a86ee31-eca7-4aa3-aa57-5641d07a3c79", + "heartbeatRunId": "12cc7b1f-b698-4465-b9bb-dc3258328afb", + "provider": "daytona", + "providerLeaseId": "79fd034e-b161-4ce0-87b0-a54c430be3f4", + "status": "released", + "releasedAt": "2026-11-04T00:24:50.619Z", + "cleanupStatus": "success", + "receipt": { + "runId": "12cc7b1f-b698-4465-b9bb-dc3258328afb", + "state": "stopped", + "schema": "paperclip.remote-termination.v1", + "leaseId": "73a0f737-9830-44b2-9055-19278957ef4e", + "provider": "daytona", + "companyId": "1211ed47-48bf-4607-91e3-8680d05c840d", + "confirmedAt": "2026-11-04T00:24:50.619Z", + "providerLeaseId": "79fd034e-b161-4ce0-87b0-a54c430be3f4" + } + }, + "providerReceipt": { + "runId": "12cc7b1f-b698-4465-b9bb-dc3258328afb", + "state": "stopped", + "schema": "paperclip.remote-termination.v1", + "leaseId": "73a0f737-9830-44b2-9055-19278957ef4e", + "provider": "daytona", + "companyId": "1211ed47-48bf-4607-91e3-8680d05c840d", + "confirmedAt": "2026-11-04T00:24:50.619Z", + "providerLeaseId": "79fd034e-b161-4ce0-87b0-a54c430be3f4" + }, + "independentProviderState": { + "id": "79fd034e-b161-4ce0-87b0-a54c430be3f4", + "state": "stopped", + "labels": { + "paperclip-provider": "daytona", + "paperclip-company-id": "1211ed47-48bf-4607-91e3-8680d05c840d", + "paperclip-environment-id": "0a86ee31-eca7-4aa3-aa57-5641d07a3c79", + "paperclip-run-id": "12cc7b1f-b698-4465-b9bb-dc3258328afb", + "paperclip-reuse-lease": "false" + }, + "observedAt": 1793751891040 + }, + "databaseProof": { + "runId": "12cc7b1f-b698-4465-b9bb-dc3258328afb", + "companyId": "1211ed47-48bf-4607-91e3-8680d05c840d", + "status": "cancelled", + "nextEventSeq": 83, + "eventCursorProjectionSeq": 82, + "eventCursorRebuildPending": false, + "globalEvents": 82, + "canonicalPayloadDigestsVerified": 16, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "d4246355-c2ee-40a2-b297-5c4f65faf5c5", + "events": 16, + "contiguousSourceSeq": 15, + "pendingRanges": [ + { + "startSeq": 19, + "endSeq": 19 + } + ] + } + ] + }, + "cleanup": { + "normalEnvironmentDelete": true, + "exactOwnedFallbackDelete": true, + "providerAbsence": [ + { + "id": "79fd034e-b161-4ce0-87b0-a54c430be3f4", + "state": "absent", + "observedAt": 1793751895675 + } + ], + "watchdogDisarmed": true + }, + "sourceDurabilityLimit": "Cursor reconstruction requires the complete retained fixture history; no production or raw wire ACK durability claim", + "historicalProofScope": "Prior137 proof remains historical because the upstream native Stop finalizer/recovery/run-log wrapper changed. This fresh journey tests914 with the newly built eac015 runner.", + "artifactIntegrityAfterQualification": { + "status": "passed", + "checkedAt": "2026-11-04T00:30:10.858Z", + "sourceRevision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "trackedSourceClean": true, + "compiledBuildStamp": { + "commit": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da" + }, + "compiledHashesAfterQualification": { + "server": "7db3ab75d39e5d6a467f961a267991cc55781439572cc37e45b4599908d1e23b", + "ui": "847e43ecc2469a5b54dcf89e0be4a71ba750f9d123e9d893ed90a3b3f693959e", + "macRunner": "3e9acd73d2fb380d3d95e35de0e6ed78232b1be5dcbb3a96c76ca0d8b8c951b2", + "vendorMacRunner": "3e9acd73d2fb380d3d95e35de0e6ed78232b1be5dcbb3a96c76ca0d8b8c951b2" + }, + "matchedBeforeQualificationBuild": true + } +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/toolchain.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/toolchain.json new file mode 100644 index 00000000000..9262d6be941 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop-ce33-914/toolchain.json @@ -0,0 +1,63 @@ +{ + "runnerRustTree": "eac015472eb88b49aa92e6f30def348aa34b4911", + "mac": { + "rustc": "rustc 1.97.1 (8bab26f4f 2026-07-14) (Homebrew)\nbinary: rustc\ncommit-hash: 8bab26f4f68e0e26f0bb7960be334d5b520ea452\ncommit-date: 2026-07-14\nhost: aarch64-apple-darwin\nrelease: 1.97.1\nLLVM version: 22.1.8", + "cargo": "cargo 1.97.1 (c980f4866 2026-06-30) (Homebrew)\nrelease: 1.97.1\ncommit-hash: c980f4866141969fab6254a680546a277789d6f0\ncommit-date: 2026-06-30\nhost: aarch64-apple-darwin\nlibgit2: 1.9.7 (sys:0.20.4 system)\nlibcurl: 8.7.1 (sys:0.4.88+curl-8.20.0 system ssl:(SecureTransport) LibreSSL/3.3.6)\nos: Mac OS 26.5.2 [64-bit]", + "overrideFacts": { + "RUSTUP_TOOLCHAIN": null, + "RUSTFLAGS": null, + "CARGO_ENCODED_RUSTFLAGS": null, + "CARGO_TARGET_DIR": null, + "RUSTC": null, + "RUSTC_WRAPPER": null + }, + "artifactFingerprintFacts": [ + { + "artifact": "paperclip-runner-core-a22b1419a8800206/bin-paperclip-runnerd.json", + "rustflags": [], + "profile": 2040997289075261400, + "compileKind": 0 + } + ], + "ownFreshTarget": true, + "oldTargetReused": false + }, + "linux": { + "sourceTree": "eac015472eb88b49aa92e6f30def348aa34b4911", + "binarySha256": "6873b6325850a0f9eeedb43eca83413ac2379e7e804dee0ef614aa87c57859c8", + "binaryMode": "0o755", + "platform": "linux-x64", + "architecture": "x86_64", + "rustInputObjects": { + "runner": "eac015472eb88b49aa92e6f30def348aa34b4911", + "runner/Cargo.toml": "e5ea7fb7c65397eab9ba3f4e69b0beff80d59aaf", + "runner/Cargo.lock": "809b81962dbabf3aa3dce11aa4a4b3384c55f31d", + "runner/crates/runner-core/Cargo.toml": "08ded319dbaf35e7d63e143f6dd0fc9a3a24388f", + "package.json": "ae8b5a3ad0d89e3e2bc387fca1a3d11676212ac0", + "rust-toolchain.toml": "cf01515b3a12ae24e658a2a30c47b2d4b8f97dfb", + "protocol": "fd1ab6ec49a32eda46608036faaa8a87bb7952eb", + "protocol/manifest.json": "b0c918e3514e8f1d6a32a6bdcdd069db840a7622", + "generated/semantic-action-catalog.json": "10800237d17af2eb0839220f98c8ec58a49bce83", + "antigravity-distributions.json": "c6836e59b9888d99cfdfe0f8f48c01a5f566ff33", + "scripts": "85cc40f3658570b1bfd24723e62ba73224c7dfbd", + "acpx-profiles.json": "a3993363cd05df1c158ede0ba798d10506cceaec", + "cursor-distributions.json": "c049c080192c62f9300953d7a28f617e45555812", + "cursor-contract.json": "a7f8eca443cd7bbaf094da808be0650edf063db9", + "src/drivers/acpx/kimi-closure-pins.ts": "f7942c01f1eaa18c592c41344f15710b0507732c" + }, + "rustcVersion": "rustc 1.97.1 (8bab26f4f 2026-07-14)\nbinary: rustc\ncommit-hash: 8bab26f4f68e0e26f0bb7960be334d5b520ea452\ncommit-date: 2026-07-14\nhost: x86_64-unknown-linux-gnu\nrelease: 1.97.1\nLLVM version: 22.1.6\n", + "cargoVersion": "cargo 1.97.1 (c980f4866 2026-06-30)\nrelease: 1.97.1\ncommit-hash: c980f4866141969fab6254a680546a277789d6f0\ncommit-date: 2026-06-30\nhost: x86_64-unknown-linux-gnu\nlibgit2: 1.9.2 (sys:0.20.4 vendored)\nlibcurl: 8.20.0-DEV (sys:0.4.88+curl-8.20.0 vendored ssl:OpenSSL/3.6.2)\nssl: OpenSSL 3.6.2 7 Apr 2026\nos: Ubuntu 24.4.0 (noble) [64-bit]\n", + "effectiveRustEnvironment": { + "RUSTUP_TOOLCHAIN": "1.97.1", + "CARGO_TARGET_DIR": null, + "RUSTFLAGS": null, + "CARGO_ENCODED_RUSTFLAGS": null, + "CARGO_BUILD_TARGET": null, + "releaseProfile": "Cargo defaults; no custom target/features" + }, + "oldRustTargetNotReused": true, + "qualifiedBuildReceiptSha256": "e54b67a017ea8223effde240c94863dddc027817ea7d82b852290f71ec5118c2" + }, + "oldLinuxBfArtifactUsed": false, + "normalProductionBuildAndCleanTrackedSource": true +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/README.md b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/README.md new file mode 100644 index 00000000000..382ef89c70a --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/README.md @@ -0,0 +1,25 @@ +# Fresh real-provider functional acceptance on 914129af + +The normally built controller at `914129afb141d81c0d6d8c5f0ea11c20bd69a3da` passed one success journey and one Stop journey on November 3, 2026 local time (November 4 UTC). OpenCode and `openrouter/deepseek/deepseek-v4-flash-0731` were live dependencies. No deterministic provider supplied this journey. + +The original helper's selected source52 delta was observed preterminal during `workspace_finalizing`; it proves preterminal visibility. Earlier retained JPEG/DOM/source excerpts independently establish progressive visible content before subsequent ordinary deltas and result acceptance, as detailed below. No literal provider-active nativePhase is asserted from the retained public API shape. `qualification.json` also records ordinary task/chat UI creation, the completed native `list_task_documents` tool, accepted result and reload. + +The separate Stop task ran a real `sleep 60`. An independent monitor captured the exact same-run Runner/provider/shell/sleep parent chain and OS birth identity. The private browser adaptation waited for positive live proof, about one second into the sleep, before clicking the ordinary Stop control. The same captured birth chain was independently absent after settlement. Cancellation acknowledged and reload created no successor. The observed 372 ms acknowledgement is one functional sample and establishes no performance gain. + +The read-only raw PostgreSQL verifier checked 122/71 global rows and 72/27 canonical digests, 99 total, with current projection markers. Success cursors were 70/2; the stopped source correctly retained cursor 26 and pending range `[29,29]`. Separate raw SQL recorded `native.local_process_stopped` at global sequence 70. This preserves the existing raw-output recovery policy; ordinary wire ACKs do not universally establish durable storage. + +The source/dependencies/build outputs were consumed read-only from the build owner's fresh checkout. Build/runtime hashes and clean unchanged source were rechecked after the journeys. Every app/browser/monitor/verification command exited 0. The recorded owned app and PostgreSQL were stopped, exact ports independently refused connections, and only the recorded credential-bearing home was removed. No shared built app or other controller was changed. + +Inspect the compact JSON, screenshots, patch and exact executed helper copies. The only new browser adaptation adds the positive OS birth-proof prerequisite at the existing Stop boundary; UI routes, prompts, model, provider, streaming, tool, result, reload and database gates are unchanged. `live-browser.mjs` remains the immutable corrected original. `helpers/live-browser.guarded.mjs` is the exact executed private source. Public copies of the process monitor and birth-settlement checker accept an evidence-directory argument instead of the recorded private path; these path-only adaptations were not used for a second live claim. Exact executed private source hashes remain in `execution-receipts.json`, and `public-projections.json` identifies the adaptations. Execution command shapes and source SHAs are in `execution-receipts.json`. Use the repository's normal Node+TSX loader, normal copied launcher and compiled app; supply the qualification key privately through its existing environment variable. No key belongs in arguments or files. The provider configuration is the same approved one used for earlier live qualification. + +All retained raw files and decompressed trace entries passed an exact credential scan before this compact selection was copied. Full trace/video and controller logs remain in private task evidence. This is functional local acceptance, separate from paired ingestion performance and real-cloud Stop acceptance, with no deployed-production claim. No new provider/model/configuration or retry loop was used. + +## Retained progressive streaming evidence and verification correction + +The original accepted `visibleDelta` source52 happened during `workspace_finalizing`; it proves pre-terminal visibility only. A read-only review of the immutable full trace found earlier progressive content which the old selector excluded. The two unmodified screencast JPEGs now included show a short partial phrase at00:11:58.573Z and partially completed numbered sentences at00:11:59.142Z. DOM after@call@120 at00:11:58.580Z contains the partial interstitial ending in `1`, matching ordinary source29/30. Later ordinary source31..52 were emitted/committed at00:11:59.037Z through00:12:00.230Z; accepted result was committed at00:12:02.610Z. The recorded UI live-runs read at00:11:58.443Z reported native/running, but omitted nativePhase. This supports visible progressive content before later deltas/result, without inventing a literal provider-active phase predicate. `retained-progressive-streaming-proof.json` supplies original trace hash, resource names, frame IDs, source excerpts, and explicit limits. The nearest first JPEG precedes that DOM snapshot by about7ms and therefore shows only the shorter source29 prefix; it is not claimed to contain the entire slightly later DOM text. + +The old selector returned an empty array atcall120 because the live interstitial had no settled-agent-bubble ancestor. The future helper now scopes to the owned issue/thread, includes normal phase-interstitial/live-text branches, and requires an open started ordinary item plus a fresh running full GET, rejecting known finalization, completed items, accepted results and terminal state. Five offline real-browser DOM/source tests and the existing nine controlled source seams passed. These future helper changes were not executed in the recorded live journey. Original executed e6e/d719 helper copies and the original22-file bundle remain unchanged; the future source and regression receipt are labelled separately in this augmented bundle. No SUT, model, prompt, provider, app or cloud resource was changed or restarted. + +## Public packaging + +Resource-specific absolute checkout/evidence/temporary-home paths were replaced with descriptive references. `public-projections.json` records original and public hashes for each projection. Revisions, event identities, times, counts, hashes, cleanup outcomes and limits are unchanged. The launcher retains its already published generic macOS temporary-directory prefix, not a resource-specific private path. The two argument-adapted monitor/checker copies were syntax checked only; original executed hashes remain distinct. Run the future offline verification explicitly with `node --test scripts/native-ingestion/live-browser-streaming-check.mjs`; it is an opt-in Node/Playwright check outside the current default Vitest projects. The original22-file proof bundle remains unchanged. diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/after-live-stop.png b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/after-live-stop.png new file mode 100644 index 00000000000..a0afac193b6 Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/after-live-stop.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/artifact-secret-scan.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/artifact-secret-scan.json new file mode 100644 index 00000000000..98c792774f1 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/artifact-secret-scan.json @@ -0,0 +1,7 @@ +{ + "checkedAt": "2026-11-04T00:16:43.513320+00:00", + "exactQualificationCredentialPresent": false, + "scannedEntries": 2050, + "matchedFiles": [], + "scope": "All retained files including compact public selection and decompressed trace entries" +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/augmented-artifact-scan.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/augmented-artifact-scan.json new file mode 100644 index 00000000000..877b3a94196 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/augmented-artifact-scan.json @@ -0,0 +1,10 @@ +{ + "passed": true, + "checkedAt": "2026-11-04T00:47:02.146764+00:00", + "scope": "Copied compact proof after public path projections; original full-trace/decompressed scan retained separately", + "files": 32, + "exactQualificationCredentialMatches": [], + "resourceSpecificAbsolutePrivatePaths": [], + "privateHomeOrCredentialManagerMetadataPatterns": [], + "intentionalGenericPlatformPrefix": "The unchanged public launcher mkdtemp macOS prefix is generic, not a resource-specific private path" +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/before-live-stop.png b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/before-live-stop.png new file mode 100644 index 00000000000..fae7246a85d Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/before-live-stop.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/build-ready.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/build-ready.json new file mode 100644 index 00000000000..8ca1fa98d95 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/build-ready.json @@ -0,0 +1,59 @@ +{ + "status": "ready", + "builtAt": "2026-11-04T00:06:06.855Z", + "appRoot": "", + "sourceRevision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "compiledBuildStamp": { + "commit": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da" + }, + "runnerRustTree": "eac015472eb88b49aa92e6f30def348aa34b4911", + "cleanSource": true, + "independentLockedDependencies": true, + "normalProductionBuild": true, + "oldRustTargetReused": false, + "platform": { + "os": "darwin", + "architecture": "arm64", + "node": "v26.4.0", + "rustc": "rustc 1.97.1 (8bab26f4f 2026-07-14) (Homebrew)" + }, + "hashes": { + "server": "7db3ab75d39e5d6a467f961a267991cc55781439572cc37e45b4599908d1e23b", + "ui": "847e43ecc2469a5b54dcf89e0be4a71ba750f9d123e9d893ed90a3b3f693959e", + "macRunner": "3e9acd73d2fb380d3d95e35de0e6ed78232b1be5dcbb3a96c76ca0d8b8c951b2", + "vendorMacRunner": "3e9acd73d2fb380d3d95e35de0e6ed78232b1be5dcbb3a96c76ca0d8b8c951b2" + }, + "macRunnerBuildMetadata": { + "binaryContractVersion": 2, + "binaryName": "paperclip-runnerd", + "durableSessionCapabilities": [ + "unlimited_runtime", + "connection_lease_renewal" + ], + "externalProviderCapabilities": [ + "openai_dot_mcp", + "grok_bot_mcp", + "muse_external_v1" + ], + "harnessDriverVersion": 1, + "nativeExecutionVersion": 1, + "packageName": "@paperclipai/paperclip-runner", + "packageVersion": "0.0.0", + "providerPacks": [], + "prp": { + "maximumVersion": 2, + "minimumVersion": 1, + "name": "paperclip.runner" + }, + "prpTransportModes": [ + "dial_ws_loopback", + "dial_wss", + "listen_ws" + ], + "schema": "paperclip-runner/runnerd-build-metadata/v1" + }, + "readOnlyConsumers": [ + "delivery_evidence" + ], + "loader": "/server/node_modules/tsx/dist/loader.mjs" +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/cleanup.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/cleanup.json new file mode 100644 index 00000000000..ef47416385f --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/cleanup.json @@ -0,0 +1,17 @@ +{ + "startedAt": "2026-11-04T00:13:57.940Z", + "appPid": 58087, + "postgresPid": 68248, + "httpPort": 3104, + "postgresPort": 54333, + "removedOwnedDataDirs": [ + "" + ], + "errors": [], + "appAbsent": true, + "postgresAbsent": true, + "appPortClosed": true, + "postgresPortClosed": true, + "passed": true, + "finishedAt": "2026-11-04T00:13:59.945Z" +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/execution-receipts.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/execution-receipts.json new file mode 100644 index 00000000000..c005baebb13 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/execution-receipts.json @@ -0,0 +1,29 @@ +{ + "revision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "appExitCode": 0, + "browserExitCode": 0, + "processMonitorExitCode": 0, + "independentDbVerifierExitCode": 0, + "independentBirthSettlementExitCode": 0, + "cleanupExitCode": 0, + "sourceAndArtifactsReadOnly": true, + "helperSha256": { + "independent-lifecycle-process-proof.helper.mjs": "816b30b6686a4626f72b61b64452518f6ac87b11b5338e61c1db00baa3c3ba17", + "live-launch.mjs": "67b5c1f367be4451ee70b969f9cf9ad10f5cb48ba4cf9ff0ea51f6f4d39b0ebb", + "verify-database.mjs": "d3bec80c1886792e3d2e92489a1e4aa5e79ad55947218f03fb6cd9a5762fd22b", + "cleanup.helper.mjs": "20685c6246bf2ba2194d9e43fee9526a64c32c03a757ca9ff2c8c3d0cd4a5fb9", + "verify-birth-settlement.mjs": "ce6a5aca9a0fdeef21c0e5b785d2d3ac643da62c992465f8b9b4e4c41b947cef", + "process-monitor.mjs": "4f003d4db2ce70006028265735ecb10151710fccb8b6c701837ba1a21b150ab0", + "live-browser.guarded.mjs": "d7196cc7cf021252c386228cd1dc0ce59bde213fd464dc898a2f22155ff4b37b", + "live-browser.mjs": "e6e6af5670e8da496a66f420ad32bf5fe7df208e421d1254392df37106a21848" + }, + "normalCommands": [ + "node --import APP/server/node_modules/tsx/dist/loader.mjs live-launch.mjs APP 914129afb141d81c0d6d8c5f0ea11c20bd69a3da EVIDENCE", + "node process-monitor.mjs", + "node --import APP/server/node_modules/tsx/dist/loader.mjs live-browser.guarded.mjs EVIDENCE/live-launch.json", + "node independent-lifecycle-process-proof.helper.mjs", + "node verify-birth-settlement.mjs", + "node cleanup.helper.mjs" + ], + "adaptation": "Root-authorized private Stop boundary waits for same-run positive exact OS sleep60 birth identity before ordinary UI Stop; original corrected helper remains immutable; no provider/model/prompt/app changes" +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/future-helper-review-regressions.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/future-helper-review-regressions.json new file mode 100644 index 00000000000..ae72ddb93ce --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/future-helper-review-regressions.json @@ -0,0 +1,60 @@ +{ + "passed": true, + "scope": "Controlled helper contract/observation seams extracted from actual archived and changed source; no app, browser, provider, database or network is started.", + "historicalLiveProof": "Historical B9 functional proof; exact before-helper hashes retained below", + "historicalProofUsesOriginalHelpers": true, + "files": { + "launcher": { + "beforeSha256": "353a1b1cf2ee38551349617e91c95cdfb4d0021abf55be62e7058319b8e5b6bd", + "afterSha256": "67b5c1f367be4451ee70b969f9cf9ad10f5cb48ba4cf9ff0ea51f6f4d39b0ebb" + }, + "browser": { + "beforeSha256": "cf34b0f7bb8a4f84786a38f1c90b750e7a336b126e6e2bc25647095c1eac9fab", + "afterSha256": "5ff4cd9b187829e706ff770b96a110f654b087c92a5a97c8f4ce0a1319dcf2be" + } + }, + "cases": [ + { + "name": "original launcher rejects ordinary-checkout contract", + "outcome": "expected failure reproduced" + }, + { + "name": "fixed launcher forwards linkedWorktree=false", + "outcome": "passed" + }, + { + "name": "fixed launcher forwards linkedWorktree=true", + "outcome": "passed" + }, + { + "name": "fixed launcher forwards the prepared instance identity", + "outcome": "passed" + }, + { + "name": "original observer rejects a run settling immediately after successful active poll", + "outcome": "expected false negative reproduced" + }, + { + "name": "fixed observer retains exact successful active-stream observation across terminal race", + "outcome": "passed", + "calls": [ + "task-runs", + "visible-text", + "durable-events", + "status-after-visible" + ] + }, + { + "name": "terminal status after visibility still rejects streaming proof", + "outcome": "expected rejection passed" + }, + { + "name": "missing visible matching text still rejects streaming proof", + "outcome": "expected rejection passed" + }, + { + "name": "reasoning-only event still rejects ordinary-assistant proof", + "outcome": "expected rejection passed" + } + ] +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/future-helper-streaming-regressions.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/future-helper-streaming-regressions.json new file mode 100644 index 00000000000..5491c958bb4 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/future-helper-streaming-regressions.json @@ -0,0 +1,18 @@ +{ + "passed": true, + "scope": "Offline verification-only, network aborted, no app/provider/database/resources", + "command": "node --test scripts/native-ingestion/live-browser-streaming-check.mjs", + "exit": 0, + "testsPassed": 5, + "checkedAt": "2026-11-04T00:47:02.143566+00:00", + "helperSha256": "5ff4cd9b187829e706ff770b96a110f654b087c92a5a97c8f4ce0a1319dcf2be", + "testSha256": "3f26b2df27f9f40e5f941706a38623cf391fe92bb4091f903745a17d963e315c", + "cases": [ + "capturedpartialDOMoldselectormisses/newobserves", + "live-tail/settled compatibility", + "hidden/human/otherthread rejection", + "openitem/freshrunning positive/no closedphaseallowlist", + "finalizing/accepted/terminal/closeditem rejection" + ], + "renamedOptInCommandVerified": true +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helper-source-hashes.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helper-source-hashes.json new file mode 100644 index 00000000000..3d534c923f2 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helper-source-hashes.json @@ -0,0 +1,10 @@ +{ + "revision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "helperSha256": { + "live-launch.mjs": "67b5c1f367be4451ee70b969f9cf9ad10f5cb48ba4cf9ff0ea51f6f4d39b0ebb", + "verify-database.mjs": "d3bec80c1886792e3d2e92489a1e4aa5e79ad55947218f03fb6cd9a5762fd22b", + "live-browser.mjs": "e6e6af5670e8da496a66f420ad32bf5fe7df208e421d1254392df37106a21848", + "live-browser.guarded.mjs": "d7196cc7cf021252c386228cd1dc0ce59bde213fd464dc898a2f22155ff4b37b" + }, + "adaptation": "Root-authorized private Stop-boundary wait for positive owned live sleep60 birth identity; all UI/model/workload/persistence gates unchanged" +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser-streaming-check.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser-streaming-check.mjs new file mode 100644 index 00000000000..62af248b0ef --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser-streaming-check.mjs @@ -0,0 +1,98 @@ +import assert from 'node:assert/strict'; +import { before, after, test } from 'node:test'; +import { readFile } from 'node:fs/promises'; +import { createRequire } from 'node:module'; + +// Exact concrete native interstitial from retained914 trace after@call@120. +// Original trace SHA256: a6337930fdeeca5aca127bd7c028ed5cd92e754412e2a6157a8ac3678776f5bb +// No server, provider or network is used; only the real browser DOM selector runs. +const capturedInterstitial = ["DIV", {"class": "min-w-0 px-1 text-sm text-foreground/90", "data-testid": "task-chat-phase-interstitial", "data-conversation-text": "progress"}, ["DIV", {"class": "paperclip-markdown prose prose-sm min-w-0 max-w-full break-words overflow-hidden", "style": "overflow-wrap: anywhere; word-break: break-word;"}, ["P", {"style": "overflow-wrap: anywhere; word-break: break-word;"}, "The progress message was sent and ", ["CODE", {"style": "overflow-wrap: anywhere; word-break: break-word;"}, "list_task_documents"], " returned 0 documents on this task."], "\n", ["P", {"style": "overflow-wrap: anywhere; word-break: break-word;"}, "1"]]]; +const source = await readFile(new URL('./live-browser.mjs', import.meta.url), 'utf8'); +const require = createRequire(new URL('../../package.json', import.meta.url)); +const { chromium } = require('@playwright/test'); +const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor; +let browser, page; +before(async () => { + browser = await chromium.launch({ headless: true }); + page = await browser.newPage(); + await page.route('**/*', route => route.abort()); +}); +after(async () => { await browser?.close(); }); +const escape = value => String(value).replaceAll('&', '&').replaceAll('<', '<').replaceAll('"', '"'); +function html(node) { + if (typeof node === 'string') return escape(node); + if (!Array.isArray(node) || typeof node[0] !== 'string') return ''; + return '<' + node[0] + Object.entries(node[1]).map(([name, value]) => ' ' + name + '="' + escape(value) + '"').join('') + + '>' + node.slice(2).map(html).join('') + ''; +} +const owned = content => '
' + content + '
'; +const observerStart = source.indexOf('const visibleAgentText = '); +const observerEnd = source.indexOf('const normalizeVisible = ', observerStart); +assert.ok(observerStart >= 0 && observerEnd > observerStart); +const readVisible = new AsyncFunction('page', source.slice(observerStart, observerEnd) + '\nreturn visibleAgentText();'); + +test('captured partial native interstitial is observed while the original selector misses it', async () => { + await page.setContent(owned('
' + + '
' + html(capturedInterstitial) + '
')); + const oldSelector = '[data-conversation-message="agent"] [data-conversation-text]:visible, [data-testid="task-chat-live-self-talk"]:visible'; + assert.deepEqual(await page.locator(oldSelector).allInnerTexts(), []); + assert.ok((await readVisible(page)).includes('The progress message was sent and')); + assert.ok((await readVisible(page)).endsWith('1')); +}); + +test('normal live-tail text and settled agent messages remain observable', async () => { + await page.setContent(owned('
Live tail partial
' + + '
Settled result
')); + assert.equal(await readVisible(page), 'Live tail partial\nSettled result'); +}); + +test('hidden output, human prompts and a different thread cannot prove owned live text', async () => { + await page.setContent(owned('
' + html(capturedInterstitial) + '
' + + '
The progress message was sent and
') + + ''); + assert.equal(await readVisible(page), ''); +}); + +async function observation({ phase = 'observed', status = 'running', finishedAt = null, + accepted = false, completedItem = false, startedItem = true, terminal = false, kind = 'agentMessage' } = {}) { + const start = source.indexOf(' const streaming = await poll('); + const end = source.indexOf(' await page.screenshot(', start); + const normalizer = source.split('\n').find(line => line.startsWith('const normalizeVisible = ')); + const summaryStart = source.indexOf('const runSummary = '), summaryEnd = source.indexOf('async function taskRuns(', summaryStart); + assert.ok(start >= 0 && end > start && normalizer && summaryStart >= 0 && summaryEnd > summaryStart); + const execute = new AsyncFunction('poll', 'taskRuns', 'visibleAgentText', 'eventsFor', 'api', 'created', 'journey', 'expect', + normalizer + '\n' + source.slice(summaryStart, summaryEnd) + '\n' + source.slice(start, end) + '\nreturn journey;'); + const active = { id: 'owned-run', status: 'running', runtimeMode: 'native' }; + const identity = { runId: active.id, sourceInstanceId: 'source-1', turnId: 'turn-1', itemId: 'item-1' }; + const row = event => ({ eventType: event.eventType, payload: { prpEvent: { ...identity, ...event } } }); + const events = [ + ...(startedItem ? [row({ eventType: 'item.started', sourceEventId: 'started' })] : []), + row({ eventType: 'item.delta', sourceEventId: 'delta-29', sourceSeq: 29, payload: { kind, text: 'The progress message was sent and' } }), + ...(completedItem ? [row({ eventType: 'item.completed' })] : []), + ...(accepted ? [row({ eventType: 'run.result.accepted' })] : []), + ...(terminal ? [row({ eventType: 'run.terminal' })] : []), + ]; + return execute(async (_label, load, accept) => { + const value = await load(); + if (!accept(value)) throw new Error('streaming_observation_not_proven'); + return value; + }, async () => [active], async () => 'The progress message was sent and', async () => events, + async () => ({ ...active, status, nativePhase: phase, finishedAt }), { issueId: 'owned-issue' }, {}, + value => ({ toBe(expected) { assert.equal(value, expected); } })); +} + +test('fresh running observation with a still-open ordinary item proves streaming', async () => { + const value = await observation(); + assert.equal(value.visibleBeforeTerminal, true); + assert.equal(value.visibleDelta.sourceEventId, 'delta-29'); + assert.equal(value.streamingRunStatuses[0].nativePhase, 'observed'); + // Native phases are not a closed verification allowlist. + assert.equal((await observation({ phase: 'another_nonfinal_phase' })).visibleBeforeTerminal, true); +}); + +test('known finalization, accepted results, terminal state and closed items are rejected', async () => { + for (const options of [{ phase: 'workspace_finalizing' }, { accepted: true }, { terminal: true }, + { completedItem: true }, { startedItem: false }, { status: 'succeeded' }, { finishedAt: 'fixture-time' }, { kind: 'reasoning' }]) { + await assert.rejects(observation(options), /streaming_observation_not_proven/); + } +}); diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser.future.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser.future.mjs new file mode 100644 index 00000000000..693ca92ff88 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser.future.mjs @@ -0,0 +1,183 @@ +import { createRequire } from "node:module"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { pathToFileURL } from "node:url"; + +// Private acceptance helper. Configuration contains only an owned loopback +// origin, company/agent IDs and evidence paths; never provider credentials. +const config = JSON.parse(await readFile(process.argv[2], "utf8")); +const origin = new URL(config.origin); +if (!['127.0.0.1', 'localhost', '[::1]'].includes(origin.hostname)) throw new Error("Owned loopback application required"); +const require = createRequire(join(config.repository, "package.json")); +const { chromium, expect } = require("@playwright/test"); +const { createTaskThroughUi } = await import(pathToFileURL(join(config.repository, "tests/runner-e2e/user-actions.ts")).href); +await mkdir(config.evidenceDir, { recursive: true }); +const evidence = { startedAt: new Date().toISOString(), config, journeys: [], status: "running" }; +const save = () => writeFile(join(config.evidenceDir, "live-provider-result.json"), JSON.stringify(evidence, null, 2)); +const api = async route => { + const response = await fetch(new URL(route, origin)); + if (!response.ok) throw new Error(`Read ${route}: HTTP ${response.status}`); + return response.json(); +}; +const pause = ms => new Promise(resolve => setTimeout(resolve, ms)); +async function poll(label, load, accept, deadlineMs = 180_000) { + const deadline = Date.now() + deadlineMs; + while (true) { + const value = await load(); + if (accept(value)) return value; + if (Date.now() >= deadline) throw new Error(`Timed out waiting for ${label}`); + await pause(250); + } +} +async function eventsFor(runId) { + const result = []; + while (true) { + const page = await api(`/api/heartbeat-runs/${runId}/events?afterSeq=${result.at(-1)?.seq ?? 0}&limit=1000`); + result.push(...page); + if (page.length < 1000) return result; + } +} +const runSummary = run => ({ id: run.id, status: run.status, runtimeMode: run.runtimeMode, + driverKind: run.driverKind, providerSessionId: run.providerSessionId ?? run.sessionIdAfter ?? run.runnerProfileJson?.sessionCheckpoint?.providerSessionId, + nativeSessionId: run.nativeSessionId, + startedAt: run.startedAt, finishedAt: run.finishedAt, error: run.error, + nativePhase: run.nativePhase, provider: run.runnerProfileJson?.nativeExecutionInput?.provider ?? run.runnerProfileJson?.provider, + cancellation: run.resultJson?.nativeCancellation }); +async function taskRuns(issueId) { + const rows = await api(`/api/companies/${config.company.id}/heartbeat-runs?agentId=${config.agent.id}&limit=100`); + const details = await Promise.all(rows.map(run => api(`/api/heartbeat-runs/${run.id}`))); + return details.filter(run => run.nativeIssueId === issueId || run.contextSnapshot?.issueId === issueId); +} +const browser = await chromium.launch({ headless: true }); +const context = await browser.newContext({ viewport: { width: 1440, height: 1000 }, + recordVideo: { dir: join(config.evidenceDir, "video"), size: { width: 1440, height: 1000 } } }); +await context.tracing.start({ screenshots: true, snapshots: true }); +const page = await context.newPage(); +const visibleAgentText = async () => (await page.getByTestId("issue-detail-content") + .getByTestId("task-chat-thread") + .locator('[data-conversation-message="agent"] [data-conversation-text]:visible, [data-testid="task-chat-phase-interstitial"]:visible, [data-testid="task-chat-live-text"]:visible, [data-testid="task-chat-live-self-talk"]:visible') + .allInnerTexts()).join("\n"); +const normalizeVisible = value => String(value).replace(/[*_`#]/g, "").replace(/\s+/g, " ").trim(); +let activeIssueId; +try { + const health = await api("/api/health"); + const agent = await api(`/api/agents/${config.agent.id}`); + if (agent.companyId !== config.company.id || agent.adapterType !== "paperclip_runner" || agent.adapterConfig?.provider !== "opencode") + throw new Error("Configured agent is not the owned native OpenCode agent"); + evidence.health = { status: health.status, gitCommit: health.gitCommit ?? health.commit ?? health.version }; + evidence.provider = { adapterType: agent.adapterType, provider: agent.adapterConfig.provider, model: agent.adapterConfig.model }; + const marker = `LIVE-INGESTION-${Date.now()}`; + const journey = { kind: "stream-tool-result-reload", marker, status: "running" }; + evidence.journeys.push(journey); + const created = await createTaskThroughUi({ page, origin: origin.href, issuePrefix: config.company.issuePrefix, + agentName: config.agent.name, title: `Native live ingestion ${marker}`, workMode: "standard", requireExplicitTitle: true, + prompt: `Verify this isolated test task. First send a short progress message with ${marker}-START. Use the native Paperclip list_task_documents tool to inspect this task's documents and report the count. Then provide twelve numbered sentences about what this check proves. Include ${marker}-DONE in the final response and finish the task. Do not modify files or access unrelated tasks or services.` }); + activeIssueId = created.issueId; journey.issueId = created.issueId; + await page.goto(new URL(`/${config.company.issuePrefix}/issues/${created.issueId}`, origin).href); + const streaming = await poll("ordinary assistant delta visibly rendered before terminal", async () => { + const runs = await taskRuns(created.issueId); + const active = runs.find(run => run.status === "running"); + if (!active) return null; + const text = normalizeVisible(await visibleAgentText()); + const visibleAt = Date.now(); + const rows = await eventsFor(active.id); + const events = rows.map(row => row.payload?.prpEvent).filter(Boolean); + if (rows.some(row => ["run.result.accepted", "run.terminal"].includes(row.eventType) + || ["run.result.accepted", "run.terminal"].includes(row.payload?.prpEvent?.eventType))) return null; + const sameItem = (left, right) => left.runId === right.runId && left.sourceInstanceId === right.sourceInstanceId + && left.turnId === right.turnId && left.itemId === right.itemId; + const delta = events.find(event => event?.eventType === "item.delta" && event.runId === active.id + && event.payload.kind === "agentMessage" && normalizeVisible(event.payload.text).length >= 12 + && text.includes(normalizeVisible(event.payload.text)) && typeof event.itemId === "string" + && events.some(item => item.eventType === "item.started" && sameItem(item, event)) + && !events.some(item => ["item.completed", "item.failed", "item.cancelled"].includes(item.eventType) && sameItem(item, event))); + if (!delta) return null; + // Re-read status after observing the matching UI text. A completed run + // discovered on the next poll cannot establish streaming visibility. + const observedRun = await api(`/api/heartbeat-runs/${active.id}`); + if (observedRun.status !== "running" || observedRun.finishedAt || observedRun.nativePhase === "workspace_finalizing") return null; + return { runId: active.id, sourceEventId: delta.sourceEventId, sourceSeq: delta.sourceSeq, + emittedAt: delta.emittedAt, visibleText: text.slice(-1000), visibleAt, + run: runSummary(observedRun), statusObservedAt: Date.now() }; + }, Boolean); + journey.visibleDelta = streaming; + journey.firstVisibleAt = streaming.visibleAt; + journey.streamingRunStatuses = [streaming.run]; + journey.visibleBeforeTerminal = streaming.run.status === "running"; + expect(journey.visibleBeforeTerminal).toBe(true); + await page.screenshot({ path: join(config.evidenceDir, "live-streaming.png"), fullPage: true }); + const finished = await poll("completed native run", () => taskRuns(created.issueId), runs => + runs.length > 0 && runs.every(run => !["queued", "running", "scheduled_retry"].includes(run.status))); + journey.runs = finished.map(runSummary); + if (finished.length !== 1 || finished[0].status !== "succeeded" || finished[0].runtimeMode !== "native") + throw new Error(`Live run did not succeed: ${JSON.stringify(journey.runs)}`); + const rows = await eventsFor(finished[0].id); + const prp = rows.map(row => row.payload?.prpEvent).filter(Boolean); + journey.eventCounts = Object.fromEntries([...new Set(prp.map(event => event.eventType))].map(type => [type, prp.filter(event => event.eventType === type).length])); + journey.toolEvidence = prp.filter(event => event.eventType.startsWith("tool.") + && JSON.stringify(event.payload).includes("list_task_documents")) + .map(event => ({ eventType: event.eventType, sourceEventId: event.sourceEventId, sourceSeq: event.sourceSeq, + name: event.payload.name, operation: event.payload.operation, status: event.payload.status })); + expect(prp.some(event => event.eventType === "item.delta" && event.payload.kind === "agentMessage")).toBe(true); + expect(journey.toolEvidence.some(event => event.eventType === "tool.execution.completed")).toBe(true); + expect(prp.some(event => event.eventType === "run.result.accepted")).toBe(true); + await expect.poll(visibleAgentText, { timeout: 30_000 }).toContain(`${marker}-DONE`); + await page.screenshot({ path: join(config.evidenceDir, "live-result.png"), fullPage: true }); + await page.reload({ waitUntil: "domcontentloaded" }); + await expect.poll(visibleAgentText, { timeout: 30_000 }).toContain(`${marker}-DONE`); + await page.screenshot({ path: join(config.evidenceDir, "live-reloaded.png"), fullPage: true }); + journey.persistedAfterReload = true; journey.status = "passed"; + activeIssueId = undefined; + await save(); + + const stopped = { kind: "stop-live-provider", status: "running" }; + evidence.journeys.push(stopped); + const stopTask = await createTaskThroughUi({ page, origin: origin.href, issuePrefix: config.company.issuePrefix, + agentName: config.agent.name, title: `Native live Stop ${marker}`, workMode: "standard", requireExplicitTitle: true, + prompt: `This isolated task verifies the Stop control. First send a short progress message containing ${marker}-STOP-START. Then use your native shell tool to run exactly sleep 60 in the task workspace. Wait for the command and report only after it finishes. Do not modify files or use unrelated resources.` }); + activeIssueId = stopTask.issueId; stopped.issueId = stopTask.issueId; + await page.goto(new URL(`/${config.company.issuePrefix}/issues/${stopTask.issueId}`, origin).href); + const active = await poll("live provider tool execution", async () => { + const runs = await taskRuns(stopTask.issueId); + if (runs.length !== 1 || runs[0].status !== "running") return null; + const events = await eventsFor(runs[0].id); + const started = events.map(row => row.payload?.prpEvent).filter(event => event?.eventType === "tool.execution.started"); + stopped.startedTools = started.map(event => ({ name: event.payload.name, operation: event.payload.operation, + sourceEventId: event.sourceEventId })); + return started.some(event => /shell|bash|execute/i.test(String(event.payload.name) + String(event.payload.operation))) ? runs[0] : null; + }, Boolean); + stopped.activeRunId = active.id; + await page.screenshot({ path: join(config.evidenceDir, "before-live-stop.png"), fullPage: true }); + stopped.requestedAt = Date.now(); + await page.getByTestId("task-chat-composer-stop").last().click(); + const afterStop = await poll("native cancellation acknowledgement", async () => { + const runs = await taskRuns(stopTask.issueId); + return runs.length === 1 ? await api(`/api/heartbeat-runs/${runs[0].id}`) : null; + }, run => run?.status === "cancelled" && run.resultJson?.nativeCancellation?.dispatchState === "acknowledged"); + stopped.acknowledgedAt = Date.now(); stopped.run = runSummary(afterStop); + expect(afterStop.id).toBe(active.id); + await page.reload({ waitUntil: "domcontentloaded" }); + await pause(1000); + stopped.runIdsAfterReload = (await taskRuns(stopTask.issueId)).map(run => run.id); + expect(stopped.runIdsAfterReload).toEqual([active.id]); + await page.screenshot({ path: join(config.evidenceDir, "after-live-stop.png"), fullPage: true }); + stopped.status = "passed"; activeIssueId = undefined; + evidence.status = "passed"; +} catch (error) { + evidence.status = "failed"; + evidence.error = error instanceof Error ? { name: error.name, message: error.message, stack: error.stack } : String(error); + await page.screenshot({ path: join(config.evidenceDir, "live-failure.png"), fullPage: true }).catch(() => undefined); + if (activeIssueId) { + // Cleanup uses the same visible Stop action; failure remains in evidence. + const stop = page.getByTestId("task-chat-composer-stop").last(); + if (await stop.isVisible().catch(() => false)) await stop.click().catch(() => undefined); + } +} finally { + evidence.finishedAt = new Date().toISOString(); + await save(); + await context.tracing.stop({ path: join(config.evidenceDir, "live-provider-trace.zip") }); + await context.close(); await browser.close(); +} +console.log(JSON.stringify({ status: evidence.status, evidenceDir: config.evidenceDir, journeys: evidence.journeys.map(journey => + ({ kind: journey.kind, status: journey.status, issueId: journey.issueId, runs: journey.runs ?? journey.run })) })); +if (evidence.status !== "passed") process.exitCode = 1; diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser.guarded.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser.guarded.mjs new file mode 100644 index 00000000000..652ef9ff702 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser.guarded.mjs @@ -0,0 +1,187 @@ +import { createRequire } from "node:module"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { pathToFileURL } from "node:url"; + +// Private acceptance helper. Configuration contains only an owned loopback +// origin, company/agent IDs and evidence paths; never provider credentials. +const config = JSON.parse(await readFile(process.argv[2], "utf8")); +const origin = new URL(config.origin); +if (!['127.0.0.1', 'localhost', '[::1]'].includes(origin.hostname)) throw new Error("Owned loopback application required"); +const require = createRequire(join(config.repository, "package.json")); +const { chromium, expect } = require("@playwright/test"); +const { createTaskThroughUi } = await import(pathToFileURL(join(config.repository, "tests/runner-e2e/user-actions.ts")).href); +await mkdir(config.evidenceDir, { recursive: true }); +const evidence = { startedAt: new Date().toISOString(), config, journeys: [], status: "running" }; +const save = () => writeFile(join(config.evidenceDir, "live-provider-result.json"), JSON.stringify(evidence, null, 2)); +const api = async route => { + const response = await fetch(new URL(route, origin)); + if (!response.ok) throw new Error(`Read ${route}: HTTP ${response.status}`); + return response.json(); +}; +const pause = ms => new Promise(resolve => setTimeout(resolve, ms)); +async function poll(label, load, accept, deadlineMs = 180_000) { + const deadline = Date.now() + deadlineMs; + while (true) { + const value = await load(); + if (accept(value)) return value; + if (Date.now() >= deadline) throw new Error(`Timed out waiting for ${label}`); + await pause(250); + } +} +async function eventsFor(runId) { + const result = []; + while (true) { + const page = await api(`/api/heartbeat-runs/${runId}/events?afterSeq=${result.at(-1)?.seq ?? 0}&limit=1000`); + result.push(...page); + if (page.length < 1000) return result; + } +} +const runSummary = run => ({ id: run.id, status: run.status, runtimeMode: run.runtimeMode, + driverKind: run.driverKind, providerSessionId: run.providerSessionId ?? run.sessionIdAfter ?? run.runnerProfileJson?.sessionCheckpoint?.providerSessionId, + nativeSessionId: run.nativeSessionId, + startedAt: run.startedAt, finishedAt: run.finishedAt, error: run.error, + nativePhase: run.nativePhase, provider: run.runnerProfileJson?.nativeExecutionInput?.provider ?? run.runnerProfileJson?.provider, + cancellation: run.resultJson?.nativeCancellation }); +async function taskRuns(issueId) { + const rows = await api(`/api/companies/${config.company.id}/heartbeat-runs?agentId=${config.agent.id}&limit=100`); + const details = await Promise.all(rows.map(run => api(`/api/heartbeat-runs/${run.id}`))); + return details.filter(run => run.nativeIssueId === issueId || run.contextSnapshot?.issueId === issueId); +} +const browser = await chromium.launch({ headless: true }); +const context = await browser.newContext({ viewport: { width: 1440, height: 1000 }, + recordVideo: { dir: join(config.evidenceDir, "video"), size: { width: 1440, height: 1000 } } }); +await context.tracing.start({ screenshots: true, snapshots: true }); +const page = await context.newPage(); +const visibleAgentText = async () => (await page.locator('[data-conversation-message="agent"] [data-conversation-text]:visible, [data-testid="task-chat-live-self-talk"]:visible').allInnerTexts()).join("\n"); +const normalizeVisible = value => String(value).replace(/[*_`#]/g, "").replace(/\s+/g, " ").trim(); +let activeIssueId; +try { + const health = await api("/api/health"); + const agent = await api(`/api/agents/${config.agent.id}`); + if (agent.companyId !== config.company.id || agent.adapterType !== "paperclip_runner" || agent.adapterConfig?.provider !== "opencode") + throw new Error("Configured agent is not the owned native OpenCode agent"); + evidence.health = { status: health.status, gitCommit: health.gitCommit ?? health.commit ?? health.version }; + evidence.provider = { adapterType: agent.adapterType, provider: agent.adapterConfig.provider, model: agent.adapterConfig.model }; + const marker = `LIVE-INGESTION-${Date.now()}`; + const journey = { kind: "stream-tool-result-reload", marker, status: "running" }; + evidence.journeys.push(journey); + const created = await createTaskThroughUi({ page, origin: origin.href, issuePrefix: config.company.issuePrefix, + agentName: config.agent.name, title: `Native live ingestion ${marker}`, workMode: "standard", requireExplicitTitle: true, + prompt: `Verify this isolated test task. First send a short progress message with ${marker}-START. Use the native Paperclip list_task_documents tool to inspect this task's documents and report the count. Then provide twelve numbered sentences about what this check proves. Include ${marker}-DONE in the final response and finish the task. Do not modify files or access unrelated tasks or services.` }); + activeIssueId = created.issueId; journey.issueId = created.issueId; + await page.goto(new URL(`/${config.company.issuePrefix}/issues/${created.issueId}`, origin).href); + const streaming = await poll("ordinary assistant delta visibly rendered before terminal", async () => { + const runs = await taskRuns(created.issueId); + const active = runs.find(run => run.status === "running"); + if (!active) return null; + const text = normalizeVisible(await visibleAgentText()); + const visibleAt = Date.now(); + const rows = await eventsFor(active.id); + const delta = rows.map(row => row.payload?.prpEvent).find(event => event?.eventType === "item.delta" + && event.payload.kind === "agentMessage" && normalizeVisible(event.payload.text).length >= 12 + && text.includes(normalizeVisible(event.payload.text))); + if (!delta) return null; + // Re-read status after observing the matching UI text. A completed run + // discovered on the next poll cannot establish streaming visibility. + const observedRun = await api(`/api/heartbeat-runs/${active.id}`); + if (observedRun.status !== "running") return null; + return { runId: active.id, sourceEventId: delta.sourceEventId, sourceSeq: delta.sourceSeq, + emittedAt: delta.emittedAt, visibleText: text.slice(-1000), visibleAt, + run: runSummary(observedRun), statusObservedAt: Date.now() }; + }, Boolean); + journey.visibleDelta = streaming; + journey.firstVisibleAt = streaming.visibleAt; + journey.streamingRunStatuses = [streaming.run]; + journey.visibleBeforeTerminal = streaming.run.status === "running"; + expect(journey.visibleBeforeTerminal).toBe(true); + await page.screenshot({ path: join(config.evidenceDir, "live-streaming.png"), fullPage: true }); + const finished = await poll("completed native run", () => taskRuns(created.issueId), runs => + runs.length > 0 && runs.every(run => !["queued", "running", "scheduled_retry"].includes(run.status))); + journey.runs = finished.map(runSummary); + if (finished.length !== 1 || finished[0].status !== "succeeded" || finished[0].runtimeMode !== "native") + throw new Error(`Live run did not succeed: ${JSON.stringify(journey.runs)}`); + const rows = await eventsFor(finished[0].id); + const prp = rows.map(row => row.payload?.prpEvent).filter(Boolean); + journey.eventCounts = Object.fromEntries([...new Set(prp.map(event => event.eventType))].map(type => [type, prp.filter(event => event.eventType === type).length])); + journey.toolEvidence = prp.filter(event => event.eventType.startsWith("tool.") + && JSON.stringify(event.payload).includes("list_task_documents")) + .map(event => ({ eventType: event.eventType, sourceEventId: event.sourceEventId, sourceSeq: event.sourceSeq, + name: event.payload.name, operation: event.payload.operation, status: event.payload.status })); + expect(prp.some(event => event.eventType === "item.delta" && event.payload.kind === "agentMessage")).toBe(true); + expect(journey.toolEvidence.some(event => event.eventType === "tool.execution.completed")).toBe(true); + expect(prp.some(event => event.eventType === "run.result.accepted")).toBe(true); + await expect.poll(visibleAgentText, { timeout: 30_000 }).toContain(`${marker}-DONE`); + await page.screenshot({ path: join(config.evidenceDir, "live-result.png"), fullPage: true }); + await page.reload({ waitUntil: "domcontentloaded" }); + await expect.poll(visibleAgentText, { timeout: 30_000 }).toContain(`${marker}-DONE`); + await page.screenshot({ path: join(config.evidenceDir, "live-reloaded.png"), fullPage: true }); + journey.persistedAfterReload = true; journey.status = "passed"; + activeIssueId = undefined; + await save(); + + const stopped = { kind: "stop-live-provider", status: "running" }; + evidence.journeys.push(stopped); + const stopTask = await createTaskThroughUi({ page, origin: origin.href, issuePrefix: config.company.issuePrefix, + agentName: config.agent.name, title: `Native live Stop ${marker}`, workMode: "standard", requireExplicitTitle: true, + prompt: `This isolated task verifies the Stop control. First send a short progress message containing ${marker}-STOP-START. Then use your native shell tool to run exactly sleep 60 in the task workspace. Wait for the command and report only after it finishes. Do not modify files or use unrelated resources.` }); + activeIssueId = stopTask.issueId; stopped.issueId = stopTask.issueId; + await page.goto(new URL(`/${config.company.issuePrefix}/issues/${stopTask.issueId}`, origin).href); + const active = await poll("live provider tool execution", async () => { + const runs = await taskRuns(stopTask.issueId); + if (runs.length !== 1 || runs[0].status !== "running") return null; + const events = await eventsFor(runs[0].id); + const started = events.map(row => row.payload?.prpEvent).filter(event => event?.eventType === "tool.execution.started"); + stopped.startedTools = started.map(event => ({ name: event.payload.name, operation: event.payload.operation, + sourceEventId: event.sourceEventId })); + return started.some(event => /shell|bash|execute/i.test(String(event.payload.name) + String(event.payload.operation))) ? runs[0] : null; + }, Boolean); + stopped.activeRunId = active.id; + await page.screenshot({ path: join(config.evidenceDir, "before-live-stop.png"), fullPage: true }); + stopped.preStopProcessProof = await poll("owned sleep60 birth identity before UI Stop", async () => { + let observation; + try { observation = JSON.parse(await readFile(join(config.evidenceDir, "process-gate.json"), "utf8")); } + catch (error) { if (error.code === "ENOENT" || error instanceof SyntaxError) return null; throw error; } + const now = Date.now(); + const proof = observation.sleepCandidates.find(value => value.companyId === config.company.id && value.runId === active.id + && value.command.trim().match(/^(?:\/[\w./-]+\/)?sleep\s+60$/) + && now >= value.observedAtMs && now - value.observedAtMs <= 500 + && value.observedProcessAgeMs + now - value.observedAtMs < 20000 + && value.chain[0].pid === value.sleepPid && value.chain[0].birthTime === value.sleepBirthTime + && value.chain.at(-1).pid === value.runnerPid && value.chain.at(-1).executable === "paperclip-runnerd" + && value.chain.some(process => /verified-executable|opencode/.test(process.executable))); + return proof ? { ...proof, verifiedBeforeClickAtMs: Date.now() } : null; + }, Boolean, 20000); + stopped.requestedAt = Date.now(); + await page.getByTestId("task-chat-composer-stop").last().click(); + const afterStop = await poll("native cancellation acknowledgement", async () => { + const runs = await taskRuns(stopTask.issueId); + return runs.length === 1 ? await api(`/api/heartbeat-runs/${runs[0].id}`) : null; + }, run => run?.status === "cancelled" && run.resultJson?.nativeCancellation?.dispatchState === "acknowledged"); + stopped.acknowledgedAt = Date.now(); stopped.run = runSummary(afterStop); + expect(afterStop.id).toBe(active.id); + await page.reload({ waitUntil: "domcontentloaded" }); + await pause(1000); + stopped.runIdsAfterReload = (await taskRuns(stopTask.issueId)).map(run => run.id); + expect(stopped.runIdsAfterReload).toEqual([active.id]); + await page.screenshot({ path: join(config.evidenceDir, "after-live-stop.png"), fullPage: true }); + stopped.status = "passed"; activeIssueId = undefined; + evidence.status = "passed"; +} catch (error) { + evidence.status = "failed"; + evidence.error = error instanceof Error ? { name: error.name, message: error.message, stack: error.stack } : String(error); + await page.screenshot({ path: join(config.evidenceDir, "live-failure.png"), fullPage: true }).catch(() => undefined); + if (activeIssueId) { + // Cleanup uses the same visible Stop action; failure remains in evidence. + const stop = page.getByTestId("task-chat-composer-stop").last(); + if (await stop.isVisible().catch(() => false)) await stop.click().catch(() => undefined); + } +} finally { + evidence.finishedAt = new Date().toISOString(); + await save(); + await context.tracing.stop({ path: join(config.evidenceDir, "live-provider-trace.zip") }); + await context.close(); await browser.close(); +} +console.log(JSON.stringify({ status: evidence.status, evidenceDir: config.evidenceDir, journeys: evidence.journeys.map(journey => + ({ kind: journey.kind, status: journey.status, issueId: journey.issueId, runs: journey.runs ?? journey.run })) })); +if (evidence.status !== "passed") process.exitCode = 1; diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser.mjs new file mode 100644 index 00000000000..7bdaf9fe68d --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-browser.mjs @@ -0,0 +1,173 @@ +import { createRequire } from "node:module"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { pathToFileURL } from "node:url"; + +// Private acceptance helper. Configuration contains only an owned loopback +// origin, company/agent IDs and evidence paths; never provider credentials. +const config = JSON.parse(await readFile(process.argv[2], "utf8")); +const origin = new URL(config.origin); +if (!['127.0.0.1', 'localhost', '[::1]'].includes(origin.hostname)) throw new Error("Owned loopback application required"); +const require = createRequire(join(config.repository, "package.json")); +const { chromium, expect } = require("@playwright/test"); +const { createTaskThroughUi } = await import(pathToFileURL(join(config.repository, "tests/runner-e2e/user-actions.ts")).href); +await mkdir(config.evidenceDir, { recursive: true }); +const evidence = { startedAt: new Date().toISOString(), config, journeys: [], status: "running" }; +const save = () => writeFile(join(config.evidenceDir, "live-provider-result.json"), JSON.stringify(evidence, null, 2)); +const api = async route => { + const response = await fetch(new URL(route, origin)); + if (!response.ok) throw new Error(`Read ${route}: HTTP ${response.status}`); + return response.json(); +}; +const pause = ms => new Promise(resolve => setTimeout(resolve, ms)); +async function poll(label, load, accept, deadlineMs = 180_000) { + const deadline = Date.now() + deadlineMs; + while (true) { + const value = await load(); + if (accept(value)) return value; + if (Date.now() >= deadline) throw new Error(`Timed out waiting for ${label}`); + await pause(250); + } +} +async function eventsFor(runId) { + const result = []; + while (true) { + const page = await api(`/api/heartbeat-runs/${runId}/events?afterSeq=${result.at(-1)?.seq ?? 0}&limit=1000`); + result.push(...page); + if (page.length < 1000) return result; + } +} +const runSummary = run => ({ id: run.id, status: run.status, runtimeMode: run.runtimeMode, + driverKind: run.driverKind, providerSessionId: run.providerSessionId ?? run.sessionIdAfter ?? run.runnerProfileJson?.sessionCheckpoint?.providerSessionId, + nativeSessionId: run.nativeSessionId, + startedAt: run.startedAt, finishedAt: run.finishedAt, error: run.error, + nativePhase: run.nativePhase, provider: run.runnerProfileJson?.nativeExecutionInput?.provider ?? run.runnerProfileJson?.provider, + cancellation: run.resultJson?.nativeCancellation }); +async function taskRuns(issueId) { + const rows = await api(`/api/companies/${config.company.id}/heartbeat-runs?agentId=${config.agent.id}&limit=100`); + const details = await Promise.all(rows.map(run => api(`/api/heartbeat-runs/${run.id}`))); + return details.filter(run => run.nativeIssueId === issueId || run.contextSnapshot?.issueId === issueId); +} +const browser = await chromium.launch({ headless: true }); +const context = await browser.newContext({ viewport: { width: 1440, height: 1000 }, + recordVideo: { dir: join(config.evidenceDir, "video"), size: { width: 1440, height: 1000 } } }); +await context.tracing.start({ screenshots: true, snapshots: true }); +const page = await context.newPage(); +const visibleAgentText = async () => (await page.locator('[data-conversation-message="agent"] [data-conversation-text]:visible, [data-testid="task-chat-live-self-talk"]:visible').allInnerTexts()).join("\n"); +const normalizeVisible = value => String(value).replace(/[*_`#]/g, "").replace(/\s+/g, " ").trim(); +let activeIssueId; +try { + const health = await api("/api/health"); + const agent = await api(`/api/agents/${config.agent.id}`); + if (agent.companyId !== config.company.id || agent.adapterType !== "paperclip_runner" || agent.adapterConfig?.provider !== "opencode") + throw new Error("Configured agent is not the owned native OpenCode agent"); + evidence.health = { status: health.status, gitCommit: health.gitCommit ?? health.commit ?? health.version }; + evidence.provider = { adapterType: agent.adapterType, provider: agent.adapterConfig.provider, model: agent.adapterConfig.model }; + const marker = `LIVE-INGESTION-${Date.now()}`; + const journey = { kind: "stream-tool-result-reload", marker, status: "running" }; + evidence.journeys.push(journey); + const created = await createTaskThroughUi({ page, origin: origin.href, issuePrefix: config.company.issuePrefix, + agentName: config.agent.name, title: `Native live ingestion ${marker}`, workMode: "standard", requireExplicitTitle: true, + prompt: `Verify this isolated test task. First send a short progress message with ${marker}-START. Use the native Paperclip list_task_documents tool to inspect this task's documents and report the count. Then provide twelve numbered sentences about what this check proves. Include ${marker}-DONE in the final response and finish the task. Do not modify files or access unrelated tasks or services.` }); + activeIssueId = created.issueId; journey.issueId = created.issueId; + await page.goto(new URL(`/${config.company.issuePrefix}/issues/${created.issueId}`, origin).href); + const streaming = await poll("ordinary assistant delta visibly rendered before terminal", async () => { + const runs = await taskRuns(created.issueId); + const active = runs.find(run => run.status === "running"); + if (!active) return null; + const text = normalizeVisible(await visibleAgentText()); + const visibleAt = Date.now(); + const rows = await eventsFor(active.id); + const delta = rows.map(row => row.payload?.prpEvent).find(event => event?.eventType === "item.delta" + && event.payload.kind === "agentMessage" && normalizeVisible(event.payload.text).length >= 12 + && text.includes(normalizeVisible(event.payload.text))); + if (!delta) return null; + // Re-read status after observing the matching UI text. A completed run + // discovered on the next poll cannot establish streaming visibility. + const observedRun = await api(`/api/heartbeat-runs/${active.id}`); + if (observedRun.status !== "running") return null; + return { runId: active.id, sourceEventId: delta.sourceEventId, sourceSeq: delta.sourceSeq, + emittedAt: delta.emittedAt, visibleText: text.slice(-1000), visibleAt, + run: runSummary(observedRun), statusObservedAt: Date.now() }; + }, Boolean); + journey.visibleDelta = streaming; + journey.firstVisibleAt = streaming.visibleAt; + journey.streamingRunStatuses = [streaming.run]; + journey.visibleBeforeTerminal = streaming.run.status === "running"; + expect(journey.visibleBeforeTerminal).toBe(true); + await page.screenshot({ path: join(config.evidenceDir, "live-streaming.png"), fullPage: true }); + const finished = await poll("completed native run", () => taskRuns(created.issueId), runs => + runs.length > 0 && runs.every(run => !["queued", "running", "scheduled_retry"].includes(run.status))); + journey.runs = finished.map(runSummary); + if (finished.length !== 1 || finished[0].status !== "succeeded" || finished[0].runtimeMode !== "native") + throw new Error(`Live run did not succeed: ${JSON.stringify(journey.runs)}`); + const rows = await eventsFor(finished[0].id); + const prp = rows.map(row => row.payload?.prpEvent).filter(Boolean); + journey.eventCounts = Object.fromEntries([...new Set(prp.map(event => event.eventType))].map(type => [type, prp.filter(event => event.eventType === type).length])); + journey.toolEvidence = prp.filter(event => event.eventType.startsWith("tool.") + && JSON.stringify(event.payload).includes("list_task_documents")) + .map(event => ({ eventType: event.eventType, sourceEventId: event.sourceEventId, sourceSeq: event.sourceSeq, + name: event.payload.name, operation: event.payload.operation, status: event.payload.status })); + expect(prp.some(event => event.eventType === "item.delta" && event.payload.kind === "agentMessage")).toBe(true); + expect(journey.toolEvidence.some(event => event.eventType === "tool.execution.completed")).toBe(true); + expect(prp.some(event => event.eventType === "run.result.accepted")).toBe(true); + await expect.poll(visibleAgentText, { timeout: 30_000 }).toContain(`${marker}-DONE`); + await page.screenshot({ path: join(config.evidenceDir, "live-result.png"), fullPage: true }); + await page.reload({ waitUntil: "domcontentloaded" }); + await expect.poll(visibleAgentText, { timeout: 30_000 }).toContain(`${marker}-DONE`); + await page.screenshot({ path: join(config.evidenceDir, "live-reloaded.png"), fullPage: true }); + journey.persistedAfterReload = true; journey.status = "passed"; + activeIssueId = undefined; + await save(); + + const stopped = { kind: "stop-live-provider", status: "running" }; + evidence.journeys.push(stopped); + const stopTask = await createTaskThroughUi({ page, origin: origin.href, issuePrefix: config.company.issuePrefix, + agentName: config.agent.name, title: `Native live Stop ${marker}`, workMode: "standard", requireExplicitTitle: true, + prompt: `This isolated task verifies the Stop control. First send a short progress message containing ${marker}-STOP-START. Then use your native shell tool to run exactly sleep 60 in the task workspace. Wait for the command and report only after it finishes. Do not modify files or use unrelated resources.` }); + activeIssueId = stopTask.issueId; stopped.issueId = stopTask.issueId; + await page.goto(new URL(`/${config.company.issuePrefix}/issues/${stopTask.issueId}`, origin).href); + const active = await poll("live provider tool execution", async () => { + const runs = await taskRuns(stopTask.issueId); + if (runs.length !== 1 || runs[0].status !== "running") return null; + const events = await eventsFor(runs[0].id); + const started = events.map(row => row.payload?.prpEvent).filter(event => event?.eventType === "tool.execution.started"); + stopped.startedTools = started.map(event => ({ name: event.payload.name, operation: event.payload.operation, + sourceEventId: event.sourceEventId })); + return started.some(event => /shell|bash|execute/i.test(String(event.payload.name) + String(event.payload.operation))) ? runs[0] : null; + }, Boolean); + stopped.activeRunId = active.id; + await page.screenshot({ path: join(config.evidenceDir, "before-live-stop.png"), fullPage: true }); + stopped.requestedAt = Date.now(); + await page.getByTestId("task-chat-composer-stop").last().click(); + const afterStop = await poll("native cancellation acknowledgement", async () => { + const runs = await taskRuns(stopTask.issueId); + return runs.length === 1 ? await api(`/api/heartbeat-runs/${runs[0].id}`) : null; + }, run => run?.status === "cancelled" && run.resultJson?.nativeCancellation?.dispatchState === "acknowledged"); + stopped.acknowledgedAt = Date.now(); stopped.run = runSummary(afterStop); + expect(afterStop.id).toBe(active.id); + await page.reload({ waitUntil: "domcontentloaded" }); + await pause(1000); + stopped.runIdsAfterReload = (await taskRuns(stopTask.issueId)).map(run => run.id); + expect(stopped.runIdsAfterReload).toEqual([active.id]); + await page.screenshot({ path: join(config.evidenceDir, "after-live-stop.png"), fullPage: true }); + stopped.status = "passed"; activeIssueId = undefined; + evidence.status = "passed"; +} catch (error) { + evidence.status = "failed"; + evidence.error = error instanceof Error ? { name: error.name, message: error.message, stack: error.stack } : String(error); + await page.screenshot({ path: join(config.evidenceDir, "live-failure.png"), fullPage: true }).catch(() => undefined); + if (activeIssueId) { + // Cleanup uses the same visible Stop action; failure remains in evidence. + const stop = page.getByTestId("task-chat-composer-stop").last(); + if (await stop.isVisible().catch(() => false)) await stop.click().catch(() => undefined); + } +} finally { + evidence.finishedAt = new Date().toISOString(); + await save(); + await context.tracing.stop({ path: join(config.evidenceDir, "live-provider-trace.zip") }); + await context.close(); await browser.close(); +} +console.log(JSON.stringify({ status: evidence.status, evidenceDir: config.evidenceDir, journeys: evidence.journeys.map(journey => + ({ kind: journey.kind, status: journey.status, issueId: journey.issueId, runs: journey.runs ?? journey.run })) })); +if (evidence.status !== "passed") process.exitCode = 1; diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-launch.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-launch.mjs new file mode 100644 index 00000000000..9854bf070b4 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/live-launch.mjs @@ -0,0 +1,62 @@ +import { mkdtemp, mkdir, writeFile, readFile } from "node:fs/promises"; +import { join } from "node:path"; +import { pathToFileURL } from "node:url"; +import { createHash } from "node:crypto"; +import { execFileSync } from "node:child_process"; + +const repository = process.argv[2]; +const expectedRevision = process.argv[3]; +const evidenceDir = process.argv[4]; +if (!repository || !expectedRevision || !evidenceDir) throw new Error("repository, expected revision and evidence directory are required"); +const revision = execFileSync("git", ["rev-parse", "HEAD"], { cwd: repository, encoding: "utf8" }).trim(); +if (revision !== expectedRevision) throw new Error("Source revision changed before live acceptance"); +const buildStamp = JSON.parse(await readFile(join(repository, "server/dist/build-info.json"), "utf8")); +if (buildStamp.commit !== expectedRevision) throw new Error("Compiled server revision does not match live acceptance source"); +const credentialName = "PAPERCLIP_PI_QUALIFICATION_OPENROUTER_API_KEY"; +if (!process.env[credentialName]) throw new Error("Qualification credential is unavailable"); +const model = "openrouter/deepseek/deepseek-v4-flash-0731"; +await mkdir(evidenceDir, { recursive: true }); +const dataDir = await mkdtemp("/private/tmp/paperclip-native-live-data-"); +let server; +try { + const { prepareTestDriveEnvironment, bootstrapTestDrive } = await import(pathToFileURL(join(repository, "cli/src/commands/test-drive.ts")).href); + const options = { dataDir, companyName: "Native ingestion live acceptance", agentName: "Live ingestion agent", + harness: "opencode", model, apiKeyEnv: credentialName, browser: false }; + const { linkedWorktree } = await prepareTestDriveEnvironment(options, repository); + process.env.PAPERCLIP_UI_DEV_MIDDLEWARE = "false"; + process.env.NODE_ENV = "production"; + process.env.SERVE_UI = "true"; + const { onboard } = await import(pathToFileURL(join(repository, "cli/src/commands/onboard.ts")).href); + await onboard({ config: process.env.PAPERCLIP_CONFIG, yes: true, bind: "loopback", invokedByRun: true, installService: false }); + const { doctor } = await import(pathToFileURL(join(repository, "cli/src/commands/doctor.ts")).href); + const checked = await doctor({ config: process.env.PAPERCLIP_CONFIG, repair: true, yes: true }); + if (checked.failed > 0) throw new Error(`Doctor failed ${checked.failed} checks`); + const { startServer } = await import(pathToFileURL(join(repository, "server/dist/index.js")).href); + server = await startServer(); + const origin = `http://${server.host}:${server.listenPort}`; + const { PaperclipApiClient } = await import(pathToFileURL(join(repository, "cli/src/client/http.ts")).href); + const api = new PaperclipApiClient({ apiBase: origin }); + const fixture = await bootstrapTestDrive({ api, options, linkedWorktree, + instanceId: process.env.PAPERCLIP_INSTANCE_ID ?? "default", env: process.env }); + if (fixture.reused || !fixture.agent) throw new Error("Fresh live-provider fixture was not created"); + await api.patch("/api/instance/settings/experimental", { enableNativeRunner: true }); + const agent = await api.patch(`/api/agents/${fixture.agent.id}`, { + adapterType: "paperclip_runner", + adapterConfig: { ...fixture.agent.adapterConfig, provider: "opencode", model, lifecycleMode: "per_turn", + opencodePermissionMode: "allow", timeoutSec: 180 }, + runtimeConfig: { heartbeat: { enabled: false } }, + }); + if (agent?.adapterType !== "paperclip_runner" || agent.adapterConfig?.provider !== "opencode") throw new Error("Native agent setup did not persist"); + const config = { repository, origin, company: { id: fixture.company.id, issuePrefix: fixture.company.issuePrefix }, + agent: { id: agent.id, name: agent.name }, evidenceDir, revision, dataDir, pid: process.pid, + build: { commit: buildStamp.commit, serverSha256: createHash("sha256").update(await readFile(join(repository, "server/dist/index.js"))).digest("hex"), + uiSha256: createHash("sha256").update(await readFile(join(repository, "ui/dist/index.html"))).digest("hex") }, + platform: { node: process.version, os: process.platform, architecture: process.arch }, provider: { kind: "opencode", model } }; + await writeFile(join(evidenceDir, "live-launch.json"), JSON.stringify(config, null, 2)); + console.log(JSON.stringify({ ready: true, origin, revision, evidenceDir, configPath: join(evidenceDir, "live-launch.json"), pid: process.pid })); +} catch (error) { + await server?.shutdown?.("SIGTERM"); + const message = error instanceof Error ? error.stack ?? error.message : String(error); + console.error(message.replaceAll(process.env[credentialName] ?? "\0private-credential-unavailable", "***REDACTED***")); + process.exitCode = 1; +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/process-monitor.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/process-monitor.mjs new file mode 100644 index 00000000000..9ebd048da62 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/process-monitor.mjs @@ -0,0 +1,49 @@ +import {readFile,writeFile} from 'node:fs/promises'; +import {execFileSync} from 'node:child_process'; +import path from 'node:path'; +const directory=process.argv[2]; +if (!directory) throw new Error('Pass the owned evidence directory'); +const launch=JSON.parse(await readFile(path.join(directory,'live-launch.json'),'utf8')); +const observed=new Map(),runSnapshots=[],sleepHistory=[]; +const start=Date.now();let terminalAt,cachedRuns=[],lastApi=0,lastRunSignature=''; +const elapsedSeconds=text=>{const parts=text.split('-');const days=parts.length===2?Number(parts.shift()):0;return days*86400+parts[0].split(':').map(Number).reduce((total,value)=>total*60+value,0);}; +while(Date.now()-start<300000){ + const at=Date.now(); + if(at-lastApi>=250){ + const response=await fetch(new URL(`/api/companies/${launch.company.id}/heartbeat-runs?agentId=${launch.agent.id}&limit=100`,launch.origin)); + if(!response.ok)throw new Error('Owned run snapshot read failed');cachedRuns=await response.json();lastApi=at; + } + const output=execFileSync('ps',['-axo','pid=,ppid=,pgid=,stat=,lstart=,comm='],{encoding:'utf8'}); + const processes=output.split('\n').flatMap(line=>{const m=line.match(/^\s*(\d+)\s+(\d+)\s+(\d+)\s+(\S+)\s+(\S+\s+\S+\s+\d+\s+\d\d:\d\d:\d\d\s+\d{4})\s+(.+)$/);return m?[{pid:Number(m[1]),parentPid:Number(m[2]),processGroupId:Number(m[3]),state:m[4],birthTime:m[5],executable:path.basename(m[6])}]:[];}); + const byPid=new Map(processes.map(proc=>[proc.pid,proc])); + const roots=new Set([launch.pid,...cachedRuns.map(row=>row.processPid).filter(pid=>Number.isInteger(pid)&&pid>1)]); + for(let pass=0;pass<10;pass++)for(const proc of processes)if(roots.has(proc.parentPid))roots.add(proc.pid); + for(const proc of processes)if(roots.has(proc.pid)){const previous=observed.get(proc.pid);observed.set(proc.pid,{...proc,firstSeen:previous?.firstSeen??at,lastSeen:at});} + const signature=JSON.stringify(cachedRuns.map(run=>({id:run.id,status:run.status,processPid:run.processPid??null,processGroupId:run.processGroupId??null}))); + if(signature!==lastRunSignature){runSnapshots.push({atMs:at,runs:JSON.parse(signature)});lastRunSignature=signature;} + const sleepCandidates=[]; + for(const run of cachedRuns.filter(run=>run.status==='running'&&Number.isInteger(run.processPid))){ + const runner=byPid.get(run.processPid);if(runner?.executable!=='paperclip-runnerd')continue; + for(const proc of processes.filter(proc=>proc.executable==='sleep'&&!proc.state.startsWith('Z'))){ + const chain=[];let cursor=proc; + for(let depth=0;cursor&&depth<10;depth++){chain.push(cursor);if(cursor.pid===runner.pid)break;cursor=byPid.get(cursor.parentPid);} + if(chain.at(-1)?.pid!==runner.pid||!chain.some(row=>/verified-executable|opencode/.test(row.executable)))continue; + let detail='';try{detail=execFileSync('ps',['-p',String(proc.pid),'-o','etime=,args='],{encoding:'utf8'}).trim();}catch{continue;} + const m=detail.match(/^(\S+)\s+((?:\/[\w./-]+\/)?sleep\s+60)$/);if(!m)continue; + const ageMs=elapsedSeconds(m[1])*1000;if(ageMs>=20000)continue; + const proof={observedAtMs:Date.now(),companyId:launch.company.id,runId:run.id,runnerPid:runner.pid,sleepPid:proc.pid,sleepBirthTime:proc.birthTime,observedProcessAgeMs:ageMs,command:m[2],chain}; + sleepCandidates.push(proof); + if(!sleepHistory.some(value=>value.runId===proof.runId&&value.sleepPid===proof.sleepPid&&value.sleepBirthTime===proof.sleepBirthTime))sleepHistory.push(proof); + } + } + await writeFile(path.join(directory,'process-gate.json'),JSON.stringify({appPid:launch.pid,lastSampleAtMs:Date.now(),sleepCandidates},null,2)); + await writeFile(path.join(directory,'owned-process-observations.json'),JSON.stringify({appPid:launch.pid,observed:[...observed.values()],runSnapshots,sleepHistory,startedAtMs:start,lastSampleAtMs:at,complete:false},null,2)); + try{const result=JSON.parse(await readFile(path.join(directory,'live-provider-result.json'),'utf8'));if(['passed','failed'].includes(result.status))terminalAt??=Date.now();}catch(error){if(error.code!=='ENOENT'&&!(error instanceof SyntaxError))throw error;} + if(terminalAt&&Date.now()-terminalAt>3000)break; + await new Promise(resolve=>setTimeout(resolve,75)); +} +const finalOutput=execFileSync('ps',['-axo','pid=,comm='],{encoding:'utf8'}); +const alive=new Set(finalOutput.split('\n').flatMap(line=>{const m=line.match(/^\s*(\d+)\s+(.+)$/);return m?[Number(m[1])]:[];})); +const result={appPid:launch.pid,observed:[...observed.values()].map(proc=>({...proc,aliveAtFinalSnapshot:alive.has(proc.pid)})),runSnapshots,sleepHistory,startedAtMs:start,finishedAtMs:Date.now(),complete:Boolean(terminalAt)}; +await writeFile(path.join(directory,'owned-process-observations.json'),JSON.stringify(result,null,2)); +console.log(JSON.stringify({complete:result.complete,observedProcesses:result.observed.length,sleepBirthIdentities:result.sleepHistory.length})); diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/verify-birth-settlement.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/verify-birth-settlement.mjs new file mode 100644 index 00000000000..d763287aa23 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/helpers/verify-birth-settlement.mjs @@ -0,0 +1,21 @@ +import assert from 'node:assert/strict'; +import {readFile,writeFile} from 'node:fs/promises'; +import {execFileSync} from 'node:child_process'; +import path from 'node:path'; +const directory=process.argv[2]; +if (!directory) throw new Error('Pass the owned evidence directory'); +const browser=JSON.parse(await readFile(path.join(directory,'live-provider-result.json'),'utf8')); +const stop=browser.journeys.find(journey=>journey.kind==='stop-live-provider'); +const gate=stop.preStopProcessProof; +assert.equal(stop.status,'passed');assert.ok(gate.verifiedBeforeClickAtMs<=stop.requestedAt); +assert.equal(gate.runId,stop.activeRunId);assert.equal(gate.command,'sleep 60'); +assert.ok(gate.observedProcessAgeMs+gate.verifiedBeforeClickAtMs-gate.observedAtMs<20000); +assert.equal(gate.chain[0].pid,gate.sleepPid);assert.equal(gate.chain.at(-1).pid,gate.runnerPid); +assert.ok(gate.chain.some(proc=>/verified-executable|opencode/.test(proc.executable))); +const checks=gate.chain.map(proc=>{ + let text='';try{text=execFileSync('ps',['-p',String(proc.pid),'-o','pid=,ppid=,stat=,lstart=,comm='],{encoding:'utf8',stdio:['ignore','pipe','ignore']}).trim();}catch{} + assert.equal(text,'','Exact owned birth PID remains present after settlement'); + return {pid:proc.pid,observedBirthTime:proc.birthTime,observedExecutable:proc.executable,absent:true}; +}); +const output={verified:true,checkedAt:new Date().toISOString(),runId:gate.runId,causalOrder:'Owned live sleep60 birth identity gate completed before ordinary UI Stop click',source:'Independent postsettlement ps by exact preStop birth-identity chain',checks}; +await writeFile(path.join(directory,'independent-birth-settlement-proof.json'),JSON.stringify(output,null,2));console.log(JSON.stringify(output,null,2)); diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-birth-settlement-proof.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-birth-settlement-proof.json new file mode 100644 index 00000000000..d910825798e --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-birth-settlement-proof.json @@ -0,0 +1,33 @@ +{ + "verified": true, + "checkedAt": "2026-11-04T00:13:20.230Z", + "runId": "51ad4429-6d34-4df9-bdb9-a54b91842d5a", + "causalOrder": "Owned live sleep60 birth identity gate completed before ordinary UI Stop click", + "source": "Independent postsettlement ps by exact preStop birth-identity chain", + "checks": [ + { + "pid": 92068, + "observedBirthTime": "Tue Nov 3 18:12:17 2026", + "observedExecutable": "sleep", + "absent": true + }, + { + "pid": 90708, + "observedBirthTime": "Tue Nov 3 18:12:06 2026", + "observedExecutable": "launch", + "absent": true + }, + { + "pid": 90674, + "observedBirthTime": "Tue Nov 3 18:12:06 2026", + "observedExecutable": ".paperclip-verified-executable-0025c93ee91e494b961a901de3688522", + "absent": true + }, + { + "pid": 90624, + "observedBirthTime": "Tue Nov 3 18:12:05 2026", + "observedExecutable": "paperclip-runnerd", + "absent": true + } + ] +} \ No newline at end of file diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-cleanup-recheck.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-cleanup-recheck.json new file mode 100644 index 00000000000..9585e1b4c19 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-cleanup-recheck.json @@ -0,0 +1,27 @@ +{ + "verified": true, + "checkedAt": "2026-11-04T00:15:40.332815+00:00", + "ports": [ + { + "port": 3104, + "ECONNREFUSED": true + }, + { + "port": 54333, + "ECONNREFUSED": true + } + ], + "absentOwnedPids": [ + 58087, + 68248, + 90624, + 90674, + 90708, + 92068, + 89408, + 89567, + 89629, + 89644 + ], + "recordedCredentialBearingDataRemoved": true +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-database-proof.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-database-proof.json new file mode 100644 index 00000000000..805586e63a5 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-database-proof.json @@ -0,0 +1,62 @@ +{ + "verified": true, + "reconstructedVerifier": true, + "checkedAt": "2026-11-04T00:12:43.335Z", + "database": "Supplied disposable loopback PostgreSQL; read-only repeatable-read snapshot", + "limitation": "Cursor reconstruction requires the complete retained fixture history; no production or raw wire ACK durability claim", + "proofs": [ + { + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "status": "succeeded", + "nextEventSeq": 123, + "eventCursorProjectionSeq": 122, + "eventCursorRebuildPending": false, + "globalEvents": 122, + "canonicalPayloadDigestsVerified": 72, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "events": 70, + "contiguousSourceSeq": 70, + "pendingRanges": [] + }, + { + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:control", + "events": 2, + "contiguousSourceSeq": 2, + "pendingRanges": [] + } + ] + }, + { + "runId": "51ad4429-6d34-4df9-bdb9-a54b91842d5a", + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "status": "cancelled", + "nextEventSeq": 72, + "eventCursorProjectionSeq": 71, + "eventCursorRebuildPending": false, + "globalEvents": 71, + "canonicalPayloadDigestsVerified": 27, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "e7bce6a4-cab2-4c8b-bf2c-cc09b5e1436b", + "events": 27, + "contiguousSourceSeq": 26, + "pendingRanges": [ + { + "startSeq": 29, + "endSeq": 29 + } + ] + } + ] + } + ] +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-lifecycle-process-proof.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-lifecycle-process-proof.json new file mode 100644 index 00000000000..0634a182569 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/independent-lifecycle-process-proof.json @@ -0,0 +1,245 @@ +{ + "verified": true, + "checkedAt": "2026-11-04T00:12:43.392Z", + "revision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "database": "Independent raw SELECT on owned loopback PostgreSQL", + "proofs": [ + { + "run": { + "id": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "status": "succeeded", + "nextEventSeq": 123, + "eventCursorProjectionSeq": 122, + "eventCursorRebuildPending": false, + "processPid": 89567, + "processGroupId": 89567, + "processStartedAt": "2026-11-04T00:11:48.000Z", + "nativePhase": "committed", + "cancellationDispatchState": null + }, + "globalEvents": 122, + "deltaKinds": { + "agentMessage": 28 + }, + "lifecycle": [ + { + "seq": 107, + "eventType": "turn.completed", + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:70", + "sourceSeq": 70, + "sourcePayloadSha256": "8ef83070ee92b77ced1041c83b8185e0bb5971206a14f98891e5b440cabb11b0", + "processPid": null, + "processGroupId": null + }, + { + "seq": 109, + "eventType": "turn.completed", + "sourceInstanceId": null, + "sourceEventId": null, + "sourceSeq": null, + "sourcePayloadSha256": null, + "processPid": null, + "processGroupId": null + }, + { + "seq": 112, + "eventType": "run.terminal", + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:control", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:control:688a6c34-f658-48ce-8ce6-61aeb2f72094:2", + "sourceSeq": 2, + "sourcePayloadSha256": "77b93871761c9889a47aa4168041186bccebf75d208506c762e4ae7387206a31", + "processPid": null, + "processGroupId": null + } + ], + "tools": [ + { + "seq": 50, + "eventType": "tool.execution.started", + "name": "paperclip_report_progress", + "status": "running" + }, + { + "seq": 51, + "eventType": "tool.execution.progressed", + "name": "paperclip_report_progress", + "status": "running" + }, + { + "seq": 52, + "eventType": "tool.execution.started", + "name": "paperclip_list_task_documents", + "status": "running" + }, + { + "seq": 53, + "eventType": "tool.execution.progressed", + "name": "paperclip_list_task_documents", + "status": "running" + }, + { + "seq": 58, + "eventType": "tool.execution.completed", + "name": "paperclip_report_progress", + "status": "completed" + }, + { + "seq": 60, + "eventType": "tool.execution.completed", + "name": "paperclip_list_task_documents", + "status": "completed" + }, + { + "seq": 88, + "eventType": "tool.execution.started", + "name": "paperclip_finish", + "status": "running" + }, + { + "seq": 91, + "eventType": "tool.execution.progressed", + "name": "paperclip_finish", + "status": "running" + }, + { + "seq": 97, + "eventType": "tool.execution.completed", + "name": "paperclip_finish", + "status": "completed" + } + ] + }, + { + "run": { + "id": "51ad4429-6d34-4df9-bdb9-a54b91842d5a", + "status": "cancelled", + "nextEventSeq": 72, + "eventCursorProjectionSeq": 71, + "eventCursorRebuildPending": false, + "processPid": null, + "processGroupId": null, + "processStartedAt": null, + "nativePhase": "observed", + "cancellationDispatchState": "acknowledged" + }, + "globalEvents": 71, + "deltaKinds": { + "agentMessage": 2 + }, + "lifecycle": [ + { + "seq": 62, + "eventType": "turn.cancelled", + "sourceInstanceId": "e7bce6a4-cab2-4c8b-bf2c-cc09b5e1436b", + "sourceEventId": "e7bce6a4-cab2-4c8b-bf2c-cc09b5e1436b:51ad4429-6d34-4df9-bdb9-a54b91842d5a:29", + "sourceSeq": 29, + "sourcePayloadSha256": "d3d3d232f5211f35e61221dc59811300e7387b43dc8261d54f868ef7719749f0", + "processPid": null, + "processGroupId": null + }, + { + "seq": 64, + "eventType": "turn.cancelled", + "sourceInstanceId": null, + "sourceEventId": null, + "sourceSeq": null, + "sourcePayloadSha256": null, + "processPid": null, + "processGroupId": null + }, + { + "seq": 70, + "eventType": "native.local_process_stopped", + "sourceInstanceId": null, + "sourceEventId": null, + "sourceSeq": null, + "sourcePayloadSha256": null, + "processPid": 90624, + "processGroupId": 90624 + } + ], + "tools": [ + { + "seq": 45, + "eventType": "tool.execution.started", + "name": "paperclip_report_progress", + "status": "running" + }, + { + "seq": 49, + "eventType": "tool.execution.progressed", + "name": "paperclip_report_progress", + "status": "running" + }, + { + "seq": 52, + "eventType": "tool.execution.completed", + "name": "paperclip_report_progress", + "status": "completed" + }, + { + "seq": 57, + "eventType": "tool.execution.started", + "name": "bash", + "status": "running" + }, + { + "seq": 58, + "eventType": "tool.execution.progressed", + "name": "bash", + "status": "running" + }, + { + "seq": 59, + "eventType": "tool.execution.progressed", + "name": "bash", + "status": "running" + } + ] + } + ], + "physicalSettlement": { + "observedRuntimePids": [ + 89408, + 89567, + 89629, + 89644, + 90624, + 90708, + 92068 + ], + "checks": [ + { + "pid": 89408, + "absent": true + }, + { + "pid": 89567, + "absent": true + }, + { + "pid": 89629, + "absent": true + }, + { + "pid": 89644, + "absent": true + }, + { + "pid": 90624, + "absent": true + }, + { + "pid": 90708, + "absent": true + }, + { + "pid": 92068, + "absent": true + } + ], + "allObservedRuntimeAbsentOrExited": true + }, + "independentCanonicalProof": "independent-database-proof.json" +} \ No newline at end of file diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/live-browser-stop-gate.patch b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/live-browser-stop-gate.patch new file mode 100644 index 00000000000..5207d272579 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/live-browser-stop-gate.patch @@ -0,0 +1,23 @@ +--- live-browser.mjs ++++ live-browser.guarded.mjs +@@ -138,6 +138,20 @@ + }, Boolean); + stopped.activeRunId = active.id; + await page.screenshot({ path: join(config.evidenceDir, "before-live-stop.png"), fullPage: true }); ++ stopped.preStopProcessProof = await poll("owned sleep60 birth identity before UI Stop", async () => { ++ let observation; ++ try { observation = JSON.parse(await readFile(join(config.evidenceDir, "process-gate.json"), "utf8")); } ++ catch (error) { if (error.code === "ENOENT" || error instanceof SyntaxError) return null; throw error; } ++ const now = Date.now(); ++ const proof = observation.sleepCandidates.find(value => value.companyId === config.company.id && value.runId === active.id ++ && value.command.trim().match(/^(?:\/[\w./-]+\/)?sleep\s+60$/) ++ && now >= value.observedAtMs && now - value.observedAtMs <= 500 ++ && value.observedProcessAgeMs + now - value.observedAtMs < 20000 ++ && value.chain[0].pid === value.sleepPid && value.chain[0].birthTime === value.sleepBirthTime ++ && value.chain.at(-1).pid === value.runnerPid && value.chain.at(-1).executable === "paperclip-runnerd" ++ && value.chain.some(process => /verified-executable|opencode/.test(process.executable))); ++ return proof ? { ...proof, verifiedBeforeClickAtMs: Date.now() } : null; ++ }, Boolean, 20000); + stopped.requestedAt = Date.now(); + await page.getByTestId("task-chat-composer-stop").last().click(); + const afterStop = await poll("native cancellation acknowledgement", async () => { diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/live-reloaded.png b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/live-reloaded.png new file mode 100644 index 00000000000..d17b692162e Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/live-reloaded.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/live-streaming.png b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/live-streaming.png new file mode 100644 index 00000000000..cedef76ff70 Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/live-streaming.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/manifest.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/manifest.json new file mode 100644 index 00000000000..d63d9565d49 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/manifest.json @@ -0,0 +1,164 @@ +{ + "schema": "paperclip.acceptance-artifact-manifest.v1", + "scope": "Fresh914 functional proof with retained progressive-content evidence; future helper checks offline only; public metadata/path adaptations declared", + "testedRevision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "original22FileManifestSha256": "88689ecd60ff01b2ff6a006bc6ed61371d3d60782fa71b823a4b166dbfde9c54", + "sourceAugmented30FileManifestSha256": "3ead5254b520d372b5c48fb41108304739482ba1573326e29ffe5def5a79f043", + "files": [ + { + "path": "README.md", + "bytes": 6764, + "sha256": "e8c050cb081f4aabef32a3a832bcd90921811b4cc6adc54eb4d00f9a75a58921" + }, + { + "path": "after-live-stop.png", + "bytes": 170188, + "sha256": "8055e3d40f1b6787c18d524b10d753bd6705ef985ba3bc96cc3fe2dcc7ebc091" + }, + { + "path": "artifact-secret-scan.json", + "bytes": 249, + "sha256": "1c6434d46ce38293e48ca434fce59c30026fb9b0a00b9fd2cb595c3d14852eaf" + }, + { + "path": "augmented-artifact-scan.json", + "bytes": 499, + "sha256": "fa984899cd1487cf456078bc20900a726a6cf7d892fbae17e1185976073a5616" + }, + { + "path": "before-live-stop.png", + "bytes": 226843, + "sha256": "9ab0a24f92e1a27deedc4568178eb69ae4654b5446226572c9f8f3aafe3369ad" + }, + { + "path": "build-ready.json", + "bytes": 1845, + "sha256": "d8d696c7ccfd8cfd76dc33ee881715f2178e14c97e3f0b520a88152656d81328" + }, + { + "path": "cleanup.json", + "bytes": 389, + "sha256": "e8e1581ac2594c646ec9cae84ea1323fdbcaf3ad15f8301aad6b1acbef9f55af" + }, + { + "path": "execution-receipts.json", + "bytes": 1758, + "sha256": "307455c8760fcff888591a12f4fc1eb1f7fdbf93b5054d9e904388c74ff72a0c" + }, + { + "path": "future-helper-review-regressions.json", + "bytes": 2064, + "sha256": "234994fab5fa62ee849bc76c6d4f12de3558bae299c4d91c701d4889bd504781" + }, + { + "path": "future-helper-streaming-regressions.json", + "bytes": 765, + "sha256": "201949be1dc6992410373f84c0feb387d28662062c96a6f01b54eea63f689022" + }, + { + "path": "helper-source-hashes.json", + "bytes": 623, + "sha256": "fb5fb12070f646cbaa04b04d1811513fca93453480a6455cc71686f865b6f34b" + }, + { + "path": "helpers/live-browser-streaming-check.mjs", + "bytes": 7179, + "sha256": "3f26b2df27f9f40e5f941706a38623cf391fe92bb4091f903745a17d963e315c" + }, + { + "path": "helpers/live-browser.future.mjs", + "bytes": 13230, + "sha256": "5ff4cd9b187829e706ff770b96a110f654b087c92a5a97c8f4ce0a1319dcf2be" + }, + { + "path": "helpers/live-browser.guarded.mjs", + "bytes": 13372, + "sha256": "d7196cc7cf021252c386228cd1dc0ce59bde213fd464dc898a2f22155ff4b37b" + }, + { + "path": "helpers/live-browser.mjs", + "bytes": 12260, + "sha256": "e6e6af5670e8da496a66f420ad32bf5fe7df208e421d1254392df37106a21848" + }, + { + "path": "helpers/live-launch.mjs", + "bytes": 4730, + "sha256": "67b5c1f367be4451ee70b969f9cf9ad10f5cb48ba4cf9ff0ea51f6f4d39b0ebb" + }, + { + "path": "helpers/process-monitor.mjs", + "bytes": 4858, + "sha256": "bc06e6797f7189a4d56b09ec268de3f59267e7ed7082f37f3aab3f4d784bc1e6" + }, + { + "path": "helpers/verify-birth-settlement.mjs", + "bytes": 1752, + "sha256": "3eca4c0bd960e87225fee9c74a6f9ef0eb785b3d3766bc8435900aaa266d4d46" + }, + { + "path": "independent-birth-settlement-proof.json", + "bytes": 971, + "sha256": "c1612655f96b6a307435cb0dc5fc0929ac968b89d039431f2f2def0ec65ba197" + }, + { + "path": "independent-cleanup-recheck.json", + "bytes": 397, + "sha256": "b2bd3829f55652ff05151035e0cde8edeef130570b39107627f0d48a7ef97f7a" + }, + { + "path": "independent-database-proof.json", + "bytes": 1993, + "sha256": "6e502a4b97229d7efcf4c10bd9e5c6dcd9e00154c86c44354868a4d7579ab461" + }, + { + "path": "independent-lifecycle-process-proof.json", + "bytes": 6764, + "sha256": "8e68827a9b29107202f5529f0f7ef3d59751336a0922fa8bc5efccefdf0d2ee6" + }, + { + "path": "live-browser-stop-gate.patch", + "bytes": 1542, + "sha256": "a1052cf24a00f50156799d0248141f7e4f18ef291b2a7e49509af89e45879c74" + }, + { + "path": "live-reloaded.png", + "bytes": 207230, + "sha256": "32a037e03d4be8ad18239a196d0339fe30836cf777ee358db58dbd87679b15ad" + }, + { + "path": "live-streaming.png", + "bytes": 277767, + "sha256": "6ce388b03caad5f984df81da0c284bdb4a5954cfd8e4a7a852d75881d6784a36" + }, + { + "path": "partial-one.jpeg", + "bytes": 172052, + "sha256": "0723dcce3f0faee653691088f2fcab3371cc6b0c3225a809741aa51d8201e1dc" + }, + { + "path": "partial-two.jpeg", + "bytes": 208651, + "sha256": "bedb9d6849cdce14c03bf33f7f35b6d6334c516fb1248df370ddd562c01a5ed4" + }, + { + "path": "public-projections.json", + "bytes": 1955, + "sha256": "ead153cab4eb082664e48b4f92cd9ee844c67342823c9bd7a016c8682f7efea6" + }, + { + "path": "qualification.json", + "bytes": 11671, + "sha256": "48ed5f00aa5ee8beceae638217c7df5e8db14841ffb241ba80110292e0555dad" + }, + { + "path": "retained-progressive-streaming-proof.json", + "bytes": 21217, + "sha256": "a509d3f1b6597fe9c799054aea49f547ebbf547610c87347912779a5e60e917c" + }, + { + "path": "runtime-provenance.json", + "bytes": 1226, + "sha256": "de542183cddfa7e64fc9eb47064ec91ac2e98a1780a07fe62a8ab7bc31495a01" + } + ] +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/partial-one.jpeg b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/partial-one.jpeg new file mode 100644 index 00000000000..3d6b181ab82 Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/partial-one.jpeg differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/partial-two.jpeg b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/partial-two.jpeg new file mode 100644 index 00000000000..68e219a8765 Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/partial-two.jpeg differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/public-projections.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/public-projections.json new file mode 100644 index 00000000000..b2e99a8c95d --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/public-projections.json @@ -0,0 +1,35 @@ +{ + "scope": "Public metadata/path-only helper adaptations; not a new live acceptance run", + "projections": [ + { + "path": "cleanup.json", + "originalSha256": "094d0b21bc2aeb8fa0d6ad751d76d631e8075c193c8963ec7bd14d5aaedc627e", + "publicSha256": "e8e1581ac2594c646ec9cae84ea1323fdbcaf3ad15f8301aad6b1acbef9f55af", + "reason": "Remove the absolute credential-bearing temporary home path; cleanup facts unchanged" + }, + { + "path": "build-ready.json", + "originalSha256": "1e4c3cfd42bda732fe143eefb6676cd8d6843d4d10e3950ae1adfedfa100efad", + "publicSha256": "d8d696c7ccfd8cfd76dc33ee881715f2178e14c97e3f0b520a88152656d81328", + "reason": "Remove local checkout/loader absolute paths; revision and build hashes unchanged" + }, + { + "path": "future-helper-review-regressions.json", + "originalSha256": "81a64f34ab1e635309d2f71f4bb55072e5f7b860e349a44efecc8e10ef5b79c9", + "publicSha256": "234994fab5fa62ee849bc76c6d4f12de3558bae299c4d91c701d4889bd504781", + "reason": "Remove a local evidence-directory path; controlled regression results and source hashes unchanged" + }, + { + "path": "helpers/process-monitor.mjs", + "originalSha256": "4f003d4db2ce70006028265735ecb10151710fccb8b6c701837ba1a21b150ab0", + "publicSha256": "bc06e6797f7189a4d56b09ec268de3f59267e7ed7082f37f3aab3f4d784bc1e6", + "reason": "Public reproduction path adaptation only; executed private source SHA remains in execution receipt; adaptation was not used for another live claim" + }, + { + "path": "helpers/verify-birth-settlement.mjs", + "originalSha256": "ce6a5aca9a0fdeef21c0e5b785d2d3ac643da62c992465f8b9b4e4c41b947cef", + "publicSha256": "3eca4c0bd960e87225fee9c74a6f9ef0eb785b3d3766bc8435900aaa266d4d46", + "reason": "Public reproduction path adaptation only; executed private source SHA remains in execution receipt; adaptation was not used for another live claim" + } + ] +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/qualification.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/qualification.json new file mode 100644 index 00000000000..6580ad289f5 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/qualification.json @@ -0,0 +1,293 @@ +{ + "schema": "paperclip.native-ingestion-live-qualification.public.v2", + "status": "passed", + "testedRevision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "scope": "Fresh real local-provider functional journey on compiled914; no performance, production-deployment or remote-cloud claim", + "dependencyType": "LIVE OpenCode/OpenRouter DeepSeek; no mock provider or deterministic fixture", + "provider": { + "adapterType": "paperclip_runner", + "provider": "opencode", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + }, + "platform": { + "node": "v26.4.0", + "os": "darwin", + "architecture": "arm64" + }, + "runtimeProvenance": "runtime-provenance.json", + "checks": { + "normalUiTaskChat": true, + "ordinaryAssistantVisibleWhileFreshGetRunning": true, + "nativeMcpToolCompleted": true, + "acceptedResultAndReload": true, + "positiveOwnedSleep60BirthBeforeVisibleStop": true, + "nativeStopAcknowledged": true, + "reloadNoSuccessor": true, + "independentRawDatabaseHashCursorProof": true, + "exactBirthIdentityChainAbsent": true, + "ownedProcessPortsDataCleanup": true, + "actualCredentialScanPassed": true, + "progressiveUiBeforeLaterOrdinaryDeltasAndAcceptedResult": true + }, + "success": { + "issueId": "bddc5135-f18e-45df-a2b5-a8d09b78f697", + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "visibleDelta": { + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:52", + "sourceSeq": 52, + "emittedAt": "2026-11-04T00:12:00.229Z", + "visibleText": "LIVE-INGESTION-1793751106135-DONE Task verified. Sent the START progress message, inspected this task's documents via the native listtaskdocuments tool (count: 0 documents), and provided the twelve numbered sentences about what the check proves. No files modified, no unrelated tasks or services accessed.", + "visibleAt": 1793751122648, + "run": { + "id": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "status": "running", + "runtimeMode": "native", + "driverKind": "opencode_server", + "providerSessionId": "ses_e5c1a28adffeMCyKLH1xL98uUQ", + "nativeSessionId": "1655845e-0764-4943-af72-5dca00c1734d", + "startedAt": "2026-11-04T00:11:47.167Z", + "finishedAt": null, + "error": null, + "nativePhase": "workspace_finalizing", + "provider": { + "kind": "opencode", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "permissionMode": "allow" + } + }, + "statusObservedAt": 1793751122748 + }, + "toolEvidence": [ + { + "eventType": "tool.execution.started", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:18", + "sourceSeq": 18, + "name": "paperclip_list_task_documents", + "operation": "unknown", + "status": "running" + }, + { + "eventType": "tool.execution.progressed", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:19", + "sourceSeq": 19, + "name": "paperclip_list_task_documents", + "operation": "unknown", + "status": "running" + }, + { + "eventType": "tool.execution.completed", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:25", + "sourceSeq": 25, + "name": "paperclip_list_task_documents", + "operation": "unknown", + "status": "completed" + } + ], + "assistantDeltas": 28, + "result": "succeeded and retained after reload", + "originalCapturedObservationLimit": "Source52 was captured during workspace_finalizing and proves only pre-terminal visibility; progressive content is separately established by earlier retained images and source29/30->31..52 chronology", + "progressiveRetainedEvidence": "retained-progressive-streaming-proof.json" + }, + "stop": { + "issueId": "21ec5d77-c672-4095-bdec-e5fae752256a", + "runId": "51ad4429-6d34-4df9-bdb9-a54b91842d5a", + "beforeClickProcessGate": { + "observedAtMs": 1793751138112, + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "runId": "51ad4429-6d34-4df9-bdb9-a54b91842d5a", + "runnerPid": 90624, + "sleepPid": 92068, + "sleepBirthTime": "Tue Nov 3 18:12:17 2026", + "observedProcessAgeMs": 1000, + "command": "sleep 60", + "chain": [ + { + "pid": 92068, + "parentPid": 90708, + "processGroupId": 92068, + "state": "Ss", + "birthTime": "Tue Nov 3 18:12:17 2026", + "executable": "sleep" + }, + { + "pid": 90708, + "parentPid": 90674, + "processGroupId": 90674, + "state": "S", + "birthTime": "Tue Nov 3 18:12:06 2026", + "executable": "launch" + }, + { + "pid": 90674, + "parentPid": 90624, + "processGroupId": 90674, + "state": "S", + "birthTime": "Tue Nov 3 18:12:06 2026", + "executable": ".paperclip-verified-executable-0025c93ee91e494b961a901de3688522" + }, + { + "pid": 90624, + "parentPid": 58087, + "processGroupId": 90624, + "state": "Ss", + "birthTime": "Tue Nov 3 18:12:05 2026", + "executable": "paperclip-runnerd" + } + ], + "verifiedBeforeClickAtMs": 1793751138237 + }, + "requestedAtMs": 1793751138237, + "acknowledgedAtMs": 1793751138609, + "observedAcknowledgementMs": 372, + "timingLimit": "One functional observation; no latency comparison/distribution", + "nativeCancellation": { + "runId": "51ad4429-6d34-4df9-bdb9-a54b91842d5a", + "scope": "run", + "schema": "paperclip.native-cancellation.v1", + "effects": [ + "release_run_resources" + ], + "issueId": "21ec5d77-c672-4095-bdec-e5fae752256a", + "intentId": "native-cancellation:eadcd50b-01fa-48e6-9d8e-9498ed524299", + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "decisionId": null, + "dispatched": true, + "reasonCode": "cancellation_run_only", + "recordedAt": "2026-11-04T00:12:18.290Z", + "dispatchState": "acknowledged", + "intentAuditId": "c3a586eb-6e87-4f20-a0c5-24b04442e9ce", + "acknowledgedAt": "2026-11-04T00:12:18.394Z", + "acknowledgementAuditId": "05d82436-2bfd-451a-82ea-d7fa3bf146fd", + "priorCoordinatorDecisionId": null + }, + "runIdsAfterReload": [ + "51ad4429-6d34-4df9-bdb9-a54b91842d5a" + ] + }, + "databaseProofs": [ + { + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "status": "succeeded", + "nextEventSeq": 123, + "eventCursorProjectionSeq": 122, + "eventCursorRebuildPending": false, + "globalEvents": 122, + "canonicalPayloadDigestsVerified": 72, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "events": 70, + "contiguousSourceSeq": 70, + "pendingRanges": [] + }, + { + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:control", + "events": 2, + "contiguousSourceSeq": 2, + "pendingRanges": [] + } + ] + }, + { + "runId": "51ad4429-6d34-4df9-bdb9-a54b91842d5a", + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "status": "cancelled", + "nextEventSeq": 72, + "eventCursorProjectionSeq": 71, + "eventCursorRebuildPending": false, + "globalEvents": 71, + "canonicalPayloadDigestsVerified": 27, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "e7bce6a4-cab2-4c8b-bf2c-cc09b5e1436b", + "events": 27, + "contiguousSourceSeq": 26, + "pendingRanges": [ + { + "startSeq": 29, + "endSeq": 29 + } + ] + } + ] + } + ], + "physicalSettlement": { + "verified": true, + "checkedAt": "2026-11-04T00:13:20.230Z", + "runId": "51ad4429-6d34-4df9-bdb9-a54b91842d5a", + "causalOrder": "Owned live sleep60 birth identity gate completed before ordinary UI Stop click", + "source": "Independent postsettlement ps by exact preStop birth-identity chain", + "checks": [ + { + "pid": 92068, + "observedBirthTime": "Tue Nov 3 18:12:17 2026", + "observedExecutable": "sleep", + "absent": true + }, + { + "pid": 90708, + "observedBirthTime": "Tue Nov 3 18:12:06 2026", + "observedExecutable": "launch", + "absent": true + }, + { + "pid": 90674, + "observedBirthTime": "Tue Nov 3 18:12:06 2026", + "observedExecutable": ".paperclip-verified-executable-0025c93ee91e494b961a901de3688522", + "absent": true + }, + { + "pid": 90624, + "observedBirthTime": "Tue Nov 3 18:12:05 2026", + "observedExecutable": "paperclip-runnerd", + "absent": true + } + ] + }, + "sourceDurabilityLimit": "Verifies database-retained events and cursors; does not equate every ordinary streaming wire ACK with durable storage", + "execution": { + "appExitCode": 0, + "browserExitCode": 0, + "monitorExitCode": 0, + "independentDbVerifierExitCode": 0, + "birthSettlementExitCode": 0, + "cleanupExitCode": 0, + "firstAttemptBothJourneysPassed": true, + "helperAdaptation": "Root-authorized private Stop boundary waits for same-run positive exact OS sleep60 birth identity before ordinary UI Stop; original corrected helper remains immutable; no provider/model/prompt/app changes", + "helperSha256": { + "independent-lifecycle-process-proof.helper.mjs": "816b30b6686a4626f72b61b64452518f6ac87b11b5338e61c1db00baa3c3ba17", + "live-launch.mjs": "67b5c1f367be4451ee70b969f9cf9ad10f5cb48ba4cf9ff0ea51f6f4d39b0ebb", + "verify-database.mjs": "d3bec80c1886792e3d2e92489a1e4aa5e79ad55947218f03fb6cd9a5762fd22b", + "cleanup.helper.mjs": "20685c6246bf2ba2194d9e43fee9526a64c32c03a757ca9ff2c8c3d0cd4a5fb9", + "verify-birth-settlement.mjs": "ce6a5aca9a0fdeef21c0e5b785d2d3ac643da62c992465f8b9b4e4c41b947cef", + "process-monitor.mjs": "4f003d4db2ce70006028265735ecb10151710fccb8b6c701837ba1a21b150ab0", + "live-browser.guarded.mjs": "d7196cc7cf021252c386228cd1dc0ce59bde213fd464dc898a2f22155ff4b37b", + "live-browser.mjs": "e6e6af5670e8da496a66f420ad32bf5fe7df208e421d1254392df37106a21848" + } + }, + "artifactsExcluded": [ + "Full trace and video (retained in private evidence)", + "Controller and database logs", + "Credential-bearing application data (removed)" + ], + "checkedAt": "2026-11-04T00:16:16.470749+00:00", + "futureVerificationHelper": { + "executedInThisLiveJourney": false, + "scope": "Verification-only selector/open-item/finalizing guards; offline regression, no new app/provider run", + "helperSha256": "5ff4cd9b187829e706ff770b96a110f654b087c92a5a97c8f4ce0a1319dcf2be", + "regressionCases": { + "focusedDomAndSource": 5, + "existingControlledSeams": 9 + } + }, + "streamingEvidenceScope": "Original selected source52 was preterminal at workspace_finalizing. Earlier retained JPEG/DOM/source excerpts establish progressive visible content before subsequent ordinary deltas and result acceptance; no literal provider-active nativePhase is asserted." +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/retained-progressive-streaming-proof.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/retained-progressive-streaming-proof.json new file mode 100644 index 00000000000..25925a2502d --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/retained-progressive-streaming-proof.json @@ -0,0 +1,580 @@ +{ + "schema": "paperclip.retained-progressive-streaming-evidence.v1", + "scope": "Offline retained-trace evidence, no fresh app/provider run", + "sourceRevision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "originalTraceSha256": "a6337930fdeeca5aca127bd7c028ed5cd92e754412e2a6157a8ac3678776f5bb", + "originalTraceBytes": 262107893, + "partialFrames": [ + { + "file": "partial-one.jpeg", + "jpegSha256": "0723dcce3f0faee653691088f2fcab3371cc6b0c3225a809741aa51d8201e1dc", + "originalResource": "resources/page@2719473e3c38e5dbeb4c0220c2f5a13f-1793751118578.jpeg", + "screencastMetadata": { + "type": "screencast-frame", + "pageId": "page@2719473e3c38e5dbeb4c0220c2f5a13f", + "sha1": "page@2719473e3c38e5dbeb4c0220c2f5a13f-1793751118578.jpeg", + "width": 1440, + "height": 1000, + "timestamp": 13068.798, + "frameSwapWallTime": 1793751118573.473 + }, + "domSnapshotName": "after@call@120", + "domWallTime": 1793751118580, + "domTimestamp": 13070.948 + }, + { + "file": "partial-two.jpeg", + "jpegSha256": "bedb9d6849cdce14c03bf33f7f35b6d6334c516fb1248df370ddd562c01a5ed4", + "originalResource": "resources/page@2719473e3c38e5dbeb4c0220c2f5a13f-1793751119146.jpeg", + "screencastMetadata": { + "type": "screencast-frame", + "pageId": "page@2719473e3c38e5dbeb4c0220c2f5a13f", + "sha1": "page@2719473e3c38e5dbeb4c0220c2f5a13f-1793751119146.jpeg", + "width": 1440, + "height": 1000, + "timestamp": 13637.225, + "frameSwapWallTime": 1793751119141.607 + }, + "domSnapshotName": "after@call@124", + "domWallTime": 1793751119149, + "domTimestamp": 13639.679 + } + ], + "firstConcretePartialDom": { + "text": "The progress message was sent and list_task_documents returned 0 documents on this task.\n1", + "ancestors": [ + { + "tag": "HTML", + "style": "color-scheme: light;" + }, + { + "tag": "BODY", + "style": "overflow: clip;" + }, + { + "tag": "DIV" + }, + { + "tag": "DIV", + "data-testid": "shell-prepared-outlet" + }, + { + "tag": "DIV" + }, + { + "tag": "DIV" + }, + { + "tag": "DIV" + }, + { + "tag": "DIV" + }, + { + "tag": "DIV" + }, + { + "tag": "MAIN" + }, + { + "tag": "DIV", + "data-task-chat-shell": "" + }, + { + "tag": "DIV", + "data-orientation": "horizontal", + "data-slot": "tabs" + }, + { + "tag": "DIV", + "data-state": "active", + "data-orientation": "horizontal", + "data-slot": "tabs-content", + "data-testid": "issue-detail-content", + "style": "" + }, + { + "tag": "DIV" + }, + { + "tag": "DIV", + "data-testid": "task-chat-thread" + }, + { + "tag": "DIV" + }, + { + "tag": "DIV", + "data-testid": "task-chat-history-content" + }, + { + "tag": "DIV" + }, + { + "tag": "DIV", + "data-testid": "task-chat-scroller" + }, + { + "tag": "DIV" + }, + { + "tag": "DIV", + "data-conversation-typography": "" + }, + { + "tag": "DIV", + "data-testid": "task-chat-live-transcript" + }, + { + "tag": "DIV", + "data-testid": "task-chat-runner-turn" + }, + { + "tag": "DIV", + "data-testid": "task-chat-turn-timeline" + }, + { + "tag": "DIV", + "data-testid": "task-chat-turn-timeline-row", + "data-timeline-row-id": "688a6c34-f658-48ce-8ce6-61aeb2f72094:msg:10:phase", + "data-thread-anchor": "688a6c34-f658-48ce-8ce6-61aeb2f72094:msg:10:phase" + }, + { + "tag": "SECTION", + "data-testid": "task-chat-activity-phase", + "data-activity-group": "688a6c34-f658-48ce-8ce6-61aeb2f72094:msg:10:phase", + "data-expanded": "false" + }, + { + "tag": "DIV", + "data-testid": "task-chat-phase-interstitial", + "data-conversation-text": "progress" + } + ], + "node": [ + "DIV", + { + "class": "min-w-0 px-1 text-sm text-foreground/90", + "data-testid": "task-chat-phase-interstitial", + "data-conversation-text": "progress" + }, + [ + "DIV", + { + "class": "paperclip-markdown prose prose-sm min-w-0 max-w-full break-words overflow-hidden", + "style": "overflow-wrap: anywhere; word-break: break-word;" + }, + [ + "P", + { + "style": "overflow-wrap: anywhere; word-break: break-word;" + }, + "The progress message was sent and ", + [ + "CODE", + { + "style": "overflow-wrap: anywhere; word-break: break-word;" + }, + "list_task_documents" + ], + " returned 0 documents on this task." + ], + "\n", + [ + "P", + { + "style": "overflow-wrap: anywhere; word-break: break-word;" + }, + "1" + ] + ] + ] + }, + "originalSelectorEmptyResult": [ + { + "type": "before", + "callId": "call@120", + "startTime": 13037.129, + "class": "Frame", + "method": "evalOnSelectorAll", + "params": { + "selector": "[data-conversation-message=\"agent\"] [data-conversation-text]:visible, [data-testid=\"task-chat-live-self-talk\"]:visible", + "expression": "(ee) => ee.map((e) => e.innerText)", + "isFunction": true, + "arg": { + "value": { + "v": "undefined" + }, + "handles": [] + } + }, + "pageId": "page@2719473e3c38e5dbeb4c0220c2f5a13f", + "beforeSnapshot": "before@call@120" + }, + { + "type": "after", + "callId": "call@120", + "endTime": 13065.759, + "result": { + "value": { + "a": [], + "id": 1 + } + }, + "afterSnapshot": "after@call@120" + } + ], + "matchedAndLaterOrdinaryEvents": [ + { + "id": 64, + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "agentId": "64291cf2-e6a9-44f9-b2d5-821f95370539", + "seq": 64, + "eventType": "item.delta", + "stream": "system", + "level": "info", + "color": null, + "message": null, + "payload": { + "prpEvent": { + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "schema": "paperclip.prp.event.v1", + "turnId": "turn-5ba4a018210a9251fd4885db", + "payload": { + "kind": "agentMessage", + "text": "The progress message was sent and", + "update": { + "kind": "agentMessage", + "text": "The progress message was sent and", + "delta": "The progress message was sent and", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "turnId": "turn-5ba4a018210a9251fd4885db", + "channel": "progress", + "provider": "opencode", + "threadId": "ses_e5c1a28adffeMCyKLH1xL98uUQ", + "providerMethod": "item/agentMessage/delta" + }, + "channel": "unknown", + "providerMethod": "item/agentMessage/delta" + }, + "priority": 1, + "emittedAt": "2026-11-04T00:11:58.550Z", + "eventType": "item.delta", + "sourceSeq": 29, + "sourceKind": "runner", + "schemaVersion": 1, + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:29", + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "normalizedSessionId": "1655845e-0764-4943-af72-5dca00c1734d" + } + }, + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:29", + "sourceSeq": 29, + "sourcePayloadSha256": "accc6086453a15c52b23a6f1af3fa1e9d263ed712b877025ab345a3570fa3b91", + "protocolSchemaVersion": 1, + "createdAt": "2026-11-04T00:11:58.551Z" + }, + { + "id": 65, + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "agentId": "64291cf2-e6a9-44f9-b2d5-821f95370539", + "seq": 65, + "eventType": "item.delta", + "stream": "system", + "level": "info", + "color": null, + "message": null, + "payload": { + "prpEvent": { + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "schema": "paperclip.prp.event.v1", + "turnId": "turn-5ba4a018210a9251fd4885db", + "payload": { + "kind": "agentMessage", + "text": " `list_task_documents` returned 0 documents on this task.\n\n1", + "update": { + "kind": "agentMessage", + "text": " `list_task_documents` returned 0 documents on this task.\n\n1", + "delta": " `list_task_documents` returned 0 documents on this task.\n\n1", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "turnId": "turn-5ba4a018210a9251fd4885db", + "channel": "progress", + "provider": "opencode", + "threadId": "ses_e5c1a28adffeMCyKLH1xL98uUQ", + "providerMethod": "item/agentMessage/delta" + }, + "channel": "unknown", + "providerMethod": "item/agentMessage/delta" + }, + "priority": 1, + "emittedAt": "2026-11-04T00:11:58.560Z", + "eventType": "item.delta", + "sourceSeq": 30, + "sourceKind": "runner", + "schemaVersion": 1, + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:30", + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "normalizedSessionId": "1655845e-0764-4943-af72-5dca00c1734d" + } + }, + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:30", + "sourceSeq": 30, + "sourcePayloadSha256": "bdbcc855be10315cc52758103f2286e0a246e3bf16271a87beb7a46bf34ed156", + "protocolSchemaVersion": 1, + "createdAt": "2026-11-04T00:11:58.560Z" + }, + { + "id": 66, + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "agentId": "64291cf2-e6a9-44f9-b2d5-821f95370539", + "seq": 66, + "eventType": "item.delta", + "stream": "system", + "level": "info", + "color": null, + "message": null, + "payload": { + "prpEvent": { + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "schema": "paperclip.prp.event.v1", + "turnId": "turn-5ba4a018210a9251fd4885db", + "payload": { + "kind": "agentMessage", + "text": ". This check proves that native Paperclip issue", + "update": { + "kind": "agentMessage", + "text": ". This check proves that native Paperclip issue", + "delta": ". This check proves that native Paperclip issue", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "turnId": "turn-5ba4a018210a9251fd4885db", + "channel": "progress", + "provider": "opencode", + "threadId": "ses_e5c1a28adffeMCyKLH1xL98uUQ", + "providerMethod": "item/agentMessage/delta" + }, + "channel": "unknown", + "providerMethod": "item/agentMessage/delta" + }, + "priority": 1, + "emittedAt": "2026-11-04T00:11:59.037Z", + "eventType": "item.delta", + "sourceSeq": 31, + "sourceKind": "runner", + "schemaVersion": 1, + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:31", + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "normalizedSessionId": "1655845e-0764-4943-af72-5dca00c1734d" + } + }, + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:31", + "sourceSeq": 31, + "sourcePayloadSha256": "e3be52cdc2f73fececbeab637b2c81cf6d40c8edf9c4e80f5c7bf6fd287e1076", + "protocolSchemaVersion": 1, + "createdAt": "2026-11-04T00:11:59.037Z" + }, + { + "id": 73, + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "agentId": "64291cf2-e6a9-44f9-b2d5-821f95370539", + "seq": 73, + "eventType": "item.delta", + "stream": "system", + "level": "info", + "color": null, + "message": null, + "payload": { + "prpEvent": { + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "schema": "paperclip.prp.event.v1", + "turnId": "turn-5ba4a018210a9251fd4885db", + "payload": { + "kind": "agentMessage", + "text": " instructions without external references.\n5. It proves the progress-reporting", + "update": { + "kind": "agentMessage", + "text": " instructions without external references.\n5. It proves the progress-reporting", + "delta": " instructions without external references.\n5. It proves the progress-reporting", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "turnId": "turn-5ba4a018210a9251fd4885db", + "channel": "progress", + "provider": "opencode", + "threadId": "ses_e5c1a28adffeMCyKLH1xL98uUQ", + "providerMethod": "item/agentMessage/delta" + }, + "channel": "unknown", + "providerMethod": "item/agentMessage/delta" + }, + "priority": 1, + "emittedAt": "2026-11-04T00:11:59.120Z", + "eventType": "item.delta", + "sourceSeq": 38, + "sourceKind": "runner", + "schemaVersion": 1, + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:38", + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "normalizedSessionId": "1655845e-0764-4943-af72-5dca00c1734d" + } + }, + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:38", + "sourceSeq": 38, + "sourcePayloadSha256": "77153c9d279e6755477b91843cfc9d9d9782406c0d9fa7a445eb99a67715136d", + "protocolSchemaVersion": 1, + "createdAt": "2026-11-04T00:11:59.128Z" + }, + { + "id": 74, + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "agentId": "64291cf2-e6a9-44f9-b2d5-821f95370539", + "seq": 74, + "eventType": "item.delta", + "stream": "system", + "level": "info", + "color": null, + "message": null, + "payload": { + "prpEvent": { + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "schema": "paperclip.prp.event.v1", + "turnId": "turn-5ba4a018210a9251fd4885db", + "payload": { + "kind": "agentMessage", + "text": " path works, since the START", + "update": { + "kind": "agentMessage", + "text": " path works, since the START", + "delta": " path works, since the START", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "turnId": "turn-5ba4a018210a9251fd4885db", + "channel": "progress", + "provider": "opencode", + "threadId": "ses_e5c1a28adffeMCyKLH1xL98uUQ", + "providerMethod": "item/agentMessage/delta" + }, + "channel": "unknown", + "providerMethod": "item/agentMessage/delta" + }, + "priority": 1, + "emittedAt": "2026-11-04T00:11:59.626Z", + "eventType": "item.delta", + "sourceSeq": 39, + "sourceKind": "runner", + "schemaVersion": 1, + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:39", + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "normalizedSessionId": "1655845e-0764-4943-af72-5dca00c1734d" + } + }, + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:39", + "sourceSeq": 39, + "sourcePayloadSha256": "6e5c08d82f493d5d296aedf4248baad2f83c193b9e5fa538aa1611da6075851a", + "protocolSchemaVersion": 1, + "createdAt": "2026-11-04T00:11:59.627Z" + }, + { + "id": 87, + "companyId": "b86f6d47-2c51-4aeb-88e1-f2ab18844f97", + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "agentId": "64291cf2-e6a9-44f9-b2d5-821f95370539", + "seq": 87, + "eventType": "item.delta", + "stream": "system", + "level": "info", + "color": null, + "message": null, + "payload": { + "prpEvent": { + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "schema": "paperclip.prp.event.v1", + "turnId": "turn-5ba4a018210a9251fd4885db", + "payload": { + "kind": "agentMessage", + "text": "3751106135-DONE\n\n", + "update": { + "kind": "agentMessage", + "text": "3751106135-DONE\n\n", + "delta": "3751106135-DONE\n\n", + "itemId": "prt_1a3e5f1ec001eLyZXOHRGC0Nw8", + "turnId": "turn-5ba4a018210a9251fd4885db", + "channel": "progress", + "provider": "opencode", + "threadId": "ses_e5c1a28adffeMCyKLH1xL98uUQ", + "providerMethod": "item/agentMessage/delta" + }, + "channel": "unknown", + "providerMethod": "item/agentMessage/delta" + }, + "priority": 1, + "emittedAt": "2026-11-04T00:12:00.229Z", + "eventType": "item.delta", + "sourceSeq": 52, + "sourceKind": "runner", + "schemaVersion": 1, + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:52", + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "normalizedSessionId": "1655845e-0764-4943-af72-5dca00c1734d" + } + }, + "sourceInstanceId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:688a6c34-f658-48ce-8ce6-61aeb2f72094:52", + "sourceSeq": 52, + "sourcePayloadSha256": "66f5c4c4b3d14c7b34716eeb7963edbeee9ff6e2389c262b825e63e96b6351d1", + "protocolSchemaVersion": 1, + "createdAt": "2026-11-04T00:12:00.230Z" + } + ], + "ordinaryEventsOriginalResource": "resources/486c57bffe212a181f572aabe1b9b15efb13fa72.json", + "acceptedResult": [ + { + "runId": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "seq": 110, + "eventType": "run.result.accepted", + "createdAt": "2026-11-04T00:12:02.610Z", + "sourceEventId": "208ee7e9-3edd-43c1-ad8f-7dee4034b667:control:688a6c34-f658-48ce-8ce6-61aeb2f72094:1", + "sourceSeq": 1, + "sourcePayloadSha256": "f2d1b753315b39ed27a260f93b7f589b618302bfafc2380732ca7bfbbe3dceba" + } + ], + "acceptedOriginalResource": "resources/f907be612c098df7a1ffacb309b3f21f98003187.json", + "freshUiRunRead": { + "startedDateTime": "2026-11-04T00:11:58.443Z", + "durationMs": 6.236, + "originalResource": "resources/4db38fadf6a163c20961a1d67189d4c9e4b777f5.json", + "run": { + "id": "688a6c34-f658-48ce-8ce6-61aeb2f72094", + "status": "running", + "runtimeMode": "native", + "startedAt": "2026-11-04T00:11:47.167Z", + "finishedAt": null + }, + "nativePhaseFieldPresent": false + }, + "observations": [ + "Original JPEGs show text growing from a short partial phrase to partial numbered sentences while the task is in progress", + "Source29/30 match the partial DOM; later ordinary source31..52 were emitted and committed after the first partial frame", + "Accepted result was committed later at00:12:02.610Z", + "Original helper excluded live interstitial/live-text branches, returning an empty array atcall120" + ], + "limits": [ + "Native phase was omitted by the retained live-runs response; this does not assert an explicit provider-active full-GET phase", + "The earlier source52 observation occurred in workspace_finalizing and proves only pre-terminal visibility", + "This is a real functional progression observation, not latency/throughput acceptance or a new provider run" + ], + "visuallyInspectedOriginalJpegs": true +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/runtime-provenance.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/runtime-provenance.json new file mode 100644 index 00000000000..ccb703f0fea --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/runtime-provenance.json @@ -0,0 +1,21 @@ +{ + "sourceRevision": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da", + "compiledBuildStamp": { + "commit": "914129afb141d81c0d6d8c5f0ea11c20bd69a3da" + }, + "sourceStatus": "", + "hashes": { + "server/dist/index.js": "7db3ab75d39e5d6a467f961a267991cc55781439572cc37e45b4599908d1e23b", + "ui/dist/index.html": "847e43ecc2469a5b54dcf89e0be4a71ba750f9d123e9d893ed90a3b3f693959e", + "server/dist/services/heartbeat-run-events.js": "f841f01af7f5a19f9f0b3be5e49dd26f66d9f869ff858d70b9e30dbacf3b29ef", + "server/dist/services/native-runtime/native-session-executor.js": "cc5367b5c207dc32bc11b8483492331d4fd6959af61ca3cd15ac1d62a359d587", + "server/dist/vendor/paperclip-runner/native-session-runtime.js": "5da124ef0d8e537c1c55a390e32c80b5a7d01dbdeeba8b08ee6c907886e43a11", + "server/dist/vendor/paperclip-runner/live/runnerd-codex-transport.js": "6075d27d51482b164c60890691d1a32d3f6b3074e160a5acfc86f57b93161ea0", + "server/dist/vendor/paperclip-runner/bin/paperclip-runnerd": "3e9acd73d2fb380d3d95e35de0e6ed78232b1be5dcbb3a96c76ca0d8b8c951b2" + }, + "sourceAndArtifactRecheckAfterJourneys": { + "checkedAt": "2026-11-04T00:14:39.100218+00:00", + "sourceClean": true, + "allRecordedArtifactsUnchanged": true + } +} diff --git a/scripts/native-ingestion/README-live.md b/scripts/native-ingestion/README-live.md index 817e2e1839d..49b3cdf7811 100644 --- a/scripts/native-ingestion/README-live.md +++ b/scripts/native-ingestion/README-live.md @@ -41,7 +41,7 @@ and checked that completed, missing-text and reasoning-only observations remain ineligible. They start no app, provider, database or network. The [receipt](../../doc/benchmarks/native-event-ingestion-2026-11-03/live/helper-review-regressions.json) records both source hashes and the -[portable source-seam check](../../doc/benchmarks/native-event-ingestion-2026-11-03/live/helper-review-regressions.mjs) +[current source-seam check](live-helper-contract-check.mjs) accepts the current helper directory, an original B9 helper directory and a new owned output path as its three arguments. @@ -168,3 +168,9 @@ private fixture to use the raw driver's explicit JSON object wrapper produced the successful qualification without changing verifier code. This is fixture qualification evidence, not a repeated live-provider journey. Every future real database invocation must retain its own revision/environment/cleanup receipt. + +## Current 914 journey and observer follow-up + +The current [914 functional packet](../../doc/benchmarks/native-event-ingestion-2026-11-03/live-ce33-914/README.md) records the exact executed helper hashes, real model/tool/result/reload, and independent live-process Stop settlement. Its initially selected source52 observation was preterminal during finalization. Original retained JPEGs/DOM/source timestamps independently prove earlier progressive content before later deltas/result; the retained UI response omitted native phase. The original executed helpers and historical proof manifests remain unchanged. + +The current browser helper includes the owned thread's normal live interstitial and live-text branches and requires an open started ordinary item plus a fresh running full GET, rejecting known finalization, accepted results, terminal state and closed items. These follow-up checks were verified offline and were not retroactively executed in the recorded journey. Run the five DOM/source checks explicitly with `node --test scripts/native-ingestion/live-browser-streaming-check.mjs`; this starts an offline browser with network aborted and belongs outside default Vitest. The current nine-case source-seam checker is `live-helper-contract-check.mjs`; the archived historical checker and receipt retain their original bytes. diff --git a/scripts/native-ingestion/live-browser-streaming-check.mjs b/scripts/native-ingestion/live-browser-streaming-check.mjs new file mode 100644 index 00000000000..62af248b0ef --- /dev/null +++ b/scripts/native-ingestion/live-browser-streaming-check.mjs @@ -0,0 +1,98 @@ +import assert from 'node:assert/strict'; +import { before, after, test } from 'node:test'; +import { readFile } from 'node:fs/promises'; +import { createRequire } from 'node:module'; + +// Exact concrete native interstitial from retained914 trace after@call@120. +// Original trace SHA256: a6337930fdeeca5aca127bd7c028ed5cd92e754412e2a6157a8ac3678776f5bb +// No server, provider or network is used; only the real browser DOM selector runs. +const capturedInterstitial = ["DIV", {"class": "min-w-0 px-1 text-sm text-foreground/90", "data-testid": "task-chat-phase-interstitial", "data-conversation-text": "progress"}, ["DIV", {"class": "paperclip-markdown prose prose-sm min-w-0 max-w-full break-words overflow-hidden", "style": "overflow-wrap: anywhere; word-break: break-word;"}, ["P", {"style": "overflow-wrap: anywhere; word-break: break-word;"}, "The progress message was sent and ", ["CODE", {"style": "overflow-wrap: anywhere; word-break: break-word;"}, "list_task_documents"], " returned 0 documents on this task."], "\n", ["P", {"style": "overflow-wrap: anywhere; word-break: break-word;"}, "1"]]]; +const source = await readFile(new URL('./live-browser.mjs', import.meta.url), 'utf8'); +const require = createRequire(new URL('../../package.json', import.meta.url)); +const { chromium } = require('@playwright/test'); +const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor; +let browser, page; +before(async () => { + browser = await chromium.launch({ headless: true }); + page = await browser.newPage(); + await page.route('**/*', route => route.abort()); +}); +after(async () => { await browser?.close(); }); +const escape = value => String(value).replaceAll('&', '&').replaceAll('<', '<').replaceAll('"', '"'); +function html(node) { + if (typeof node === 'string') return escape(node); + if (!Array.isArray(node) || typeof node[0] !== 'string') return ''; + return '<' + node[0] + Object.entries(node[1]).map(([name, value]) => ' ' + name + '="' + escape(value) + '"').join('') + + '>' + node.slice(2).map(html).join('') + ''; +} +const owned = content => '
' + content + '
'; +const observerStart = source.indexOf('const visibleAgentText = '); +const observerEnd = source.indexOf('const normalizeVisible = ', observerStart); +assert.ok(observerStart >= 0 && observerEnd > observerStart); +const readVisible = new AsyncFunction('page', source.slice(observerStart, observerEnd) + '\nreturn visibleAgentText();'); + +test('captured partial native interstitial is observed while the original selector misses it', async () => { + await page.setContent(owned('
' + + '
' + html(capturedInterstitial) + '
')); + const oldSelector = '[data-conversation-message="agent"] [data-conversation-text]:visible, [data-testid="task-chat-live-self-talk"]:visible'; + assert.deepEqual(await page.locator(oldSelector).allInnerTexts(), []); + assert.ok((await readVisible(page)).includes('The progress message was sent and')); + assert.ok((await readVisible(page)).endsWith('1')); +}); + +test('normal live-tail text and settled agent messages remain observable', async () => { + await page.setContent(owned('
Live tail partial
' + + '
Settled result
')); + assert.equal(await readVisible(page), 'Live tail partial\nSettled result'); +}); + +test('hidden output, human prompts and a different thread cannot prove owned live text', async () => { + await page.setContent(owned('
' + html(capturedInterstitial) + '
' + + '
The progress message was sent and
') + + ''); + assert.equal(await readVisible(page), ''); +}); + +async function observation({ phase = 'observed', status = 'running', finishedAt = null, + accepted = false, completedItem = false, startedItem = true, terminal = false, kind = 'agentMessage' } = {}) { + const start = source.indexOf(' const streaming = await poll('); + const end = source.indexOf(' await page.screenshot(', start); + const normalizer = source.split('\n').find(line => line.startsWith('const normalizeVisible = ')); + const summaryStart = source.indexOf('const runSummary = '), summaryEnd = source.indexOf('async function taskRuns(', summaryStart); + assert.ok(start >= 0 && end > start && normalizer && summaryStart >= 0 && summaryEnd > summaryStart); + const execute = new AsyncFunction('poll', 'taskRuns', 'visibleAgentText', 'eventsFor', 'api', 'created', 'journey', 'expect', + normalizer + '\n' + source.slice(summaryStart, summaryEnd) + '\n' + source.slice(start, end) + '\nreturn journey;'); + const active = { id: 'owned-run', status: 'running', runtimeMode: 'native' }; + const identity = { runId: active.id, sourceInstanceId: 'source-1', turnId: 'turn-1', itemId: 'item-1' }; + const row = event => ({ eventType: event.eventType, payload: { prpEvent: { ...identity, ...event } } }); + const events = [ + ...(startedItem ? [row({ eventType: 'item.started', sourceEventId: 'started' })] : []), + row({ eventType: 'item.delta', sourceEventId: 'delta-29', sourceSeq: 29, payload: { kind, text: 'The progress message was sent and' } }), + ...(completedItem ? [row({ eventType: 'item.completed' })] : []), + ...(accepted ? [row({ eventType: 'run.result.accepted' })] : []), + ...(terminal ? [row({ eventType: 'run.terminal' })] : []), + ]; + return execute(async (_label, load, accept) => { + const value = await load(); + if (!accept(value)) throw new Error('streaming_observation_not_proven'); + return value; + }, async () => [active], async () => 'The progress message was sent and', async () => events, + async () => ({ ...active, status, nativePhase: phase, finishedAt }), { issueId: 'owned-issue' }, {}, + value => ({ toBe(expected) { assert.equal(value, expected); } })); +} + +test('fresh running observation with a still-open ordinary item proves streaming', async () => { + const value = await observation(); + assert.equal(value.visibleBeforeTerminal, true); + assert.equal(value.visibleDelta.sourceEventId, 'delta-29'); + assert.equal(value.streamingRunStatuses[0].nativePhase, 'observed'); + // Native phases are not a closed verification allowlist. + assert.equal((await observation({ phase: 'another_nonfinal_phase' })).visibleBeforeTerminal, true); +}); + +test('known finalization, accepted results, terminal state and closed items are rejected', async () => { + for (const options of [{ phase: 'workspace_finalizing' }, { accepted: true }, { terminal: true }, + { completedItem: true }, { startedItem: false }, { status: 'succeeded' }, { finishedAt: 'fixture-time' }, { kind: 'reasoning' }]) { + await assert.rejects(observation(options), /streaming_observation_not_proven/); + } +}); diff --git a/scripts/native-ingestion/live-browser.mjs b/scripts/native-ingestion/live-browser.mjs index 7bdaf9fe68d..693ca92ff88 100644 --- a/scripts/native-ingestion/live-browser.mjs +++ b/scripts/native-ingestion/live-browser.mjs @@ -53,7 +53,10 @@ const context = await browser.newContext({ viewport: { width: 1440, height: 1000 recordVideo: { dir: join(config.evidenceDir, "video"), size: { width: 1440, height: 1000 } } }); await context.tracing.start({ screenshots: true, snapshots: true }); const page = await context.newPage(); -const visibleAgentText = async () => (await page.locator('[data-conversation-message="agent"] [data-conversation-text]:visible, [data-testid="task-chat-live-self-talk"]:visible').allInnerTexts()).join("\n"); +const visibleAgentText = async () => (await page.getByTestId("issue-detail-content") + .getByTestId("task-chat-thread") + .locator('[data-conversation-message="agent"] [data-conversation-text]:visible, [data-testid="task-chat-phase-interstitial"]:visible, [data-testid="task-chat-live-text"]:visible, [data-testid="task-chat-live-self-talk"]:visible') + .allInnerTexts()).join("\n"); const normalizeVisible = value => String(value).replace(/[*_`#]/g, "").replace(/\s+/g, " ").trim(); let activeIssueId; try { @@ -78,14 +81,21 @@ try { const text = normalizeVisible(await visibleAgentText()); const visibleAt = Date.now(); const rows = await eventsFor(active.id); - const delta = rows.map(row => row.payload?.prpEvent).find(event => event?.eventType === "item.delta" + const events = rows.map(row => row.payload?.prpEvent).filter(Boolean); + if (rows.some(row => ["run.result.accepted", "run.terminal"].includes(row.eventType) + || ["run.result.accepted", "run.terminal"].includes(row.payload?.prpEvent?.eventType))) return null; + const sameItem = (left, right) => left.runId === right.runId && left.sourceInstanceId === right.sourceInstanceId + && left.turnId === right.turnId && left.itemId === right.itemId; + const delta = events.find(event => event?.eventType === "item.delta" && event.runId === active.id && event.payload.kind === "agentMessage" && normalizeVisible(event.payload.text).length >= 12 - && text.includes(normalizeVisible(event.payload.text))); + && text.includes(normalizeVisible(event.payload.text)) && typeof event.itemId === "string" + && events.some(item => item.eventType === "item.started" && sameItem(item, event)) + && !events.some(item => ["item.completed", "item.failed", "item.cancelled"].includes(item.eventType) && sameItem(item, event))); if (!delta) return null; // Re-read status after observing the matching UI text. A completed run // discovered on the next poll cannot establish streaming visibility. const observedRun = await api(`/api/heartbeat-runs/${active.id}`); - if (observedRun.status !== "running") return null; + if (observedRun.status !== "running" || observedRun.finishedAt || observedRun.nativePhase === "workspace_finalizing") return null; return { runId: active.id, sourceEventId: delta.sourceEventId, sourceSeq: delta.sourceSeq, emittedAt: delta.emittedAt, visibleText: text.slice(-1000), visibleAt, run: runSummary(observedRun), statusObservedAt: Date.now() }; diff --git a/scripts/native-ingestion/live-helper-contract-check.mjs b/scripts/native-ingestion/live-helper-contract-check.mjs new file mode 100644 index 00000000000..cd0e483b6c5 --- /dev/null +++ b/scripts/native-ingestion/live-helper-contract-check.mjs @@ -0,0 +1,101 @@ +import assert from 'node:assert/strict'; +import { readFile, writeFile } from 'node:fs/promises'; +import { createHash } from 'node:crypto'; +const root = process.argv[2]; +const archived = process.argv[3]; +const output = process.argv[4]; +assert.ok(root && archived && output, 'Pass after-helper directory, before-helper directory and owned output JSON path'); +const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor; +const source = async (dir, name) => readFile(`${dir}/${name}`, 'utf8'); +const digest = text => createHash('sha256').update(text).digest('hex'); +const before = { launcher: await source(archived, 'live-launch.mjs'), browser: await source(archived, 'live-browser.mjs') }; +const after = { launcher: await source(root, 'live-launch.mjs'), browser: await source(root, 'live-browser.mjs') }; +const cases = []; +async function checkoutContract(text, linkedWorktree, instanceId = 'default') { + const setup = text.match(/(?:const \{ linkedWorktree \} = )?await prepareTestDriveEnvironment\(options, repository\);/)?.[0]; + const start = text.indexOf(' const fixture = await bootstrapTestDrive('); + const end = text.indexOf(';\n', start) + 1; + assert.ok(setup && start >= 0 && end > start, 'Exact helper contract statements must exist'); + const execute = new AsyncFunction('prepareTestDriveEnvironment', 'bootstrapTestDrive', 'options', 'repository', 'api', 'process', + `${setup}\n${text.slice(start, end)}\nreturn fixture;`); + const env = { PAPERCLIP_INSTANCE_ID: instanceId }; + return execute(async () => ({ dataDir: '/owned/data', linkedWorktree }), async input => { + assert.equal(input.linkedWorktree, linkedWorktree, 'Forward the detected checkout mode'); + assert.equal(input.instanceId, instanceId, 'Forward the prepared instance identity'); + assert.equal(input.env, env); + return { linkedWorktree: input.linkedWorktree, instanceId: input.instanceId }; + }, {}, '/owned/repository', {}, { env }); +} +await assert.rejects(checkoutContract(before.launcher, false), /Forward the detected checkout mode/); +cases.push({ name: 'original launcher rejects ordinary-checkout contract', outcome: 'expected failure reproduced' }); +for (const linkedWorktree of [false, true]) { + await checkoutContract(after.launcher, linkedWorktree); + cases.push({ name: `fixed launcher forwards linkedWorktree=${linkedWorktree}`, outcome: 'passed' }); +} +await checkoutContract(after.launcher, false, 'prepared-instance'); +cases.push({ name: 'fixed launcher forwards the prepared instance identity', outcome: 'passed' }); + +async function streamingContract(text, { afterVisibleStatus = 'running', visible = true, ordinaryDelta = true } = {}) { + const start = text.indexOf(' const streaming = await poll('); + const end = text.indexOf(' await page.screenshot(', start); + const normalizer = text.split('\n').find(line => line.startsWith('const normalizeVisible = ')); + const summaryStart = text.indexOf('const runSummary = '), summaryEnd = text.indexOf('async function taskRuns(', summaryStart); + assert.ok(start >= 0 && end > start && normalizer && summaryStart >= 0 && summaryEnd > summaryStart); + const execute = new AsyncFunction('poll', 'taskRuns', 'visibleAgentText', 'eventsFor', 'api', 'created', 'journey', 'expect', + `${normalizer}\n${text.slice(summaryStart, summaryEnd)}\n${text.slice(start, end)}\nreturn journey;`); + const calls = [], active = { id: 'native-run', status: 'running', runtimeMode: 'native' }; + let taskRunReads = 0, visibleRead = false; + const journey = {}; + const result = await execute(async (label, load, accept) => { + const value = await load(); + if (!accept(value)) throw new Error('streaming_observation_not_proven'); + return value; + }, async () => { + calls.push('task-runs'); taskRunReads++; + // The run settles directly after the successful active observation. + return [{ ...active, status: taskRunReads === 1 ? 'running' : 'succeeded' }]; + }, async () => { calls.push('visible-text'); visibleRead = true; return visible ? 'ordinary assistant streaming delta' : 'unrelated content'; }, + async () => { calls.push('durable-events'); return [ + { payload: { prpEvent: { eventType: 'item.started', runId: 'native-run', sourceInstanceId: 'source-1', + turnId: 'turn-1', itemId: 'item-1', sourceEventId: 'item-started' } } }, + { payload: { prpEvent: { + eventType: 'item.delta', runId: 'native-run', sourceInstanceId: 'source-1', turnId: 'turn-1', itemId: 'item-1', + sourceEventId: 'event-1', sourceSeq: 1, emittedAt: 'fixture-time', + payload: { kind: ordinaryDelta ? 'agentMessage' : 'reasoning', text: 'ordinary assistant streaming delta' }, + } } }, + ]; }, + async () => { + assert.equal(visibleRead, true, 'Status must be read after visible text'); calls.push('status-after-visible'); + return { ...active, status: afterVisibleStatus }; + }, { issueId: 'owned-issue' }, journey, + value => ({ toBe(expected) { assert.equal(value, expected); } })); + return { result, calls, taskRunReads }; +} +await assert.rejects(streamingContract(before.browser), /false !== true/); +cases.push({ name: 'original observer rejects a run settling immediately after successful active poll', outcome: 'expected false negative reproduced' }); +const success = await streamingContract(after.browser); +assert.equal(success.taskRunReads, 1, 'No extra run-state read can invalidate the captured observation'); +assert.deepEqual(success.calls, ['task-runs', 'visible-text', 'durable-events', 'status-after-visible']); +assert.equal(success.result.visibleBeforeTerminal, true); +assert.equal(success.result.streamingRunStatuses[0].status, 'running'); +assert.equal(success.result.visibleDelta.sourceEventId, 'event-1'); +assert.equal(success.result.firstVisibleAt, success.result.visibleDelta.visibleAt); +cases.push({ name: 'fixed observer retains exact successful active-stream observation across terminal race', outcome: 'passed', calls: success.calls }); +for (const [name, options] of [ + ['terminal status after visibility still rejects streaming proof', { afterVisibleStatus: 'succeeded' }], + ['missing visible matching text still rejects streaming proof', { visible: false }], + ['reasoning-only event still rejects ordinary-assistant proof', { ordinaryDelta: false }], +]) { + await assert.rejects(streamingContract(after.browser, options), /streaming_observation_not_proven/); + cases.push({ name, outcome: 'expected rejection passed' }); +} +const report = { + passed: true, + scope: 'Controlled helper contract/observation seams extracted from actual archived and changed source; no app, browser, provider, database or network is started.', + historicalLiveProof: '/private/tmp/native-ingestion-live-proof-base61-b9', + historicalProofUsesOriginalHelpers: true, + files: Object.fromEntries(Object.keys(before).map(name => [name, { beforeSha256: digest(before[name]), afterSha256: digest(after[name]) }])), + cases, +}; +await writeFile(output, JSON.stringify(report, null, 2) + '\n'); +console.log(JSON.stringify(report, null, 2));