diff --git a/doc/benchmarks/2026-11-03-native-event-ingestion-live.md b/doc/benchmarks/2026-11-03-native-event-ingestion-live.md new file mode 100644 index 00000000000..abb26287d41 --- /dev/null +++ b/doc/benchmarks/2026-11-03-native-event-ingestion-live.md @@ -0,0 +1,74 @@ +# Native ingestion live-provider verification (2026-11-03) + +The ordinary native task flow and Stop journey passed against the compiled +application at `b9a66be242395a4bab671d512643e338fbac77fe`. This is functional +integration acceptance, separate from paired performance measurements. + +The environment was macOS arm64, Node 26.4.0, embedded PostgreSQL 18.1, +OpenCode and the real provider model +`openrouter/deepseek/deepseek-v4-flash-0731`. The database and company were +isolated disposable fixtures. Server/UI build hashes and exact identifiers are +retained in [launch provenance](native-event-ingestion-2026-11-03/live/live-launch.json). + +| Journey | Observed and independently checked outcome | +| --- | --- | +| Normal visible task creation | Ordinary assistant text rendered while the run was active; native `list_task_documents` completed; the result was accepted. | +| Reload | The completed transcript and result remained present. The run stored 33 `item.delta` events; an ordinary assistant delta was independently matched while active. | +| Stop | A second task executed native `bash` with `sleep 60`; clicking Stop produced canonical cancellation acknowledgement. Reload retained the stopped transcript without a second run. | +| Persistence | Direct PostgreSQL reads verified 126/69 global rows, company/run/source identities, all 101 canonical payload hashes and projection markers. Physical process settlement was independently recorded. | +| Cleanup | Owned server and PostgreSQL stopped, both ports independently closed, temporary credential-bearing directories removed. | + +Success run: `e51fe504-e971-4ba1-a146-d91b5b8c92ae`. +Stop run: `9807b34f-4325-462d-bcd2-4addf66696e3`. +The stopped source's cursor remains at 24 before missing raw source sequences +25/26 and retained sequence 27. This preserves the sparse-source contract. A raw +wire acknowledgement does not universally imply ordinary transcript durability. + +The observed Stop acknowledgement interval was 481 ms. It is one functional +sample; no performance comparison or tail-latency claim follows from it. + +## Inspect the evidence + +- [Browser journey outcomes and identifiers](native-event-ingestion-2026-11-03/live/live-provider-result.json) +- [Independent PostgreSQL proof](native-event-ingestion-2026-11-03/live/independent-database-proof.json) +- [Independent lifecycle and OS settlement proof](native-event-ingestion-2026-11-03/live/independent-lifecycle-process-proof.json) +- [Build receipt](native-event-ingestion-2026-11-03/live/build-receipt.json) and [command exit/helper hash receipts](native-event-ingestion-2026-11-03/live/execution-receipts.json) +- [Cleanup receipt](native-event-ingestion-2026-11-03/live/cleanup.json) +- [Credential scan](native-event-ingestion-2026-11-03/live/artifact-secret-scan.json) and [file SHA256 manifest](native-event-ingestion-2026-11-03/live/manifest.json) + +The original artifact scan inspected 3,710 retained files and decompressed trace +entries and found zero exact qualification-credential matches. These selected +public files were scanned again before copying. No credential is committed. +The screenshots below supply the reviewable visual evidence. The video and +full trace remain optional local supplements. + +![Ordinary assistant text during an active native run](native-event-ingestion-2026-11-03/live/live-streaming.png) + +![Accepted native result](native-event-ingestion-2026-11-03/live/live-result.png) + +![Persisted result after reload](native-event-ingestion-2026-11-03/live/live-reloaded.png) + +![Active native tool before Stop](native-event-ingestion-2026-11-03/live/before-live-stop.png) + +![Stopped native run after reload](native-event-ingestion-2026-11-03/live/after-live-stop.png) + +## Reproduce + +[The live-provider helpers and instructions](../../scripts/native-ingestion/README-live.md) +document the tested helper origins and follow-up review fixes. The journey used +the original helper bytes recorded in the B9 source tag and command receipts; +current helpers repair setup/observation false failures, with a separate nine-case +[red/green receipt](native-event-ingestion-2026-11-03/live/helper-review-regressions.json). The independent database +verifier is a reconstructed utility, now exercised against this live journey's +raw PostgreSQL rows. Its [exact command record](native-event-ingestion-2026-11-03/live/independent-database-verifier-command.json) +and aggregate proof are included. Its separate real-PostgreSQL qualification +also rejected persisted hash/cursor corruption. The owned live database was +removed after verification. The earlier 6afeba4 journey remains historical; +its different 84-digest proof did not use this newer verifier. + +The earlier browser attempt failed observer qualification: it watched a tool +progress marker only in completed transcript bubbles and missed its live +presentation. The successful helper requires a matching ordinary delta visibly +rendered while the run is still active. The failed attempt remains excluded. +This work does not establish deployed production performance or a 10,000-agent +qualification. diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/after-live-stop.png b/doc/benchmarks/native-event-ingestion-2026-11-03/live/after-live-stop.png new file mode 100644 index 00000000000..9c16e8d9f82 Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/live/after-live-stop.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/artifact-secret-scan.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live/artifact-secret-scan.json new file mode 100644 index 00000000000..44f79f2a775 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/artifact-secret-scan.json @@ -0,0 +1,6 @@ +{ + "checkedAt": "2026-11-03T21:38:25.653756+00:00", + "exactQualificationCredentialPresent": false, + "scannedEntries": 3710, + "matchedFiles": [] +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/before-live-stop.png b/doc/benchmarks/native-event-ingestion-2026-11-03/live/before-live-stop.png new file mode 100644 index 00000000000..6eb8b7637a9 Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/live/before-live-stop.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/build-receipt.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live/build-receipt.json new file mode 100644 index 00000000000..6f99ef188be --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/build-receipt.json @@ -0,0 +1,11 @@ +{ + "cwd": "/private/tmp/native-ingestion-delivery-batching", + "revision": "b9a66be242395a4bab671d512643e338fbac77fe", + "command": [ + "/private/tmp/paperclip-event-tools/pnpm", + "build" + ], + "start": "2026-11-03T21:31:57.328820+00:00", + "exit": 0, + "end": "2026-11-03T21:33:05.903194+00:00" +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/cleanup.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live/cleanup.json new file mode 100644 index 00000000000..fc0cb6c5811 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/cleanup.json @@ -0,0 +1,17 @@ +{ + "startedAt": "2026-11-03T21:37:36.355Z", + "appPid": 67350, + "postgresPid": 69270, + "httpPort": 3104, + "postgresPort": 54333, + "removedOwnedDataDirs": [ + "/private/tmp/paperclip-native-live-data-SJZHZZ" + ], + "errors": [], + "appAbsent": true, + "postgresAbsent": true, + "appPortClosed": true, + "postgresPortClosed": true, + "passed": true, + "finishedAt": "2026-11-03T21:37:37.457Z" +} \ No newline at end of file diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/execution-receipts.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live/execution-receipts.json new file mode 100644 index 00000000000..b59b0cf4ae0 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/execution-receipts.json @@ -0,0 +1,14 @@ +{ + "appExitCode": 0, + "browserExitCode": 0, + "processMonitorExitCode": 0, + "independentDbVerifierExitCode": 0, + "helperSha256": { + "independent-lifecycle-process-proof.helper.mjs": "9cbaab0f93b941062632a04ee695b05bb6569f7d66662baa5d3f2bfba943e1bc", + "live-launch.mjs": "353a1b1cf2ee38551349617e91c95cdfb4d0021abf55be62e7058319b8e5b6bd", + "owned-process-observations.helper.mjs": "ec1df17f395fa8d2015352f3e050dba3712b102554e3396ef161b13ba07909f5", + "verify-database.mjs": "d3bec80c1886792e3d2e92489a1e4aa5e79ad55947218f03fb6cd9a5762fd22b", + "cleanup.helper.mjs": "c53164efe5ceb4c1506adea2a0250120da6ff55e78ce4fd3002496b96a0ad67a", + "live-browser.mjs": "cf34b0f7bb8a4f84786a38f1c90b750e7a336b126e6e2bc25647095c1eac9fab" + } +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/helper-review-regressions.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live/helper-review-regressions.json new file mode 100644 index 00000000000..64192944838 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/helper-review-regressions.json @@ -0,0 +1,60 @@ +{ + "passed": true, + "scope": "Controlled helper contract/observation seams extracted from actual archived and changed source; no app, browser, provider, database or network is started.", + "historicalLiveProof": "/private/tmp/native-ingestion-live-proof-base61-b9", + "historicalProofUsesOriginalHelpers": true, + "files": { + "launcher": { + "beforeSha256": "353a1b1cf2ee38551349617e91c95cdfb4d0021abf55be62e7058319b8e5b6bd", + "afterSha256": "67b5c1f367be4451ee70b969f9cf9ad10f5cb48ba4cf9ff0ea51f6f4d39b0ebb" + }, + "browser": { + "beforeSha256": "cf34b0f7bb8a4f84786a38f1c90b750e7a336b126e6e2bc25647095c1eac9fab", + "afterSha256": "e6e6af5670e8da496a66f420ad32bf5fe7df208e421d1254392df37106a21848" + } + }, + "cases": [ + { + "name": "original launcher rejects ordinary-checkout contract", + "outcome": "expected failure reproduced" + }, + { + "name": "fixed launcher forwards linkedWorktree=false", + "outcome": "passed" + }, + { + "name": "fixed launcher forwards linkedWorktree=true", + "outcome": "passed" + }, + { + "name": "fixed launcher forwards the prepared instance identity", + "outcome": "passed" + }, + { + "name": "original observer rejects a run settling immediately after successful active poll", + "outcome": "expected false negative reproduced" + }, + { + "name": "fixed observer retains exact successful active-stream observation across terminal race", + "outcome": "passed", + "calls": [ + "task-runs", + "visible-text", + "durable-events", + "status-after-visible" + ] + }, + { + "name": "terminal status after visibility still rejects streaming proof", + "outcome": "expected rejection passed" + }, + { + "name": "missing visible matching text still rejects streaming proof", + "outcome": "expected rejection passed" + }, + { + "name": "reasoning-only event still rejects ordinary-assistant proof", + "outcome": "expected rejection passed" + } + ] +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/helper-review-regressions.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/live/helper-review-regressions.mjs new file mode 100644 index 00000000000..5e5eba23eb7 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/helper-review-regressions.mjs @@ -0,0 +1,96 @@ +import assert from 'node:assert/strict'; +import { readFile, writeFile } from 'node:fs/promises'; +import { createHash } from 'node:crypto'; +const root = process.argv[2]; +const archived = process.argv[3]; +const output = process.argv[4]; +assert.ok(root && archived && output, 'Pass after-helper directory, before-helper directory and owned output JSON path'); +const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor; +const source = async (dir, name) => readFile(`${dir}/${name}`, 'utf8'); +const digest = text => createHash('sha256').update(text).digest('hex'); +const before = { launcher: await source(archived, 'live-launch.mjs'), browser: await source(archived, 'live-browser.mjs') }; +const after = { launcher: await source(root, 'live-launch.mjs'), browser: await source(root, 'live-browser.mjs') }; +const cases = []; +async function checkoutContract(text, linkedWorktree, instanceId = 'default') { + const setup = text.match(/(?:const \{ linkedWorktree \} = )?await prepareTestDriveEnvironment\(options, repository\);/)?.[0]; + const start = text.indexOf(' const fixture = await bootstrapTestDrive('); + const end = text.indexOf(';\n', start) + 1; + assert.ok(setup && start >= 0 && end > start, 'Exact helper contract statements must exist'); + const execute = new AsyncFunction('prepareTestDriveEnvironment', 'bootstrapTestDrive', 'options', 'repository', 'api', 'process', + `${setup}\n${text.slice(start, end)}\nreturn fixture;`); + const env = { PAPERCLIP_INSTANCE_ID: instanceId }; + return execute(async () => ({ dataDir: '/owned/data', linkedWorktree }), async input => { + assert.equal(input.linkedWorktree, linkedWorktree, 'Forward the detected checkout mode'); + assert.equal(input.instanceId, instanceId, 'Forward the prepared instance identity'); + assert.equal(input.env, env); + return { linkedWorktree: input.linkedWorktree, instanceId: input.instanceId }; + }, {}, '/owned/repository', {}, { env }); +} +await assert.rejects(checkoutContract(before.launcher, false), /Forward the detected checkout mode/); +cases.push({ name: 'original launcher rejects ordinary-checkout contract', outcome: 'expected failure reproduced' }); +for (const linkedWorktree of [false, true]) { + await checkoutContract(after.launcher, linkedWorktree); + cases.push({ name: `fixed launcher forwards linkedWorktree=${linkedWorktree}`, outcome: 'passed' }); +} +await checkoutContract(after.launcher, false, 'prepared-instance'); +cases.push({ name: 'fixed launcher forwards the prepared instance identity', outcome: 'passed' }); + +async function streamingContract(text, { afterVisibleStatus = 'running', visible = true, ordinaryDelta = true } = {}) { + const start = text.indexOf(' const streaming = await poll('); + const end = text.indexOf(' await page.screenshot(', start); + const normalizer = text.split('\n').find(line => line.startsWith('const normalizeVisible = ')); + const summaryStart = text.indexOf('const runSummary = '), summaryEnd = text.indexOf('async function taskRuns(', summaryStart); + assert.ok(start >= 0 && end > start && normalizer && summaryStart >= 0 && summaryEnd > summaryStart); + const execute = new AsyncFunction('poll', 'taskRuns', 'visibleAgentText', 'eventsFor', 'api', 'created', 'journey', 'expect', + `${normalizer}\n${text.slice(summaryStart, summaryEnd)}\n${text.slice(start, end)}\nreturn journey;`); + const calls = [], active = { id: 'native-run', status: 'running', runtimeMode: 'native' }; + let taskRunReads = 0, visibleRead = false; + const journey = {}; + const result = await execute(async (label, load, accept) => { + const value = await load(); + if (!accept(value)) throw new Error('streaming_observation_not_proven'); + return value; + }, async () => { + calls.push('task-runs'); taskRunReads++; + // The run settles directly after the successful active observation. + return [{ ...active, status: taskRunReads === 1 ? 'running' : 'succeeded' }]; + }, async () => { calls.push('visible-text'); visibleRead = true; return visible ? 'ordinary assistant streaming delta' : 'unrelated content'; }, + async () => { calls.push('durable-events'); return [{ payload: { prpEvent: { + eventType: 'item.delta', sourceEventId: 'event-1', sourceSeq: 1, emittedAt: 'fixture-time', + payload: { kind: ordinaryDelta ? 'agentMessage' : 'reasoning', text: 'ordinary assistant streaming delta' }, + } } }]; }, + async () => { + assert.equal(visibleRead, true, 'Status must be read after visible text'); calls.push('status-after-visible'); + return { ...active, status: afterVisibleStatus }; + }, { issueId: 'owned-issue' }, journey, + value => ({ toBe(expected) { assert.equal(value, expected); } })); + return { result, calls, taskRunReads }; +} +await assert.rejects(streamingContract(before.browser), /false !== true/); +cases.push({ name: 'original observer rejects a run settling immediately after successful active poll', outcome: 'expected false negative reproduced' }); +const success = await streamingContract(after.browser); +assert.equal(success.taskRunReads, 1, 'No extra run-state read can invalidate the captured observation'); +assert.deepEqual(success.calls, ['task-runs', 'visible-text', 'durable-events', 'status-after-visible']); +assert.equal(success.result.visibleBeforeTerminal, true); +assert.equal(success.result.streamingRunStatuses[0].status, 'running'); +assert.equal(success.result.visibleDelta.sourceEventId, 'event-1'); +assert.equal(success.result.firstVisibleAt, success.result.visibleDelta.visibleAt); +cases.push({ name: 'fixed observer retains exact successful active-stream observation across terminal race', outcome: 'passed', calls: success.calls }); +for (const [name, options] of [ + ['terminal status after visibility still rejects streaming proof', { afterVisibleStatus: 'succeeded' }], + ['missing visible matching text still rejects streaming proof', { visible: false }], + ['reasoning-only event still rejects ordinary-assistant proof', { ordinaryDelta: false }], +]) { + await assert.rejects(streamingContract(after.browser, options), /streaming_observation_not_proven/); + cases.push({ name, outcome: 'expected rejection passed' }); +} +const report = { + passed: true, + scope: 'Controlled helper contract/observation seams extracted from actual archived and changed source; no app, browser, provider, database or network is started.', + historicalLiveProof: '/private/tmp/native-ingestion-live-proof-base61-b9', + historicalProofUsesOriginalHelpers: true, + files: Object.fromEntries(Object.keys(before).map(name => [name, { beforeSha256: digest(before[name]), afterSha256: digest(after[name]) }])), + cases, +}; +await writeFile(output, JSON.stringify(report, null, 2) + '\n'); +console.log(JSON.stringify(report, null, 2)); diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/independent-database-proof.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live/independent-database-proof.json new file mode 100644 index 00000000000..a3d345074b8 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/independent-database-proof.json @@ -0,0 +1,62 @@ +{ + "verified": true, + "reconstructedVerifier": true, + "checkedAt": "2026-11-03T21:36:21.151Z", + "database": "Supplied disposable loopback PostgreSQL; read-only repeatable-read snapshot", + "limitation": "Cursor reconstruction requires the complete retained fixture history; no production or raw wire ACK durability claim", + "proofs": [ + { + "runId": "e51fe504-e971-4ba1-a146-d91b5b8c92ae", + "companyId": "09885f08-ef5e-42e4-b87b-29575f393527", + "status": "succeeded", + "nextEventSeq": 127, + "eventCursorProjectionSeq": 126, + "eventCursorRebuildPending": false, + "globalEvents": 126, + "canonicalPayloadDigestsVerified": 76, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "8f70aafb-262f-4091-8a9e-6da213aacc2c", + "events": 74, + "contiguousSourceSeq": 74, + "pendingRanges": [] + }, + { + "sourceInstanceId": "8f70aafb-262f-4091-8a9e-6da213aacc2c:control", + "events": 2, + "contiguousSourceSeq": 2, + "pendingRanges": [] + } + ] + }, + { + "runId": "9807b34f-4325-462d-bcd2-4addf66696e3", + "companyId": "09885f08-ef5e-42e4-b87b-29575f393527", + "status": "cancelled", + "nextEventSeq": 70, + "eventCursorProjectionSeq": 69, + "eventCursorRebuildPending": false, + "globalEvents": 69, + "canonicalPayloadDigestsVerified": 25, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "2901c54d-9367-44e7-8106-b5652c7f3fb7", + "events": 25, + "contiguousSourceSeq": 24, + "pendingRanges": [ + { + "startSeq": 27, + "endSeq": 27 + } + ] + } + ] + } + ] +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/independent-database-verifier-command.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live/independent-database-verifier-command.json new file mode 100644 index 00000000000..341e8e1bc00 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/independent-database-verifier-command.json @@ -0,0 +1,14 @@ +{ + "executable": "/opt/homebrew/Cellar/node/26.4.0/bin/node", + "script": "/private/tmp/native-ingestion-delivery-batching/scripts/native-ingestion/verify-database.mjs", + "revision": "b9a66be242395a4bab671d512643e338fbac77fe", + "companyId": "09885f08-ef5e-42e4-b87b-29575f393527", + "runIds": [ + "e51fe504-e971-4ba1-a146-d91b5b8c92ae", + "9807b34f-4325-462d-bcd2-4addf66696e3" + ], + "exit": 0, + "signal": null, + "readOnlyDatabase": "Owned loopback embedded PostgreSQL", + "databasePort": 54333 +} \ No newline at end of file diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/independent-lifecycle-process-proof.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live/independent-lifecycle-process-proof.json new file mode 100644 index 00000000000..c7fa61cf4d0 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/independent-lifecycle-process-proof.json @@ -0,0 +1,228 @@ +{ + "verified": true, + "checkedAt": "2026-11-03T21:36:21.196Z", + "revision": "b9a66be242395a4bab671d512643e338fbac77fe", + "database": "Independent raw SELECT on owned loopback PostgreSQL", + "proofs": [ + { + "run": { + "id": "e51fe504-e971-4ba1-a146-d91b5b8c92ae", + "status": "succeeded", + "nextEventSeq": 127, + "eventCursorProjectionSeq": 126, + "eventCursorRebuildPending": false, + "processPid": 72760, + "processGroupId": 72760, + "processStartedAt": "2026-11-03T21:34:10.000Z", + "nativePhase": "committed", + "cancellationDispatchState": null + }, + "globalEvents": 126, + "lifecycle": [ + { + "seq": 111, + "eventType": "turn.completed", + "sourceInstanceId": "8f70aafb-262f-4091-8a9e-6da213aacc2c", + "sourceEventId": "8f70aafb-262f-4091-8a9e-6da213aacc2c:e51fe504-e971-4ba1-a146-d91b5b8c92ae:74", + "sourceSeq": 74, + "sourcePayloadSha256": "e184e63babc8a13fc240cff3b232d48cd1f3917f231271d4ddeebc1366ccda9f", + "processPid": null, + "processGroupId": null + }, + { + "seq": 113, + "eventType": "turn.completed", + "sourceInstanceId": null, + "sourceEventId": null, + "sourceSeq": null, + "sourcePayloadSha256": null, + "processPid": null, + "processGroupId": null + }, + { + "seq": 116, + "eventType": "run.terminal", + "sourceInstanceId": "8f70aafb-262f-4091-8a9e-6da213aacc2c:control", + "sourceEventId": "8f70aafb-262f-4091-8a9e-6da213aacc2c:control:e51fe504-e971-4ba1-a146-d91b5b8c92ae:2", + "sourceSeq": 2, + "sourcePayloadSha256": "9c2ebff4157977a5a24c941e26af3c7924413ee1a81e9894e99018b1ef06b71a", + "processPid": null, + "processGroupId": null + } + ], + "tools": [ + { + "seq": 45, + "eventType": "tool.execution.started", + "name": "paperclip_report_progress", + "status": "running" + }, + { + "seq": 49, + "eventType": "tool.execution.progressed", + "name": "paperclip_report_progress", + "status": "running" + }, + { + "seq": 50, + "eventType": "tool.execution.started", + "name": "paperclip_list_task_documents", + "status": "running" + }, + { + "seq": 53, + "eventType": "tool.execution.progressed", + "name": "paperclip_list_task_documents", + "status": "running" + }, + { + "seq": 58, + "eventType": "tool.execution.completed", + "name": "paperclip_report_progress", + "status": "completed" + }, + { + "seq": 60, + "eventType": "tool.execution.completed", + "name": "paperclip_list_task_documents", + "status": "completed" + }, + { + "seq": 94, + "eventType": "tool.execution.started", + "name": "paperclip_finish", + "status": "running" + }, + { + "seq": 97, + "eventType": "tool.execution.progressed", + "name": "paperclip_finish", + "status": "running" + }, + { + "seq": 102, + "eventType": "tool.execution.completed", + "name": "paperclip_finish", + "status": "completed" + } + ] + }, + { + "run": { + "id": "9807b34f-4325-462d-bcd2-4addf66696e3", + "status": "cancelled", + "nextEventSeq": 70, + "eventCursorProjectionSeq": 69, + "eventCursorRebuildPending": false, + "processPid": null, + "processGroupId": null, + "processStartedAt": null, + "nativePhase": "observed", + "cancellationDispatchState": "acknowledged" + }, + "globalEvents": 69, + "lifecycle": [ + { + "seq": 60, + "eventType": "turn.cancelled", + "sourceInstanceId": "2901c54d-9367-44e7-8106-b5652c7f3fb7", + "sourceEventId": "2901c54d-9367-44e7-8106-b5652c7f3fb7:9807b34f-4325-462d-bcd2-4addf66696e3:27", + "sourceSeq": 27, + "sourcePayloadSha256": "96b669b6d65bc578ac85e14a27ba78282bc54bb21a960afaf90b9761f949426a", + "processPid": null, + "processGroupId": null + }, + { + "seq": 62, + "eventType": "turn.cancelled", + "sourceInstanceId": null, + "sourceEventId": null, + "sourceSeq": null, + "sourcePayloadSha256": null, + "processPid": null, + "processGroupId": null + }, + { + "seq": 68, + "eventType": "native.local_process_stopped", + "sourceInstanceId": null, + "sourceEventId": null, + "sourceSeq": null, + "sourcePayloadSha256": null, + "processPid": 78261, + "processGroupId": 78261 + } + ], + "tools": [ + { + "seq": 48, + "eventType": "tool.execution.started", + "name": "paperclip_report_progress", + "status": "running" + }, + { + "seq": 49, + "eventType": "tool.execution.progressed", + "name": "paperclip_report_progress", + "status": "running" + }, + { + "seq": 52, + "eventType": "tool.execution.completed", + "name": "paperclip_report_progress", + "status": "completed" + }, + { + "seq": 55, + "eventType": "tool.execution.started", + "name": "bash", + "status": "running" + }, + { + "seq": 56, + "eventType": "tool.execution.progressed", + "name": "bash", + "status": "running" + }, + { + "seq": 57, + "eventType": "tool.execution.progressed", + "name": "bash", + "status": "running" + } + ] + } + ], + "physicalSettlement": { + "observedRuntimePids": [ + 78261, + 80608, + 82911, + 89372 + ], + "checks": [ + { + "pid": 78261, + "absent": true + }, + { + "pid": 80608, + "absent": true + }, + { + "pid": 82911, + "absent": true + }, + { + "pid": 89372, + "absent": true + }, + { + "pid": 72760, + "absent": true + } + ], + "allObservedRuntimeAbsentOrExited": true + }, + "independentCanonicalProof": "independent-database-proof.json" +} \ No newline at end of file diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-launch.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-launch.json new file mode 100644 index 00000000000..1a0d07325a0 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-launch.json @@ -0,0 +1,30 @@ +{ + "repository": "/private/tmp/native-ingestion-delivery-batching", + "origin": "http://127.0.0.1:3104", + "company": { + "id": "09885f08-ef5e-42e4-b87b-29575f393527", + "issuePrefix": "NAT" + }, + "agent": { + "id": "41c128d6-d378-4f4c-9d68-939ac46697d7", + "name": "Live ingestion agent" + }, + "evidenceDir": "/private/tmp/native-ingestion-live-proof-base61-b9", + "revision": "b9a66be242395a4bab671d512643e338fbac77fe", + "dataDir": "/private/tmp/paperclip-native-live-data-SJZHZZ", + "pid": 67350, + "build": { + "commit": "b9a66be242395a4bab671d512643e338fbac77fe", + "serverSha256": "7db3ab75d39e5d6a467f961a267991cc55781439572cc37e45b4599908d1e23b", + "uiSha256": "7b0deb598621b2d6a276263a05073eb8215008fa297ebc41975e4871b08f6529" + }, + "platform": { + "node": "v26.4.0", + "os": "darwin", + "architecture": "arm64" + }, + "provider": { + "kind": "opencode", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + } +} \ No newline at end of file diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-provider-result.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-provider-result.json new file mode 100644 index 00000000000..5501a46913d --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-provider-result.json @@ -0,0 +1,205 @@ +{ + "startedAt": "2026-11-03T21:34:05.755Z", + "config": { + "repository": "/private/tmp/native-ingestion-delivery-batching", + "origin": "http://127.0.0.1:3104", + "company": { + "id": "09885f08-ef5e-42e4-b87b-29575f393527", + "issuePrefix": "NAT" + }, + "agent": { + "id": "41c128d6-d378-4f4c-9d68-939ac46697d7", + "name": "Live ingestion agent" + }, + "evidenceDir": "/private/tmp/native-ingestion-live-proof-base61-b9", + "revision": "b9a66be242395a4bab671d512643e338fbac77fe", + "dataDir": "/private/tmp/paperclip-native-live-data-SJZHZZ", + "pid": 67350, + "build": { + "commit": "b9a66be242395a4bab671d512643e338fbac77fe", + "serverSha256": "7db3ab75d39e5d6a467f961a267991cc55781439572cc37e45b4599908d1e23b", + "uiSha256": "7b0deb598621b2d6a276263a05073eb8215008fa297ebc41975e4871b08f6529" + }, + "platform": { + "node": "v26.4.0", + "os": "darwin", + "architecture": "arm64" + }, + "provider": { + "kind": "opencode", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + } + }, + "journeys": [ + { + "kind": "stream-tool-result-reload", + "marker": "LIVE-INGESTION-1793741646239", + "status": "passed", + "issueId": "bda268d7-f40b-4572-b2d6-2b21fd5128fe", + "visibleDelta": { + "runId": "e51fe504-e971-4ba1-a146-d91b5b8c92ae", + "sourceEventId": "8f70aafb-262f-4091-8a9e-6da213aacc2c:e51fe504-e971-4ba1-a146-d91b5b8c92ae:68", + "sourceSeq": 68, + "emittedAt": "2026-11-03T21:34:39.482Z", + "visibleText": "Completed. Sent the START marker, inspected task documents (count: 0), delivered the twelve numbered sentences, and included LIVE-INGESTION-1793741646239-DONE. Task finished." + }, + "firstVisibleAt": 1793741680262, + "streamingRunStatuses": [ + { + "id": "e51fe504-e971-4ba1-a146-d91b5b8c92ae", + "status": "running", + "runtimeMode": "native", + "driverKind": "opencode_server", + "providerSessionId": "ses_e5caa7197ffe0i2yXDZpgkjmou", + "nativeSessionId": "5d0b36b7-e0c8-49e1-9e7b-6771395593a0", + "startedAt": "2026-11-03T21:34:07.427Z", + "finishedAt": null, + "error": null, + "nativePhase": "workspace_finalizing", + "provider": { + "kind": "opencode", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "permissionMode": "allow" + } + } + ], + "visibleBeforeTerminal": true, + "runs": [ + { + "id": "e51fe504-e971-4ba1-a146-d91b5b8c92ae", + "status": "succeeded", + "runtimeMode": "native", + "driverKind": "opencode_server", + "providerSessionId": "ses_e5caa7197ffe0i2yXDZpgkjmou", + "nativeSessionId": "5d0b36b7-e0c8-49e1-9e7b-6771395593a0", + "startedAt": "2026-11-03T21:34:07.427Z", + "finishedAt": "2026-11-03T21:34:41.115Z", + "error": null, + "nativePhase": "committed", + "provider": { + "kind": "opencode", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "permissionMode": "allow" + } + } + ], + "eventCounts": { + "session.started": 1, + "session.capabilities.updated": 1, + "session.goal.snapshot": 1, + "item.completed": 17, + "turn.submitted": 1, + "harness.diagnostic": 2, + "turn.started": 1, + "turn.accepted": 1, + "tool.execution.started": 3, + "item.started": 5, + "tool.execution.progressed": 3, + "tool.execution.completed": 3, + "item.delta": 33, + "run.result.proposed": 1, + "turn.completed": 1, + "run.result.accepted": 1, + "run.terminal": 1 + }, + "toolEvidence": [ + { + "eventType": "tool.execution.started", + "sourceEventId": "8f70aafb-262f-4091-8a9e-6da213aacc2c:e51fe504-e971-4ba1-a146-d91b5b8c92ae:16", + "sourceSeq": 16, + "name": "paperclip_list_task_documents", + "operation": "unknown", + "status": "running" + }, + { + "eventType": "tool.execution.progressed", + "sourceEventId": "8f70aafb-262f-4091-8a9e-6da213aacc2c:e51fe504-e971-4ba1-a146-d91b5b8c92ae:19", + "sourceSeq": 19, + "name": "paperclip_list_task_documents", + "operation": "unknown", + "status": "running" + }, + { + "eventType": "tool.execution.completed", + "sourceEventId": "8f70aafb-262f-4091-8a9e-6da213aacc2c:e51fe504-e971-4ba1-a146-d91b5b8c92ae:25", + "sourceSeq": 25, + "name": "paperclip_list_task_documents", + "operation": "unknown", + "status": "completed" + } + ], + "persistedAfterReload": true + }, + { + "kind": "stop-live-provider", + "status": "passed", + "issueId": "224e4c2d-2c2b-40a2-9a1a-00324107c2ce", + "startedTools": [ + { + "name": "paperclip_report_progress", + "operation": "unknown", + "sourceEventId": "2901c54d-9367-44e7-8106-b5652c7f3fb7:9807b34f-4325-462d-bcd2-4addf66696e3:14" + }, + { + "name": "bash", + "operation": "unknown", + "sourceEventId": "2901c54d-9367-44e7-8106-b5652c7f3fb7:9807b34f-4325-462d-bcd2-4addf66696e3:21" + } + ], + "activeRunId": "9807b34f-4325-462d-bcd2-4addf66696e3", + "requestedAt": 1793741702357, + "acknowledgedAt": 1793741702838, + "run": { + "id": "9807b34f-4325-462d-bcd2-4addf66696e3", + "status": "cancelled", + "runtimeMode": "native", + "driverKind": "opencode_server", + "providerSessionId": "ses_e5ca9d521ffeueFZ6fehxzu4m2", + "nativeSessionId": "c05af93c-9841-48d0-9e30-64627ef8ef26", + "startedAt": "2026-11-03T21:34:45.913Z", + "finishedAt": "2026-11-03T21:35:02.584Z", + "error": "Cancelled by a board operator", + "nativePhase": "observed", + "provider": { + "kind": "opencode", + "model": "openrouter/deepseek/deepseek-v4-flash-0731", + "permissionMode": "allow" + }, + "cancellation": { + "runId": "9807b34f-4325-462d-bcd2-4addf66696e3", + "scope": "run", + "schema": "paperclip.native-cancellation.v1", + "effects": [ + "release_run_resources" + ], + "issueId": "224e4c2d-2c2b-40a2-9a1a-00324107c2ce", + "intentId": "native-cancellation:4c22f136-eff2-4b1b-a246-078dbe2a9308", + "companyId": "09885f08-ef5e-42e4-b87b-29575f393527", + "decisionId": null, + "dispatched": true, + "reasonCode": "cancellation_run_only", + "recordedAt": "2026-11-03T21:35:02.426Z", + "dispatchState": "acknowledged", + "intentAuditId": "caf271ae-08eb-4aae-9ccb-985bc825a6bb", + "acknowledgedAt": "2026-11-03T21:35:02.582Z", + "acknowledgementAuditId": "3500ea19-25e8-47c3-b132-98084238af76", + "priorCoordinatorDecisionId": null + } + }, + "runIdsAfterReload": [ + "9807b34f-4325-462d-bcd2-4addf66696e3" + ] + } + ], + "status": "passed", + "health": { + "status": "ok", + "gitCommit": "b9a66be242395a4bab671d512643e338fbac77fe" + }, + "provider": { + "adapterType": "paperclip_runner", + "provider": "opencode", + "model": "openrouter/deepseek/deepseek-v4-flash-0731" + }, + "finishedAt": "2026-11-03T21:35:04.145Z" +} \ No newline at end of file diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-reloaded.png b/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-reloaded.png new file mode 100644 index 00000000000..4e96782f891 Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-reloaded.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-result.png b/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-result.png new file mode 100644 index 00000000000..b220bf66d4c Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-result.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-streaming.png b/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-streaming.png new file mode 100644 index 00000000000..09d984a3e44 Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/live/live-streaming.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/live/manifest.json b/doc/benchmarks/native-event-ingestion-2026-11-03/live/manifest.json new file mode 100644 index 00000000000..30a21cbe492 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/live/manifest.json @@ -0,0 +1,97 @@ +{ + "scope": "Fresh real-provider functional acceptance on updated common base; not paired performance", + "revision": "b9a66be242395a4bab671d512643e338fbac77fe", + "files": [ + { + "path": "after-live-stop.png", + "bytes": 167681, + "sha256": "54cbc818d08118965e9131263de4b27cefb5fbb4097e1f6a8e195559156f7881" + }, + { + "path": "artifact-secret-scan.json", + "bytes": 150, + "sha256": "8a7c5bfc703ea787bd63b6eaafe66dcb8a7fe443ca598ce9b5f6d696d5d511ea" + }, + { + "path": "before-live-stop.png", + "bytes": 220987, + "sha256": "9aeb1fe9202f92f0457329e58f301c616321e28bf95c499c3e4433ba7508213a" + }, + { + "path": "build-receipt.json", + "bytes": 305, + "sha256": "e909096ebb0641218e4a81dd5a26791c10b7ef5963b0d43d51b2abb4564523a5" + }, + { + "path": "cleanup.json", + "bytes": 399, + "sha256": "fe80f860ebb891b16fce2792c7b27be290d831fb7b9b7e41d2b8a9b7d59c2328" + }, + { + "path": "execution-receipts.json", + "bytes": 747, + "sha256": "1d7a9e19f820a97027f8f836a4f34318b4b7a3c9cc871688e7878de897da0487" + }, + { + "path": "helper-review-regressions.json", + "bytes": 2041, + "sha256": "88a9c79f256b753d6f7eae36b6cf2962acb75450d2561d49fedfe4b2a9961ba9" + }, + { + "path": "helper-review-regressions.mjs", + "bytes": 6731, + "sha256": "66310e800d849c0ca0c049441cce9e751a20e6c2fb3d126f75ce978f450df166" + }, + { + "path": "independent-database-proof.json", + "bytes": 1993, + "sha256": "a555cd435ac3ca1b01f95c5df95776ddf2772878497a3a7f75cdb629aace5b22" + }, + { + "path": "independent-database-verifier-command.json", + "bytes": 506, + "sha256": "94886a00da917498f7f31db9df1a960e2b5ce54e35e80b59a374708b8adeb14f" + }, + { + "path": "independent-lifecycle-process-proof.json", + "bytes": 6487, + "sha256": "9e778505bb8ec2cf9308b8142a665a818443a1022de311af713b647dfab3f0c3" + }, + { + "path": "live-launch.json", + "bytes": 954, + "sha256": "4634459ce5180f73d45dc01675e76ee5270a3e58871fad644e6d3c264f91380c" + }, + { + "path": "live-provider-result.json", + "bytes": 7461, + "sha256": "c40f7605a2dce493cd63d81ee30c7dae1e8ccac030f62ba466619a98c15ec738" + }, + { + "path": "live-reloaded.png", + "bytes": 195097, + "sha256": "73c1f4ca32d01b75acaddd97594303e78ba4cc5764cebeae2da05dd79d4acb6e" + }, + { + "path": "live-result.png", + "bytes": 284421, + "sha256": "d83e6b43e1352365c741f64a024cc561836828323b3de544d00e32527814e25d" + }, + { + "path": "live-streaming.png", + "bytes": 290498, + "sha256": "1d3047d448e52a838e0b0bb12f4b2cf5d1abb4a1417f40e5b61fbb56d70720e2" + } + ], + "exactCredentialScanOfSelectedFiles": { + "credentialAvailableForComparison": true, + "files": 16, + "matches": 0 + }, + "supplementalLocalOnly": [ + "full browser video", + "full trace.zip", + "full OS process observations" + ], + "supplementalPolicy": "Required visual, persistence, lifecycle, source and cleanup evidence is included here." +} diff --git a/scripts/native-ingestion/README-live.md b/scripts/native-ingestion/README-live.md index 1d7f74862d3..817e2e1839d 100644 --- a/scripts/native-ingestion/README-live.md +++ b/scripts/native-ingestion/README-live.md @@ -6,8 +6,9 @@ experiment and profile-only tools are documented in the shared `README.md`. ## Exact helper provenance -The following files are byte-for-byte copies of the successful November 3, 2026 -helpers. Paths in this table describe their original temporary locations. +The original successful November 3, 2026 helpers were snapshotted from the +paths below. These hashes describe the original bytes, also used for the fresh +B9 journey; the current helpers include the review fixes described below. | File | Original in `/private/tmp/` | SHA256 | | --- | --- | --- | @@ -23,6 +24,27 @@ provider/model selection. It is not a qualified portable launcher. It reads an encrypted instance secret through the normal test-drive API bootstrap. Supply the credential privately; do not put it in arguments, logs or evidence. +Those original helper bytes passed fresh verification on the updated common-base revision +`b9a66be242395a4bab671d512643e338fbac77fe`. The reconstructed verifier below +checked that journey's actual raw database rows: 126/69 global rows and 101 +canonical digests. Native lifecycle and independent OS checks confirmed process +settlement. The owned instance was removed and all 3,710 retained/decompressed +artifact entries passed the exact credential scan. See the +[current functional evidence](../../doc/benchmarks/2026-11-03-native-event-ingestion-live.md). + +Review follow-ups now forward the actual detected `linkedWorktree` mode and +prepared instance identity, and retain the exact successful active-stream +observation instead of fetching a later status that can race completion. The +live B9 proof above used the archived original bytes. Nine controlled cases +executed the actual old/new helper statements, reproduced both original defects, +and checked that completed, missing-text and reasoning-only observations remain +ineligible. They start no app, provider, database or network. The +[receipt](../../doc/benchmarks/native-event-ingestion-2026-11-03/live/helper-review-regressions.json) +records both source hashes and the +[portable source-seam check](../../doc/benchmarks/native-event-ingestion-2026-11-03/live/helper-review-regressions.mjs) +accepts the current helper directory, an original B9 helper directory and a new +owned output path as its three arguments. + ## Repeating the functional journey Use an independently installed and normally built application checkout. Its @@ -61,7 +83,7 @@ performance measurements. While the owned app/database remain available, run the independent verifier below. Afterwards, gracefully stop only the launcher process, confirm its HTTP and database ports and provider processes are gone, and remove only its recorded -credential-bearing `dataDir`. Keep cleanup results separately. The exact copied +credential-bearing `dataDir`. Keep cleanup results separately. The launcher retains the directory, so process exit alone is not data cleanup. Before publishing retained browser artifacts, scan the files and decompressed trace archive for the actual qualification credential without printing it.