diff --git a/doc/benchmarks/2026-11-03-native-event-ingestion-cloud-stop.md b/doc/benchmarks/2026-11-03-native-event-ingestion-cloud-stop.md new file mode 100644 index 00000000000..fd33e909ff7 --- /dev/null +++ b/doc/benchmarks/2026-11-03-native-event-ingestion-cloud-stop.md @@ -0,0 +1,13 @@ +# Real cloud Stop acceptance (2026-11-03) + +The normal task UI's Stop control cancelled a native run in a real Daytona sandbox. The production-built controller served `137501f595fd5ee13d395e26b4147ca8bc4d4eb6`; the provider was an explicitly declared deterministic Codex fixture executing a real `sleep 60`. This adds cloud termination coverage to the separate [live-LLM journey](2026-11-03-native-event-ingestion-live.md). + +The [recorded proof](native-event-ingestion-2026-11-03/cloud-stop/qualification.json) contains the original native acknowledgement, shared cancellation acknowledgement with `provider_termination_receipt`, matching stopped-provider receipt, and the exact lease's successful release. Independent SDK observations confirmed child retirement, absence of its forbidden effect, and the same labelled sandbox's stopped state. Reload and direct PostgreSQL checks verified the retained transcript, canonical event hashes, cursor projection, and absence of a successor or accepted result. + +This is one functional sample. Its acknowledgement timings establish no latency distribution, throughput improvement, or deployed-production gain. The database checks retain the existing sparse-source/cancellation policy; they do not equate every ordinary streaming wire ACK with database durability. + +All five created sandboxes were independently confirmed absent. The [attempt ledger](native-event-ingestion-2026-11-03/cloud-stop/attempts.json) preserves four earlier harness failures and their cleanup. The controller, database and credential-bearing temporary data were removed; retained and decompressed artifacts were scanned for the actual credentials. The [reproduction recipe](native-event-ingestion-2026-11-03/cloud-stop/README.md) declares live versus simulated dependencies, resource bounds, immutable image, SDK, runtime hashes, and the public helper's credential, resource-cap and clock-guard adaptations. The [manifest](native-event-ingestion-2026-11-03/cloud-stop/manifest.json) records exact artifact hashes. + +![Actual streaming and real command before Stop](native-event-ingestion-2026-11-03/cloud-stop/before-stop.png) + +![Stopped run and persisted text after reload](native-event-ingestion-2026-11-03/cloud-stop/after-stop-reload.png) diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/README.md b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/README.md new file mode 100644 index 00000000000..010e4cd55fd --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/README.md @@ -0,0 +1,40 @@ +# Real Daytona Stop acceptance + +The production-built candidate at `137501f595fd5ee13d395e26b4147ca8bc4d4eb6` passed one real-Daytona Stop journey. The provider was a declared deterministic Codex fixture; native runnerd, authenticated Daytona ingress, the actual `sleep 60` process, Paperclip persistence, the visible Stop control, provider termination receipts, and independent Daytona SDK state were real. This is functional acceptance, not a performance comparison or a live-LLM claim. + +`qualification.json` records native and shared acknowledgements, exact lease/provider binding, independent child retirement with no forbidden effect, stopped provider state, reload/no-successor checks, and direct PostgreSQL canonical-payload/cursor proof. `attempts.json` preserves the four earlier harness failures and their cleanup. All resources were deleted. The full traces and failed screenshots remain in private task evidence; this compact set includes the successful before/after screenshots. + +The shared provider receipt took about12.4 seconds in this single observation; native acknowledgement took about0.4 seconds. These are individual functional timings, not latency distribution or performance acceptance evidence. Normal environment deletion returned HTTP success; exact-owned SDK deletion then independently established absence. Local cleanup initially met readonly skill directories; only directories in the stopped, recorded temporary data root were made writable without following symlinks, then the root was removed. + +The verified cursor can contain gaps under the existing raw-transport cancellation policy. Database-retained event hashes and cursors were checked independently. This does not claim every ordinary streaming wire ACK was database durable. + +## Reproduce + +Paid opt-in. Use one sandbox, at most30 minutes, then delete it. The helper has a one-resource cap and an independent watchdog. Do not rerun with a fresh evidence directory to evade the cap. No automatic retry is provided. SDK 0.203.0 and the immutable image are pinned in the helper. Current runnerd and the self-contained fixture are staged through supported remote artifact paths; the old image runner/provider bridge is not used. + +Build an independent checkout normally with locked dependencies (`pnpm install --frozen-lockfile`, `pnpm build`). Also run frozen install/build in `packages/plugins/sandbox-providers/daytona`, which has its own lockfile. Build the same revision's Linux-x64 runnerd using the locked Cargo release build on Linux and copy the executable to the controller. The tested controller was macOS arm64/Node 26.4.0; platform portability beyond that is not claimed. + +In a shell with no inherited API keys/tokens, start the owned production controller using the app checkout's TSX loader and `helpers/launch.mjs`, with positional arguments: `APP_CHECKOUT EXPECTED_COMMIT NEW_EVIDENCE_DIRECTORY LINUX_RUNNERD_PATH PROVIDER_FIXTURE_PATH`. The last path is the absolute path of `helpers/provider.mjs`. Wait for `launch.json`. The launcher requires source and compiled build stamps to equal the expected commit. + +In a separate shell, bind `DAYTONA_API_KEY` securely from your credential manager without printing it or placing it in arguments/files. Run the app checkout's Node+TSX loader on `helpers/run.mjs NEW_EVIDENCE_DIRECTORY/launch.json`. It sends the key once through the normal encrypted company-secret API, removes inherited credential environment variables before downstream work, and retains only secret references. No public controller tunnel is used. The helper installs the first-party plugin, creates an owned environment, creates a fresh task through the ordinary UI, observes the real process and actual normalized text, clicks the ordinary Stop button, and requires native/shared/provider/SDK/reload/cleanup gates. + +Keep the isolated app/database alive for independent verification with the repository's `scripts/native-ingestion/verify-database.mjs` against the explicit owned loopback database. Then gracefully stop the recorded launcher, verify its HTTP/DB ports and process are gone, and remove only its recorded temporary data directory. Scan retained files and decompressed trace entries for the actual credential before sharing. The public helper adapts environment-supplied credentials, a one-resource cap, and the process-age guard to use the provider's own clock. It records all already-admitted owned IDs before enforcing the cap so failure cleanup can retire them. Exact executed private-helper hashes are recorded separately. These public adaptations passed syntax checks and an offline actual-source cleanup regression; they were not used for a second live acceptance claim. + +Run `node helpers/verify-failure-cleanup.mjs` from this directory for the offline regression. It exercises the reproducer's actual admission and cleanup sections with two already-admitted owned IDs and a cap of one. Qualification must fail while both IDs reach the watchdog ledger and exact SDK cleanup. All external effects are stubbed; it creates no sandbox and reads no credential. + +Example commands after setting the path variables (never include credentials in arguments): + +```sh +env -i PATH="$PATH" HOME="$HOME" TMPDIR="${TMPDIR:-/tmp}" \ + node --import "$APP_CHECKOUT/server/node_modules/tsx/dist/loader.mjs" \ + "$HELPER_DIR/launch.mjs" "$APP_CHECKOUT" "$EXPECTED_COMMIT" \ + "$EVIDENCE_DIR" "$LINUX_RUNNERD" "$HELPER_DIR/provider.mjs" + +# Separate shell: DAYTONA_API_KEY is already bound privately by the credential manager. +node --import "$APP_CHECKOUT/server/node_modules/tsx/dist/loader.mjs" \ + "$HELPER_DIR/run.mjs" "$EVIDENCE_DIR/launch.json" +``` + +The cost bound uses the official [Daytona pricing](https://www.daytona.io/pricing), checked November 3, 2026. It is a list-rate bound for the selected resources, not an invoice. + +The independent process snapshot uses the provider clock for process age. Its RPC completed before the UI click, but provider and controller clocks are independent; no cross-host age or delivery interval is inferred. The later PR revision `fcb5d94848b3c4b0b32096ff5e023618e0f0637b` differs from the tested build only in `packages/plugins/paperclip-plugin-fake-sandbox/src/lease-cleanup.ts` and its test. The actual tested serving revision remains `137501f595fd5ee13d395e26b4147ca8bc4d4eb6`. diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/after-stop-reload.png b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/after-stop-reload.png new file mode 100644 index 00000000000..8b4f7052295 Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/after-stop-reload.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/attempts.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/attempts.json new file mode 100644 index 00000000000..cba19368d0e --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/attempts.json @@ -0,0 +1,85 @@ +{ + "totalActualSandboxes": 5, + "maximumConcurrentSandboxes": 1, + "ttlMinutes": 30, + "attempts": [ + { + "attempt": 1, + "status": "failed", + "diagnosis": "Premature assertion of native mode while a queued row still legitimately had its initial legacy value.", + "visibleStopReached": false, + "runId": "cc899633-40a7-4911-bf22-2a4db3c8f0f3", + "leaseId": "a15aa112-f1fd-4bb0-84e8-c51add8aea76", + "providerLeaseId": "61ea5024-e513-4234-8925-22109c3e9649", + "admittedAtMs": 1793746211555, + "hardWatchdogDeadlineMs": 1793747931231, + "providerAbsentAtMs": 1793746214302, + "normalEnvironmentDeleteHttpSucceeded": true, + "exactOwnedSdkDeleteUsed": true, + "watchdogDisarmed": true + }, + { + "attempt": 2, + "status": "failed", + "diagnosis": "Inherited visible-text selector missed the actual remote tool/commentary layout.", + "visibleStopReached": false, + "runId": "d40574ba-a0e9-41af-b055-fb4f2208f98f", + "leaseId": "a56044f0-5ab2-480f-9cc3-373749f0281f", + "providerLeaseId": "3f713f74-c3d8-49e3-9e84-807192c6ea0e", + "admittedAtMs": 1793746376622, + "hardWatchdogDeadlineMs": 1793748171862, + "providerAbsentAtMs": 1793746414732, + "normalEnvironmentDeleteHttpSucceeded": true, + "exactOwnedSdkDeleteUsed": true, + "watchdogDisarmed": true + }, + { + "attempt": 3, + "status": "failed", + "diagnosis": "Settled run:turn anchor was absent in the actual live transcript.", + "visibleStopReached": false, + "runId": "e08343f7-0103-4be2-ab11-7dca9a67371f", + "leaseId": "edf359a3-aab6-4680-b45d-d7207ec057a9", + "providerLeaseId": "d61aec89-e2c4-4cc7-aaf3-6b7d2bf04c5e", + "admittedAtMs": 1793746775566, + "hardWatchdogDeadlineMs": 1793748568908, + "providerAbsentAtMs": 1793746810326, + "normalEnvironmentDeleteHttpSucceeded": true, + "exactOwnedSdkDeleteUsed": true, + "watchdogDisarmed": true + }, + { + "attempt": 4, + "status": "failed", + "diagnosis": "Incorrect UUID-in-URL assumption; normal UI redirects to canonical task identifier.", + "visibleStopReached": false, + "runId": "fe71280a-0452-4599-9b28-65083b151d2d", + "leaseId": "8f9fb9d2-ecfb-4581-9233-14da7072c766", + "providerLeaseId": "cbf9f558-49f2-4b84-bbe1-ea651502a6bb", + "admittedAtMs": 1793747062452, + "hardWatchdogDeadlineMs": 1793748857276, + "providerAbsentAtMs": 1793747077893, + "normalEnvironmentDeleteHttpSucceeded": true, + "exactOwnedSdkDeleteUsed": true, + "watchdogDisarmed": true + }, + { + "attempt": 5, + "status": "passed", + "diagnosis": "Passed normal visible UI Stop and all required native/shared/provider/SDK/persistence gates.", + "visibleStopReached": true, + "runId": "c42d6ef1-ae58-44e3-8b76-fcf9b682000f", + "leaseId": "1b1aca64-e936-47d4-8509-43b7fe49f75f", + "providerLeaseId": "19e2daa7-e0fa-4427-988b-de136dd1b2b0", + "admittedAtMs": 1793747204043, + "hardWatchdogDeadlineMs": 1793748998954, + "providerAbsentAtMs": 1793747239557, + "normalEnvironmentDeleteHttpSucceeded": true, + "exactOwnedSdkDeleteUsed": true, + "watchdogDisarmed": true + } + ], + "additionalZeroResourceFailure": "Retry setup attempted to install an already installed plugin; preserved separately, created no sandbox.", + "cleanupLimit": "Earlier failed attempts used retain-on-failure and recorded pending cleanup/sandbox_stop_pending before exact-owned SDK deletion. Endpoint HTTP success alone is not provider absence.", + "billing": "Actual provider invoice was not queried. Five30-minute full lifetimes at4CPU/4GiB/10GiB would be about$0.669 at the checked Linux list rates, excluding image or other provider fees; actual admitted-to-deletion lifetimes were much shorter." +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/before-stop.png b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/before-stop.png new file mode 100644 index 00000000000..b9dd4b087e6 Binary files /dev/null and b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/before-stop.png differ diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/cleanup-and-scan.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/cleanup-and-scan.json new file mode 100644 index 00000000000..16848ca9420 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/cleanup-and-scan.json @@ -0,0 +1,59 @@ +{ + "status": "passed", + "providerAndCredentialScan": { + "status": "passed", + "checkedAt": "2026-11-03T23:14:03.153Z", + "providerApiOperations": "get only; no resource mutations", + "providerStates": [ + { + "id": "61ea5024-e513-4234-8925-22109c3e9649", + "absent": true, + "errorName": "DaytonaNotFoundError", + "status": 404 + }, + { + "id": "3f713f74-c3d8-49e3-9e84-807192c6ea0e", + "absent": true, + "errorName": "DaytonaNotFoundError", + "status": 404 + }, + { + "id": "d61aec89-e2c4-4cc7-aaf3-6b7d2bf04c5e", + "absent": true, + "errorName": "DaytonaNotFoundError", + "status": 404 + }, + { + "id": "cbf9f558-49f2-4b84-bbe1-ea651502a6bb", + "absent": true, + "errorName": "DaytonaNotFoundError", + "status": 404 + }, + { + "id": "19e2daa7-e0fa-4427-988b-de136dd1b2b0", + "absent": true, + "errorName": "DaytonaNotFoundError", + "status": 404 + } + ], + "exactCredentialKinds": [ + "Daytona API key", + "1Password service-account token" + ], + "knownPreviewTokens": 0, + "scannedFilesAndDecompressedEntries": 3084, + "matches": [] + }, + "localCleanup": { + "status": "passed", + "checkedAt": "2026-11-03T23:10:48.163Z", + "ownedControllerProcessGone": true, + "ownedHttpPortClosed": true, + "ownedDatabasePortClosed": true, + "postmasterPidFileGone": true, + "credentialBearingDataRemoved": true, + "initialRemovalFailure": "EACCES unlink readonly skill bundle; retained as cleanup failure before bounded repair", + "ownedDirectoriesMadeWritable": 143, + "followedSymlinks": false + } +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/database-proof-all-attempts.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/database-proof-all-attempts.json new file mode 100644 index 00000000000..fc079e19070 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/database-proof-all-attempts.json @@ -0,0 +1,127 @@ +{ + "verified": true, + "reconstructedVerifier": true, + "checkedAt": "2026-11-03T23:08:51.233Z", + "database": "Supplied disposable loopback PostgreSQL; read-only repeatable-read snapshot", + "limitation": "Cursor reconstruction requires the complete retained fixture history; no production or raw wire ACK durability claim", + "proofs": [ + { + "runId": "cc899633-40a7-4911-bf22-2a4db3c8f0f3", + "companyId": "f2ad7f34-3348-4be1-a7ef-139e3fd83324", + "status": "cancelled", + "nextEventSeq": 6, + "eventCursorProjectionSeq": 5, + "eventCursorRebuildPending": false, + "globalEvents": 5, + "canonicalPayloadDigestsVerified": 0, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [] + }, + { + "runId": "d40574ba-a0e9-41af-b055-fb4f2208f98f", + "companyId": "f2ad7f34-3348-4be1-a7ef-139e3fd83324", + "status": "failed", + "nextEventSeq": 80, + "eventCursorProjectionSeq": 79, + "eventCursorRebuildPending": false, + "globalEvents": 79, + "canonicalPayloadDigestsVerified": 15, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "75745627-bfa7-48e8-8e81-5f5656bab8b5", + "events": 15, + "contiguousSourceSeq": 14, + "pendingRanges": [ + { + "startSeq": 18, + "endSeq": 18 + } + ] + } + ] + }, + { + "runId": "e08343f7-0103-4be2-ab11-7dca9a67371f", + "companyId": "f2ad7f34-3348-4be1-a7ef-139e3fd83324", + "status": "failed", + "nextEventSeq": 79, + "eventCursorProjectionSeq": 78, + "eventCursorRebuildPending": false, + "globalEvents": 78, + "canonicalPayloadDigestsVerified": 15, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "cf796eeb-9555-4dcd-ae16-b441f9fd49ce", + "events": 15, + "contiguousSourceSeq": 14, + "pendingRanges": [ + { + "startSeq": 18, + "endSeq": 18 + } + ] + } + ] + }, + { + "runId": "fe71280a-0452-4599-9b28-65083b151d2d", + "companyId": "f2ad7f34-3348-4be1-a7ef-139e3fd83324", + "status": "cancelled", + "nextEventSeq": 78, + "eventCursorProjectionSeq": 77, + "eventCursorRebuildPending": false, + "globalEvents": 77, + "canonicalPayloadDigestsVerified": 16, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "3b60a81c-e14c-4a0c-9475-5ad665e5d7ca", + "events": 16, + "contiguousSourceSeq": 15, + "pendingRanges": [ + { + "startSeq": 19, + "endSeq": 19 + } + ] + } + ] + }, + { + "runId": "c42d6ef1-ae58-44e3-8b76-fcf9b682000f", + "companyId": "f2ad7f34-3348-4be1-a7ef-139e3fd83324", + "status": "cancelled", + "nextEventSeq": 82, + "eventCursorProjectionSeq": 81, + "eventCursorRebuildPending": false, + "globalEvents": 81, + "canonicalPayloadDigestsVerified": 16, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "2f93856a-210e-4e98-ad38-70ee46014d9a", + "events": 16, + "contiguousSourceSeq": 15, + "pendingRanges": [ + { + "startSeq": 19, + "endSeq": 19 + } + ] + } + ] + } + ] +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helper-checks.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helper-checks.json new file mode 100644 index 00000000000..a3af58ee496 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helper-checks.json @@ -0,0 +1,110 @@ +{ + "admission-helper-check.json": { + "status": "passed", + "actualHelperSha256": "975e19141a1a70a98024fc42b54e07b91c9d63dc53be3f28884def118b1f6a70", + "actualPollExecuted": true, + "actualAdmissionPredicateExecuted": true, + "cases": 5, + "observedTransitions": [ + { + "runtimeMode": "legacy", + "status": "queued" + }, + { + "runtimeMode": "legacy", + "status": "running" + }, + { + "runtimeMode": "native", + "status": "queued" + }, + { + "runtimeMode": "native", + "status": "running" + } + ], + "readsBeforeRelease": 4, + "releasedAfterNativeRunning": true, + "resourceCalls": 0 + }, + "visible-selector-check.json": { + "status": "passed", + "resourceCalls": 0, + "actualSharedLocatorExecuted": true, + "source": "existing owned task UI plus reversible DOM-only negative cases", + "issueId": "6e6bc95f-7852-417c-95d1-20d2a535358d", + "runId": "d40574ba-a0e9-41af-b055-fb4f2208f98f", + "legacyVisibleTexts": [], + "correctedVisibleMarkerCount": 1, + "correctedMarkerVisible": true, + "negativeCases": [ + "hidden_owned_marker", + "user_prompt_only", + "unrelated_thread" + ], + "terminalStateCannotProveStreaming": true, + "runStatus": "failed", + "limitation": "Locator requires separate fresh-task/one-run ownership check; this alone cannot establish current streaming." + }, + "captured-live-dom-check.json": { + "status": "passed", + "resourceCalls": 0, + "source": "Actual third-attempt Playwright LIVE frame snapshot ancestry, replayed without network", + "traceSha256": "2a7f10af70c4d43b4e3ecbca69ea02c6ae9bd1243765aa8739d82c4149fa84e5", + "snapshotName": "after@call@44", + "correctedSharedLocatorVisible": true, + "priorExactTurnAnchorCount": 0, + "liveAncestorTestIds": [ + "shell-prepared-outlet", + "issue-detail-content", + "task-chat-thread", + "task-chat-history-content", + "task-chat-scroller", + "task-chat-live-transcript", + "task-chat-runner-turn", + "task-chat-turn-timeline", + "task-chat-turn-timeline-row", + "task-chat-activity-phase", + "task-chat-phase-interstitial" + ], + "stopControls": [ + { + "tag": "BUTTON", + "testId": "task-chat-composer-stop", + "ariaLabel": "Stop", + "disabled": null + } + ], + "limitation": "Recorded DOM locator verification; fresh live UI Stop still required." + }, + "canonical-task-route-check.json": { + "status": "passed", + "resourceCalls": 0, + "source": "actual owned fourth task API and normal visible UI after cleanup", + "issueId": "c6d400e9-031a-484e-a8d6-24194d1532a1", + "identifier": "CLO-4", + "title": "Real cloud Stop 1793747057656", + "requestedUuidUrl": "/CLO/issues/c6d400e9-031a-484e-a8d6-24194d1532a1", + "actualUrl": "/CLO/issues/CLO-4", + "headerTitleVisible": true, + "uuidInActualUrl": false, + "limitation": "Reproduces incorrect URL-only harness assertion; no Stop or streaming claim." + }, + "resourceCapFailureCleanup": { + "scope": "Offline actual-source admission and finally-cleanup sections; all SDK/filesystem effects stubbed, no cloud resources created", + "red": { + "exitCode": 1, + "rejectedBecause": "Only one of two already-admitted owned IDs reached the cleanup/watchdog ledger", + "sourceSha256": "44021e72ffe7f6d07a005fa9946d1e2af5d638dadc29f5c39a3080d5bd786854" + }, + "green": { + "status": "passed", + "cloudCalls": 0, + "observedOwnedResources": 2, + "declaredCap": 1, + "ledgeredBeforeFailure": 2, + "exactOwnedCleanupCalls": 2 + }, + "sourceSha256": "026fe92d5452062ddd5840a0bc9aeba9f6b4f7fda57040eaadb76ca80b960409" + } +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/cloud-ui.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/cloud-ui.mjs new file mode 100644 index 00000000000..3e9e5a73cef --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/cloud-ui.mjs @@ -0,0 +1,6 @@ +export function cloudStreamMarker(page, runId) { + if (!/^[a-f0-9-]{36}$/.test(runId)) throw new Error('invalid_owned_run_id'); + return page.getByTestId('task-chat-thread') + .getByTestId('task-chat-phase-interstitial') + .getByText('CLOUD-STOP-STREAM: real sandbox process is starting.', { exact: true }); +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/launch.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/launch.mjs new file mode 100644 index 00000000000..a8750bdbbba --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/launch.mjs @@ -0,0 +1,35 @@ +import { mkdtemp, mkdir, readFile, writeFile } from 'node:fs/promises'; +import { join } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { execFileSync } from 'node:child_process'; +import { createHash } from 'node:crypto'; +const [repository, revision, evidenceDir, remoteBinary, remoteCodex] = process.argv.slice(2); +if (Object.keys(process.env).some(key => /^(?:OP_|DAYTONA|BOXD)|API_KEY|TOKEN|SECRET/i.test(key))) throw new Error('credential_environment_not_stripped'); +if (execFileSync('git', ['rev-parse', 'HEAD'], { cwd: repository, encoding: 'utf8' }).trim() !== revision) throw new Error('source_revision_mismatch'); +const build = JSON.parse(await readFile(join(repository, 'server/dist/build-info.json'), 'utf8')); +if (build.commit !== revision) throw new Error('build_revision_mismatch'); +await mkdir(evidenceDir, { recursive: true }); +const dataDir = await mkdtemp('/private/tmp/paperclip-cloud-stop-data-'); +await writeFile(join(evidenceDir, 'data-ledger.json'), JSON.stringify({ dataDir, pid: process.pid })); +const { prepareTestDriveEnvironment, bootstrapTestDrive } = await import(pathToFileURL(join(repository, 'cli/src/commands/test-drive.ts')).href); +const options = { dataDir, companyName: 'Cloud Stop ingestion qualification', agentName: 'Cloud Stop agent', harness: 'codex', model: 'gpt-5.4', apiKey: 'qualification-deterministic-no-model', browser: false }; +const { linkedWorktree } = await prepareTestDriveEnvironment(options, repository); +Object.assign(process.env, { PAPERCLIP_UI_DEV_MIDDLEWARE: 'false', NODE_ENV: 'production', SERVE_UI: 'true', PAPERCLIP_RUNNER_REMOTE_BINARY_PATH: remoteBinary, PAPERCLIP_RUNNER_REMOTE_CODEX_PATH: remoteCodex }); +const { onboard } = await import(pathToFileURL(join(repository, 'cli/src/commands/onboard.ts')).href); +await onboard({ config: process.env.PAPERCLIP_CONFIG, yes: true, bind: 'loopback', invokedByRun: true, installService: false }); +const { doctor } = await import(pathToFileURL(join(repository, 'cli/src/commands/doctor.ts')).href); +if ((await doctor({ config: process.env.PAPERCLIP_CONFIG, repair: true, yes: true })).failed) throw new Error('doctor_failed'); +const { startServer } = await import(pathToFileURL(join(repository, 'server/dist/index.js')).href); +const server = await startServer(); +const origin = `http://${server.host}:${server.listenPort}`; +const { PaperclipApiClient } = await import(pathToFileURL(join(repository, 'cli/src/client/http.ts')).href); +const api = new PaperclipApiClient({ apiBase: origin }); +const fixture = await bootstrapTestDrive({ api, options, linkedWorktree, instanceId: process.env.PAPERCLIP_INSTANCE_ID, env: process.env }); +await api.patch('/api/instance/settings/experimental', { enableNativeRunner: true }); +const agent = await api.patch(`/api/agents/${fixture.agent.id}`, { adapterType: 'paperclip_runner', adapterConfig: { ...fixture.agent.adapterConfig, provider: 'codex', model: 'gpt-5.4', lifecycleMode: 'per_turn', timeoutSec: 180 }, runtimeConfig: { heartbeat: { enabled: false } } }); +const digest = async file => createHash('sha256').update(await readFile(file)).digest('hex'); +const config = { repository, origin, revision, build, evidenceDir, dataDir, pid: process.pid, company: { id: fixture.company.id, issuePrefix: fixture.company.issuePrefix }, agent: { id: agent.id, name: agent.name }, platform: { node: process.version, os: process.platform, architecture: process.arch }, hashes: { server: await digest(join(repository, 'server/dist/index.js')), ui: await digest(join(repository, 'ui/dist/index.html')), linuxRunner: await digest(remoteBinary), provider: await digest(remoteCodex) } }; +await writeFile(join(evidenceDir, 'launch.json'), JSON.stringify(config, null, 2)); +console.log(JSON.stringify({ ready: true, origin, revision, evidenceDir, pid: process.pid })); +let closing = false; +for (const signal of ['SIGINT', 'SIGTERM']) process.on(signal, async () => { if (closing) return; closing = true; await server.shutdown(signal); process.exit(0); }); diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/provider.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/provider.mjs new file mode 100644 index 00000000000..fa46f929757 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/provider.mjs @@ -0,0 +1,76 @@ +#!/opt/paperclip-runner/provider-pack/node_modules/node/bin/node +// Deterministic Codex protocol fixture; actual process execution/containment is real. +import { randomUUID } from 'node:crypto'; +import { spawn } from 'node:child_process'; +import { existsSync, readFileSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { createInterface } from 'node:readline'; +if (process.argv.includes('--version')) { console.log('codex-cli 0.115.0 (cloud-stop fixture)'); process.exit(0); } +const root = process.cwd(); +const statePath = join(root, '.cloud-stop-fixture.json'); +const effectPath = join(root, '.cloud-stop-forbidden-effect'); +const releasePath = join(root, '.cloud-stop-observer-release'); +let threadId = `cloud-stop-${randomUUID()}`, turnId, child, interrupted = false; +const turns = new Map(); +const send = value => process.stdout.write(`${JSON.stringify(value)}\n`); +const save = extra => writeFileSync(statePath, JSON.stringify({ providerPid: process.pid, threadId, turnId, cwd: root, effectPath, releasePath, ...extra }), { mode: 0o600 }); +const completed = status => { turns.set(turnId, status); send({ method: 'turn/completed', params: { threadId, turn: { id: turnId, status } } }); }; +async function run() { + save({ phase: 'awaiting-observer', effectPresent: existsSync(effectPath) }); + const deadline = Date.now() + 60000; + while (!existsSync(join(root, '.cloud-stop-start')) && Date.now() < deadline) await new Promise(resolve => setTimeout(resolve, 100)); + if (!existsSync(join(root, '.cloud-stop-start'))) throw new Error('cloud_stop_observer_not_armed'); + const itemId = `progress-${turnId}`; + send({ method: 'item/started', params: { threadId, turnId, item: { id: itemId, type: 'agentMessage', phase: 'commentary', text: '' } } }); + send({ method: 'item/agentMessage/delta', params: { threadId, turnId, itemId, delta: 'CLOUD-STOP-STREAM: real sandbox process is starting. ' } }); + send({ method: 'item/completed', params: { threadId, turnId, item: { id: itemId, type: 'agentMessage', phase: 'commentary', text: 'CLOUD-STOP-STREAM: real sandbox process is starting.' } } }); + child = spawn('/bin/sh', ['-c', 'sleep 60; printf forbidden > "$1"', 'cloud-stop-owned-child', effectPath], { cwd: root, stdio: 'ignore', detached: true }); + const stat = readFileSync(`/proc/${child.pid}/stat`, 'utf8'); + const fields = stat.slice(stat.lastIndexOf(') ') + 2).trim().split(/\s+/); + const identity = { pid: child.pid, ppid: Number(fields[1]), pgid: Number(fields[2]), startTicks: fields[19], bootId: readFileSync('/proc/sys/kernel/random/boot_id', 'utf8').trim() }; + save({ phase: 'sleeping', startedAtMs: Date.now(), child: identity, interrupted: false }); + send({ method: 'item/started', params: { threadId, turnId, item: { id: `command-${turnId}`, type: 'commandExecution', command: 'sleep 60', status: 'inProgress', cwd: root } } }); + child.on('exit', () => { + if (interrupted) return; + save({ phase: 'natural-completion', child: identity, effectPresent: existsSync(effectPath) }); + send({ method: 'item/completed', params: { threadId, turnId, item: { id: `command-${turnId}`, type: 'commandExecution', command: 'sleep 60', status: 'completed', exitCode: 0 } } }); + completed('completed'); + }); +} +async function interrupt(id) { + interrupted = true; + const state = existsSync(statePath) ? JSON.parse(readFileSync(statePath, 'utf8')) : {}; + if (child && child.exitCode === null && child.signalCode === null) { + const exited = new Promise(resolve => child.once('exit', resolve)); + try { process.kill(-child.pid, 'SIGTERM'); } catch (error) { if (error.code !== 'ESRCH') throw error; } + await exited; + } + save({ ...state, phase: 'interrupted-child-retired', interrupted: true, interruptedAtMs: Date.now(), effectPresent: existsSync(effectPath) }); + // Bound observer synchronization in this simulated provider's cleanup only. + // The real controller/Stop/lease lifecycle receives no special hooks. + const deadline = Date.now() + 5000; + while (!existsSync(releasePath) && Date.now() < deadline) await new Promise(resolve => setTimeout(resolve, 50)); + send({ id, result: {} }); + send({ method: 'item/completed', params: { threadId, turnId, item: { id: `command-${turnId}`, type: 'commandExecution', command: 'sleep 60', status: 'interrupted', exitCode: null } } }); + completed('interrupted'); +} +createInterface({ input: process.stdin }).on('line', line => { + const { id, method, params = {} } = JSON.parse(line); + if (id === undefined) return; + if (method === 'initialize') send({ id, result: { user: { sessionId: threadId } } }); + else if (['thread/start', 'thread/resume', 'thread/read'].includes(method)) { + if (params.threadId) threadId = params.threadId; + send({ id, result: { model: params.model ?? 'gpt-5.4', modelProvider: 'fixture', thread: { id: threadId, sessionId: threadId, status: { type: [...turns.values()].includes('inProgress') ? 'active' : 'idle' }, turns: [...turns].map(([id, status]) => ({ id, status })) } } }); + } else if (method === 'thread/turns/list') send({ id, result: { data: [...turns].map(([id, status]) => ({ id, status, items: [], itemsView: 'notLoaded' })), nextCursor: null } }); + else if (method === 'turn/start') { + turnId = randomUUID(); turns.set(turnId, 'inProgress'); + send({ id, result: { turn: { id: turnId, status: 'inProgress' } } }); + send({ method: 'turn/started', params: { threadId, turn: { id: turnId, status: 'inProgress' } } }); + const hello = JSON.stringify(params.input).includes('Respond only with hello. Do not use tools or inspect files.'); + setTimeout(() => { + if (hello) { send({ method: 'item/completed', params: { threadId, turnId, item: { id: `hello-${turnId}`, type: 'agentMessage', text: 'hello' } } }); completed('completed'); } + else void run().catch(() => { process.stderr.write('cloud_stop_fixture_start_failed\n'); process.exitCode = 1; completed('failed'); }); + }, 50); + } else if (method === 'turn/interrupt') void interrupt(id).catch(() => { process.stderr.write('cloud_stop_fixture_interrupt_failed\n'); process.exitCode = 1; }); + else send({ id, result: {} }); +}); diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/run.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/run.mjs new file mode 100644 index 00000000000..e1b07297549 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/run.mjs @@ -0,0 +1,266 @@ +import assert from 'node:assert/strict'; +import { createRequire } from 'node:module'; +import { execFileSync, fork } from 'node:child_process'; +import { readFile, writeFile, mkdir, readdir } from 'node:fs/promises'; +import { join, dirname } from 'node:path'; +import { pathToFileURL } from 'node:url'; +import { cloudStreamMarker } from './cloud-ui.mjs'; +const config = JSON.parse(await readFile(process.argv[2], 'utf8')); +const { repository, evidenceDir, origin } = config; +const priorIds = new Set(); +async function retainedResourceIds(dir) { for (const entry of await readdir(dir, { withFileTypes: true })) { const path = join(dir, entry.name); if (entry.isDirectory()) await retainedResourceIds(path); else if (entry.name === 'resource-ledger.json') for (const resource of JSON.parse(await readFile(path, 'utf8')).resources) priorIds.add(resource.providerLeaseId); } } +await retainedResourceIds(evidenceDir); +const priorResourceCount = priorIds.size; +const RESOURCE_CAP = 1; // One fresh sandbox per reproduction; no automatic retry. +assert.ok(priorResourceCount < RESOURCE_CAP, 'whole_qualification_cloud_resource_bound'); +const IMAGE = 'ghcr.io/paperclipai/paperclip-daytona-runner@sha256:bff4c3f291087a0eeae37e4c20dd51857b92833eaf73ba3aca4157f37de1e109'; +const NODE = '/opt/paperclip-runner/provider-pack/node_modules/node/bin/node'; +const require = createRequire(join(repository, 'package.json')); +const { chromium, expect } = require('@playwright/test'); +const pluginRequire = createRequire(join(repository, 'packages/plugins/sandbox-providers/daytona/package.json')); +const sdkEntry = pluginRequire.resolve('@daytonaio/sdk'); +const sdkPackage = JSON.parse(await readFile(join(dirname(dirname(sdkEntry)), 'package.json'), 'utf8')); +assert.equal(sdkPackage.version, '0.203.0'); +const key = process.env.DAYTONA_API_KEY?.trim(); +assert.ok(key && key.length > 12, 'credential_binding_empty'); +for (const name of Object.keys(process.env)) if (/^(OP_|DAYTONA|BOXD)|API_KEY|TOKEN|SECRET/i.test(name)) delete process.env[name]; +const { Daytona } = await import(pathToFileURL(sdkEntry).href); +const daytona = new Daytona({ apiKey: key, apiUrl: 'https://app.daytona.io/api' }); +const safeError = error => ({ name: String(error?.name ?? 'Error').replace(/[^A-Za-z0-9_]/g, '').slice(0, 60), status: Number(error?.statusCode ?? error?.status ?? error?.response?.status) || null, code: /^[-a-zA-Z0-9_]+$/.test(error?.code ?? '') ? error.code : null }); +const redact = text => String(text).replaceAll(key, '[REDACTED]'); +const sensitiveValues = new Set([key]); +function sanitized(value) { + if (typeof value === 'string') return redact(value); + if (Array.isArray(value)) return value.map(sanitized); + if (value && typeof value === 'object') return Object.fromEntries(Object.entries(value).map(([name, entry]) => { + if (/api.?key|authorization|password|secret|token$|^headers$/i.test(name)) { + if (typeof entry === 'string' && entry.length > 12) sensitiveValues.add(entry); + return [name, '[REDACTED]']; + } + return [name, sanitized(entry)]; + })); + return value; +} +const summary = { schema: 'paperclip.real-cloud-stop-qualification.v1', status: 'running', startedAt: new Date().toISOString(), revision: config.revision, build: config.build, hashes: config.hashes, platform: config.platform, dependencies: { provider: 'deterministic Codex app-server fixture with real sleep60; no model call', sandbox: 'real Daytona', sdk: sdkPackage.version, image: IMAGE, controller: 'normal production-built local Paperclip', actualLlmJourney: 'separate real OpenCode/DeepSeek acceptance' }, checks: {}, cleanup: {} }; +let phase = 'bootstrap'; +summary.priorResourceCount = priorResourceCount; +const ledgerPath = join(evidenceDir, 'resource-ledger.json'); +const ledger = { startedAt: summary.startedAt, hardDeadlineAt: Date.now() + 30 * 60_000, resources: [] }; +await writeFile(ledgerPath, JSON.stringify(ledger, null, 2)); +const cleanEnv = { PATH: process.env.PATH, HOME: process.env.HOME, TMPDIR: process.env.TMPDIR }; +const watchdog = fork(join(import.meta.dirname, 'watchdog.mjs'), [repository, ledgerPath, join(evidenceDir, 'watchdog-receipt.json')], { env: cleanEnv, stdio: ['ignore', 'ignore', 'ignore', 'ipc'], execArgv: [] }); +const armed = new Promise((resolve, reject) => { watchdog.once('message', resolve); watchdog.once('exit', () => reject(new Error('watchdog_failed_before_arm'))); }); +watchdog.send({ key, deadlineAt: ledger.hardDeadlineAt }); +await armed; +summary.watchdog = { armed: true, deadlineAt: ledger.hardDeadlineAt, pid: watchdog.pid }; +const save = () => writeFile(join(evidenceDir, 'qualification-summary.json'), JSON.stringify(sanitized(summary), null, 2)); +const evidence = async (name, data) => writeFile(join(evidenceDir, name), JSON.stringify(sanitized(data), null, 2)); +const pause = ms => new Promise(resolve => setTimeout(resolve, ms)); +async function api(method, route, body) { + const response = await fetch(new URL(route, origin), { method, headers: body === undefined ? {} : { 'Content-Type': 'application/json' }, body: body === undefined ? undefined : JSON.stringify(body), signal: AbortSignal.timeout(120_000) }); + if (!response.ok) { const error = new Error(`api_${method}_${response.status}_${route}`); error.status = response.status; throw error; } + return response.status === 204 ? null : response.json(); +} +async function poll(label, load, accept, timeout = 120_000, interval = 250) { + const deadline = Date.now() + timeout; + while (Date.now() < deadline) { const result = await load(); if (accept(result)) return result; await pause(interval); } + throw new Error(`timeout_${label}`); +} +const quote = value => `'${String(value).replaceAll("'", "'\\''")}'`; +async function remote(sandbox, code, cwd) { + const command = `/usr/bin/env -i PATH=/usr/bin:/bin ${quote(NODE)} -e ${quote(code)}`; + const result = await sandbox.process.executeCommand(command, cwd, {}, 10); + assert.equal(result.exitCode, 0, 'remote_observer_nonzero'); + assert.ok(typeof result.result === 'string' && result.result.length < 65536, 'remote_observer_bound'); + return JSON.parse(result.result); +} +const expectedLabels = runId => ({ 'paperclip-provider': 'daytona', 'paperclip-company-id': config.company.id, 'paperclip-environment-id': environment.id, 'paperclip-run-id': runId, 'paperclip-reuse-lease': 'false' }); +const assertOwned = (sandbox, row) => { assert.equal(sandbox.id, row.providerLeaseId); for (const [key, value] of Object.entries(row.expectedLabels)) assert.equal(sandbox.labels?.[key], value, `provider_label_${key}`); }; +async function providerState(row) { + try { const sandbox = await daytona.get(row.providerLeaseId); assertOwned(sandbox, row); return { id: sandbox.id, state: String(sandbox.state), labels: Object.fromEntries(Object.keys(row.expectedLabels).map(key => [key, sandbox.labels[key]])), observedAt: Date.now() }; } + catch (error) { if (Number(error?.status ?? error?.response?.status) === 404 || error?.name === 'DaytonaNotFoundError') return { id: row.providerLeaseId, state: 'absent', observedAt: Date.now() }; throw error; } +} +async function eventsFor(runId) { + const rows = []; + while (true) { const page = await api('GET', `/api/heartbeat-runs/${runId}/events?afterSeq=${rows.at(-1)?.seq ?? 0}&limit=1000`); rows.push(...page); if (page.length < 1000) return rows; } +} +async function runsFor(issueId) { + const rows = await api('GET', `/api/companies/${config.company.id}/heartbeat-runs?agentId=${config.agent.id}&limit=100`); + const runs = await Promise.all(rows.map(row => api('GET', `/api/heartbeat-runs/${row.id}`))); + return runs.filter(run => run.nativeIssueId === issueId || run.contextSnapshot?.issueId === issueId); +} +let environment, browser, context, page, issueId; +try { + await api('PATCH', '/api/instance/settings/experimental', { enableRunnerPreviewIngress: true }); + const installed = await api('GET', '/api/plugins'); + const plugin = installed.find(value => value.pluginKey === 'paperclip.daytona-sandbox-provider') ?? await api('POST', '/api/plugins/install', { packageName: join(repository, 'packages/plugins/sandbox-providers/daytona'), isLocalPath: true }); + summary.plugin = { id: plugin.id, pluginKey: plugin.pluginKey, version: plugin.manifest?.version }; + await poll('plugin_ready', async () => { + const dashboard = await api('GET', '/api/plugins/paperclip.daytona-sandbox-provider/dashboard'); + const caps = await api('GET', `/api/companies/${config.company.id}/environments/capabilities`); + return { dashboard, provider: caps.sandboxProviders?.daytona }; + }, value => value.dashboard.worker?.status === 'running' && value.provider?.status === 'supported' && value.provider.pluginKey === 'paperclip.daytona-sandbox-provider'); + const secret = await api('POST', `/api/companies/${config.company.id}/secrets`, { name: `Bounded real cloud Stop Daytona key ${Date.now()}`, key: `daytona_cloud_stop_${Date.now()}`, value: key }); + environment = await api('POST', `/api/companies/${config.company.id}/environments`, { name: 'Bounded real cloud Stop', description: 'Owned qualification; hardTTL30m; one sandbox', driver: 'sandbox', config: { provider: 'daytona', apiKey: { type: 'secret_ref', secretId: secret.id, version: 'latest' }, image: IMAGE, cpu: 4, memory: 4, disk: 10, reuseLease: false, runnerLifecycleMode: 'per_turn', autoStopInterval: 5, autoArchiveInterval: 15, autoDeleteInterval: 30, ttlMinutes: 30, timeoutMs: 300000, livenessTimeoutMs: 30000 }, envVars: {} }); + summary.environment = { id: environment.id, config: { image: IMAGE, cpu: 4, memory: 4, disk: 10, reuseLease: false, runnerLifecycleMode: 'per_turn', autoStopInterval: 5, autoArchiveInterval: 15, autoDeleteInterval: 30, ttlMinutes: 30 } }; + const agent = await api('PATCH', `/api/agents/${config.agent.id}`, { defaultEnvironmentId: environment.id }); + assert.equal(agent.defaultEnvironmentId, environment.id); + summary.checks.normalApiBindings = true; + await save(); + browser = await chromium.launch({ headless: true }); + context = await browser.newContext({ viewport: { width: 1440, height: 1000 }, recordVideo: { dir: join(evidenceDir, 'video'), size: { width: 1440, height: 1000 } } }); + await context.tracing.start({ screenshots: true, snapshots: true }); + page = await context.newPage(); + const { createTaskThroughUi } = await import(pathToFileURL(join(repository, 'tests/runner-e2e/user-actions.ts')).href); + const created = await createTaskThroughUi({ page, origin, issuePrefix: config.company.issuePrefix, agentName: config.agent.name, title: `Real cloud Stop ${Date.now()}`, prompt: 'CLOUD-STOP qualification: stream progress, then run sleep 60. The operator will Stop this task before it completes. Do not finish early.', workMode: 'standard', requireExplicitTitle: true }); + issueId = created.issueId; summary.issueId = issueId; + await page.goto(new URL(`/${config.company.issuePrefix}/issues/${issueId}`, origin).href); + const admitted = await poll('one_native_run_lease', async () => { + const runs = await runsFor(issueId); + const leases = await api('GET', `/api/environments/${environment.id}/leases`); + // Ledger every admitted host-owned ID before further observer work. + for (const lease of leases) if (lease.heartbeatRunId && lease.providerLeaseId && !ledger.resources.some(row => row.providerLeaseId === lease.providerLeaseId)) { + const row = { providerLeaseId: lease.providerLeaseId, leaseId: lease.id, runId: lease.heartbeatRunId, expectedLabels: expectedLabels(lease.heartbeatRunId), admittedAt: Date.now() }; + const sandbox = await daytona.get(row.providerLeaseId); assertOwned(sandbox, row); + ledger.resources.push(row); await writeFile(ledgerPath, JSON.stringify(ledger, null, 2)); + console.log(JSON.stringify({ resourceAdmitted: row.providerLeaseId, leaseId: row.leaseId, runId: row.runId, ttlMinutes: 30 })); + } + // These resources already exist. Record every observed owned ID before a + // bound failure so finally/watchdog cleanup can retire all of them. + assert.ok(priorResourceCount + ledger.resources.length <= RESOURCE_CAP, 'cloud_resource_bound'); + if (runs.length !== 1) return null; + if (!['running', 'queued'].includes(runs[0].status)) throw new Error('native_run_terminal_before_admission'); + const lease = leases.find(row => row.heartbeatRunId === runs[0].id && row.status === 'active' && row.providerLeaseId && row.metadata?.remoteCwd); + return lease ? { run: runs[0], lease } : null; + }, Boolean, 360000); + const { run, lease } = admitted; summary.runId = run.id; summary.leaseId = lease.id; summary.providerLeaseId = lease.providerLeaseId; + assert.equal(lease.metadata.image, IMAGE); assert.equal(lease.metadata.reuseLease, false); + const row = ledger.resources.find(row => row.providerLeaseId === lease.providerLeaseId); + assert.equal(ledger.resources.length, 1, 'unexpected_second_sandbox'); + const sandbox = await daytona.get(row.providerLeaseId); assertOwned(sandbox, row); + const cwd = lease.metadata.remoteCwd; + assert.ok(typeof cwd === 'string' && cwd.startsWith('/') && cwd !== '/' && cwd.length < 512); + phase = 'runtime_verification'; + const runtime = await poll('staged_runtime_before_test', () => remote(sandbox, `const fs=require('node:fs'),crypto=require('node:crypto'),cp=require('node:child_process');const root=${JSON.stringify(cwd)},runner=root+'/.paperclip-runtime/paperclip-runner/bin/paperclip-runnerd',fixture=root+'/.paperclip-runtime/paperclip-runner/bin/codex';if(!fs.existsSync(runner)||!fs.existsSync(fixture)||!fs.existsSync(root+'/.cloud-stop-fixture.json')){console.log('null');process.exit(0)}const sha=p=>crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex');console.log(JSON.stringify({node:process.execPath,nodeVersion:process.version,nodeSha256:sha(process.execPath),runnerSha256:sha(runner),fixtureSha256:sha(fixture),runnerMetadata:JSON.parse(cp.execFileSync(runner,['--build-metadata'],{encoding:'utf8',env:{PATH:'/usr/bin:/bin'}})),fixtureState:JSON.parse(fs.readFileSync(root+'/.cloud-stop-fixture.json','utf8')).phase}));`, cwd), value => value?.fixtureState === 'awaiting-observer', 60000, 200); + assert.equal(runtime.runnerSha256, config.hashes.linuxRunner); assert.equal(runtime.fixtureSha256, config.hashes.provider); + const version = runtime.nodeVersion.match(/^v(\d+)\.(\d+)\./); assert.ok(version && (Number(version[1]) > 24 || Number(version[1]) === 24 && Number(version[2]) >= 11), 'image_node_minimum'); + summary.runtime = runtime; summary.checks.currentDaemonAndFixture = true; + const nativeRun = await poll('native_running_before_test', () => api('GET', `/api/heartbeat-runs/${run.id}`), value => value.runtimeMode === 'native' && value.status === 'running', 30000); + summary.nativeAdmission = { runtimeMode: nativeRun.runtimeMode, status: nativeRun.status, runtimeModeReason: nativeRun.runtimeModeReason }; + await remote(sandbox, `require('node:fs').writeFileSync(${JSON.stringify(cwd + '/.cloud-stop-start')},'verified',{mode:0o600});console.log('{"started":true}');`, cwd); + const snapshotCode = `const fs=require('node:fs');const root=${JSON.stringify(cwd)},p=root+'/.cloud-stop-fixture.json';if(!fs.existsSync(p)){console.log('null');process.exit(0)}const s=JSON.parse(fs.readFileSync(p,'utf8'));let live=false,children=[];if(s.child){try{const t=fs.readFileSync('/proc/'+s.child.pid+'/stat','utf8'),f=t.slice(t.lastIndexOf(') ')+2).trim().split(/\\s+/);live=f[19]===s.child.startTicks&&f[0]!=='Z'}catch{}for(const n of fs.readdirSync('/proc')){if(!/^\\d+$/.test(n))continue;try{const t=fs.readFileSync('/proc/'+n+'/stat','utf8'),f=t.slice(t.lastIndexOf(') ')+2).trim().split(/\\s+/);if(Number(f[2])===s.child.pgid&&f[0]!=='Z')children.push({pid:Number(n),ppid:Number(f[1]),pgid:Number(f[2]),startTicks:f[19],command:fs.readFileSync('/proc/'+n+'/cmdline','utf8').replaceAll('\\0',' ')})}catch{}}}console.log(JSON.stringify({observedAtMs:Date.now(),state:s,childLive:live,groupProcesses:children,effectPresent:fs.existsSync(root+'/.cloud-stop-forbidden-effect')}));`; + const active = await poll('actual_sleep', () => remote(sandbox, snapshotCode, cwd), value => value?.state.phase === 'sleeping' && value.childLive && value.groupProcesses.some(child => /sleep 60/.test(child.command)), 60000, 150); + assert.equal(active.effectPresent, false); summary.actualProcessBeforeStop = active; + phase = 'visible_streaming'; await save(); + const marker = cloudStreamMarker(page, run.id); + const visible = async () => [await marker.innerText()]; + await expect(marker).toBeVisible({ timeout: 10000 }); + assert.equal((await api('GET', `/api/heartbeat-runs/${run.id}`)).status, 'running'); + summary.checks.visibleStreamingBeforeTerminal = true; summary.checks.realSleepObserved = true; + const freshRuns = await runsFor(issueId); + assert.equal(freshRuns.length, 1, 'fresh_owned_task_has_only_one_run'); + assert.equal(freshRuns[0].id, run.id); assert.equal(freshRuns[0].runtimeMode, 'native'); assert.equal(freshRuns[0].status, 'running'); + const sourceMarker = (await eventsFor(run.id)).map(row => row.payload?.prpEvent).find(event => event?.runId === run.id && event.eventType === 'item.delta' && event.payload.kind === 'agentMessage' && event.payload.text?.trim() === 'CLOUD-STOP-STREAM: real sandbox process is starting.'); + assert.ok(sourceMarker?.sourceEventId && sourceMarker.sourceSeq > 0, 'actual_current_run_normalized_agent_message_marker'); + summary.visibleSourceMarker = { eventType: sourceMarker.eventType, kind: sourceMarker.payload.kind, text: sourceMarker.payload.text, sourceEventId: sourceMarker.sourceEventId, sourceSeq: sourceMarker.sourceSeq }; + await page.screenshot({ path: join(evidenceDir, 'before-stop.png'), fullPage: true }); + await evidence('before-stop.json', { lease: { ...lease, metadata: { ...lease.metadata, workspaceSentinel: undefined } }, active, runtime, events: await eventsFor(run.id) }); + // Independent observer captures retirement before the deterministic provider + // fixture in REAL Daytona emits its interrupted terminal. No server hook or + // provider receipt is fabricated. + const retirement = poll('independent_child_retirement', () => remote(sandbox, snapshotCode, cwd), value => value?.state.phase === 'interrupted-child-retired' && !value.childLive && value.groupProcesses.length === 0 && !value.effectPresent, 20000, 100).then(async value => { + summary.retirement = value; + await remote(sandbox, `const fs=require('node:fs');fs.writeFileSync(${JSON.stringify(cwd + '/.cloud-stop-observer-release')},'observed',{mode:0o600});console.log('{"released":true}');`, cwd); + return value; + }); + void retirement.catch(() => {}); + const stopButton = page.getByRole('button', { name: 'Stop', exact: true }); + await expect(stopButton).toHaveCount(1); await expect(stopButton).toBeVisible(); await expect(stopButton).toBeEnabled(); + const immediate = await remote(sandbox, snapshotCode, cwd); + assert.equal(immediate.state.child.pid, active.state.child.pid); + assert.equal(immediate.state.child.startTicks, active.state.child.startTicks); + assert.equal(immediate.state.child.bootId, active.state.child.bootId); + assert.ok(immediate.childLive && immediate.groupProcesses.some(child => child.command.trim() === 'sleep 60'), 'independent_sleep_active_immediately_before_stop'); + assert.equal(immediate.effectPresent, false); + const observedProcessAgeMs = immediate.observedAtMs - immediate.state.startedAtMs; + assert.ok(observedProcessAgeMs >= 0 && observedProcessAgeMs < 20000, 'sleep_birth_age_immediately_before_stop_under_20s'); + summary.immediateBeforeStop = immediate; + summary.stopRequestedAt = Date.now(); + phase = 'visible_stop_dispatch'; await save(); + const responsePromise = page.waitForResponse(response => response.request().method() === 'POST' && response.url().endsWith(`/api/heartbeat-runs/${run.id}/cancel`), { timeout: 30000 }); + await stopButton.click(); + const response = await responsePromise; + assert.equal(response.status(), 200, 'visible_stop_response'); + summary.originalStopAcknowledgement = await response.json(); + await retirement; + const stopped = await poll('native_and_shared_ack', () => api('GET', `/api/heartbeat-runs/${run.id}`), value => value.status === 'cancelled' && value.resultJson?.nativeCancellation?.dispatchState === 'acknowledged' && value.resultJson?.executionCancellation?.state === 'acknowledged', 120000); + summary.stopSettledAt = Date.now(); + assert.equal(stopped.resultJson.executionCancellation.proof, 'provider_termination_receipt'); + assert.ok(stopped.resultJson.executionCancellation.acknowledgedAt); + summary.nativeCancellation = stopped.resultJson.nativeCancellation; summary.executionCancellation = stopped.resultJson.executionCancellation; + summary.checks.nativeAcknowledgement = true; summary.checks.sharedProviderReceiptAcknowledgement = true; summary.checks.independentChildRetirementNoEffect = true; + const terminalLeases = (await api('GET', `/api/environments/${environment.id}/leases`)).filter(row => row.heartbeatRunId === run.id); + assert.equal(terminalLeases.length, 1); + for (const lease of terminalLeases) { + assert.ok(lease.releasedAt); assert.equal(lease.cleanupStatus, 'success'); assert.ok(['released', 'expired', 'failed'].includes(lease.status)); + const receipt = lease.metadata?.remoteExecutionTermination; + assert.equal(receipt?.schema, 'paperclip.remote-termination.v1'); + for (const [key, expected] of Object.entries({ companyId: config.company.id, runId: run.id, leaseId: lease.id, provider: 'daytona', providerLeaseId: row.providerLeaseId })) assert.equal(receipt[key], expected); + assert.ok(['stopped', 'destroyed'].includes(receipt.state)); summary.providerReceipt = receipt; + } + summary.independentProviderState = await poll('provider_stopped', () => providerState(row), value => ['stopped', 'archived', 'destroyed', 'absent'].includes(value.state), 120000); + summary.checks.exactLeaseProviderReceipt = true; summary.checks.independentProviderStopped = true; + await page.reload({ waitUntil: 'domcontentloaded' }); + await expect(page.getByTestId('issue-detail-header')).toBeVisible(); + await expect(page.getByTestId('task-chat-history-loading')).toHaveCount(0); + await expect.poll(async () => (await visible()).join('\n'), { timeout: 15000 }).toContain('CLOUD-STOP-STREAM'); + await page.screenshot({ path: join(evidenceDir, 'after-stop-reload.png'), fullPage: true }); + await pause(3000); + const finalRuns = await runsFor(issueId); assert.equal(finalRuns.length, 1); assert.equal(finalRuns[0].id, run.id); assert.equal(finalRuns[0].status, 'cancelled'); + const events = await eventsFor(run.id); + assert.ok(!events.some(row => row.payload?.prpEvent?.eventType === 'run.result.accepted'), 'no_result_after_stop'); + await evidence('persisted-api-state.json', { run: finalRuns[0], events, leases: terminalLeases, issue: await api('GET', `/api/issues/${issueId}`) }); + summary.eventCount = events.length; summary.checks.reloadAndNoSuccessorOrResult = true; + summary.status = 'passed'; +} catch (error) { + summary.status = 'failed'; summary.failedPhase = phase; summary.error = safeError(error); + summary.error.message = redact(String(error?.message ?? 'controlled_unknown_failure')).replace(/(X-Daytona-Preview-Token|Authorization|apiKey|previewToken)["']?\s*[:=]\s*["']?[^,\s}"']+/gi, '$1=[REDACTED]').replace(/Bearer\s+[^\s"']+/gi, 'Bearer [REDACTED]').slice(0, 800); + // Our authored assertion messages contain only fixed keys/codes. SDK errors + // remain primitives only; never retain Axios config, headers or raw objects. + if (error instanceof assert.AssertionError || /^timeout_|^native_|^api_/.test(error?.message ?? '')) summary.error.message = redact(error.message).slice(0, 600); + await page?.screenshot({ path: join(evidenceDir, 'failure.png'), fullPage: true }).catch(() => {}); +} finally { + if (issueId && summary.status !== 'passed') { + try { const runs = await runsFor(issueId); for (const run of runs) if (['running', 'queued'].includes(run.status)) await api('POST', `/api/heartbeat-runs/${run.id}/cancel`); } catch (error) { summary.cleanup.cancelError = safeError(error); } + } + if (environment) { + try { await api('DELETE', `/api/environments/${environment.id}?destroyReusableSandboxLeases=true`); summary.cleanup.normalEnvironmentDelete = true; } catch (error) { summary.cleanup.normalEnvironmentDelete = false; summary.cleanup.normalDeleteError = safeError(error); } + } + for (const row of ledger.resources) { + try { + let state = await providerState(row); + if (state.state !== 'absent') { const sandbox = await daytona.get(row.providerLeaseId); assertOwned(sandbox, row); await daytona.delete(sandbox, 90); state = await poll('provider_absent_cleanup', () => providerState(row), value => value.state === 'absent', 90000); summary.cleanup.exactOwnedFallbackDelete = true; } + (summary.cleanup.providerAbsence ??= []).push(state); + } catch (error) { summary.cleanup.providerCleanupFailed = safeError(error); summary.status = 'failed'; } + } + if (!summary.cleanup.providerCleanupFailed) { watchdog.send('disarm'); summary.cleanup.watchdogDisarmed = true; } + await context?.tracing.stop({ path: join(evidenceDir, 'trace.zip') }).catch(() => {}); + await context?.close().catch(() => {}); await browser?.close().catch(() => {}); + summary.finishedAt = new Date().toISOString(); + await save(); + // Exact credential scan before any evidence leaves this owned directory. + const scan = { files: 0, matches: [] }; + const sensitive = [...sensitiveValues]; + function inspect(bytes, path) { scan.files++; if (sensitive.some(value => bytes.includes(Buffer.from(value)))) scan.matches.push(path); } + async function scanDirectory(dir) { for (const entry of await readdir(dir, { withFileTypes: true })) { const path = join(dir, entry.name); if (entry.isDirectory()) await scanDirectory(path); else if (entry.isFile()) { + inspect(await readFile(path), path); + if (path.endsWith('.zip')) { + const entries = execFileSync('/usr/bin/unzip', ['-Z1', path], { encoding: 'utf8', maxBuffer: 8 * 1024 * 1024 }).trim().split('\n').filter(value => value && !value.endsWith('/')); + for (const entry of entries) inspect(execFileSync('/usr/bin/unzip', ['-p', path, entry], { maxBuffer: 64 * 1024 * 1024 }), path + '!' + entry); + } + } } } + await scanDirectory(evidenceDir); + summary.credentialScan = scan; if (scan.matches.length) summary.status = 'failed'; await save(); + console.log(JSON.stringify({ status: summary.status, summary: join(evidenceDir, 'qualification-summary.json'), runId: summary.runId, providerLeaseId: summary.providerLeaseId, checks: summary.checks, cleanup: summary.cleanup, error: summary.error })); +} +if (summary.status !== 'passed') process.exitCode = 1; diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/verify-failure-cleanup.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/verify-failure-cleanup.mjs new file mode 100644 index 00000000000..b6287ccd3ba --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/verify-failure-cleanup.mjs @@ -0,0 +1,50 @@ +// Offline actual-source regression: an already-admitted resource beyond the +// cap must remain reachable by the real failure-cleanup loop. No cloud calls. +import assert from 'node:assert/strict'; +import { readFile } from 'node:fs/promises'; + +const source = await readFile(new URL('./run.mjs', import.meta.url), 'utf8'); +function section(start, end) { + assert.equal(source.split(start).length, 2, `Unique source seam: ${start}`); + const offset = source.indexOf(start); + const stop = source.indexOf(end, offset); + assert.ok(stop > offset, `Missing end seam: ${end}`); + return source.slice(offset, stop); +} +const admission = section(' for (const lease of leases)', ' if (runs.length !== 1)'); +const cleanup = section(' for (const row of ledger.resources)', ' if (!summary.cleanup.providerCleanupFailed)'); +const AsyncFunction = Object.getPrototypeOf(async function () {}).constructor; +const ledger = { resources: [] }; +const leases = [1, 2].map(index => ({ heartbeatRunId: `run-${index}`, providerLeaseId: `provider-${index}`, id: `lease-${index}` })); +const expectedLabels = runId => ({ owner: 'offline-test', runId }); +const states = new Map(leases.map(lease => [lease.providerLeaseId, 'active'])); +const deletions = []; +let persistedLedger; +const daytona = { + async get(id) { + const lease = leases.find(lease => lease.providerLeaseId === id); + assert.ok(lease, 'Only the exact test resources can be read'); + return { id, labels: expectedLabels(lease.heartbeatRunId) }; + }, + async delete(sandbox) { deletions.push(sandbox.id); states.set(sandbox.id, 'absent'); }, +}; +const assertOwned = (sandbox, row) => { + assert.equal(sandbox.id, row.providerLeaseId); + assert.deepEqual(sandbox.labels, row.expectedLabels); +}; +const writeFile = async (_path, text) => { persistedLedger = JSON.parse(text); }; +await assert.rejects(new AsyncFunction( + 'leases', 'ledger', 'priorResourceCount', 'RESOURCE_CAP', 'daytona', 'expectedLabels', + 'assertOwned', 'writeFile', 'ledgerPath', 'assert', 'console', admission, +)(leases, ledger, 0, 1, daytona, expectedLabels, assertOwned, writeFile, 'offline-ledger', assert, { log() {} }), /cloud_resource_bound/); +assert.equal(persistedLedger.resources.length, 2, 'Every observed owned ID must reach the watchdog ledger'); +const summary = { cleanup: {} }; +const providerState = async row => ({ id: row.providerLeaseId, state: states.get(row.providerLeaseId) }); +const poll = async (_label, load, accept) => { const value = await load(); assert.ok(accept(value)); return value; }; +await new AsyncFunction( + 'ledger', 'summary', 'providerState', 'daytona', 'assertOwned', 'poll', 'safeError', cleanup, +)(ledger, summary, providerState, daytona, assertOwned, poll, error => ({ message: error.message })); +assert.deepEqual(deletions, leases.map(lease => lease.providerLeaseId)); +assert.ok(summary.cleanup.providerAbsence.every(row => row.state === 'absent')); +assert.equal(summary.cleanup.providerCleanupFailed, undefined); +console.log(JSON.stringify({ status: 'passed', cloudCalls: 0, observedOwnedResources: 2, declaredCap: 1, ledgeredBeforeFailure: 2, exactOwnedCleanupCalls: 2 })); diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/watchdog.mjs b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/watchdog.mjs new file mode 100644 index 00000000000..f8eb0fb5ad3 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/helpers/watchdog.mjs @@ -0,0 +1,30 @@ +import { createRequire } from 'node:module'; +import { readFile, writeFile } from 'node:fs/promises'; +import { join } from 'node:path'; +import { pathToFileURL } from 'node:url'; +const [repository, ledgerPath, outputPath] = process.argv.slice(2); +process.once('message', async ({ key, deadlineAt }) => { + const require = createRequire(join(repository, 'packages/plugins/sandbox-providers/daytona/package.json')); + const { Daytona } = await import(pathToFileURL(require.resolve('@daytonaio/sdk')).href); + const client = new Daytona({ apiKey: key, apiUrl: 'https://app.daytona.io/api' }); + let done = false; + process.on('message', value => { if (value === 'disarm') { done = true; process.exit(0); } }); + process.send({ armed: true, deadlineAt }); + setTimeout(async () => { + if (done) return; + const results = []; + let ledger; + try { ledger = JSON.parse(await readFile(ledgerPath, 'utf8')); } catch { ledger = { resources: [] }; } + for (const row of ledger.resources) { + try { + const sandbox = await client.get(row.providerLeaseId); + const owned = Object.entries(row.expectedLabels).every(([key, value]) => sandbox.labels?.[key] === value); + if (!owned) { results.push({ id: row.providerLeaseId, status: 'ownership_mismatch' }); continue; } + await client.delete(sandbox, 90); + results.push({ id: row.providerLeaseId, status: 'deleted' }); + } catch (error) { results.push({ id: row.providerLeaseId, status: Number(error?.status ?? error?.response?.status) === 404 ? 'already_absent' : 'cleanup_failed' }); } + } + await writeFile(outputPath, JSON.stringify({ firedAt: new Date().toISOString(), results }, null, 2)); + process.exit(results.some(row => row.status === 'cleanup_failed' || row.status === 'ownership_mismatch') ? 1 : 0); + }, Math.max(1, deadlineAt - Date.now())); +}); diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/manifest.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/manifest.json new file mode 100644 index 00000000000..6f78dc0cfee --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/manifest.json @@ -0,0 +1,94 @@ +{ + "schema": "paperclip.acceptance-artifact-manifest.v1", + "scope": "Real Daytona Stop functional acceptance; no paired performance claim", + "testedRevision": "137501f595fd5ee13d395e26b4147ca8bc4d4eb6", + "files": [ + { + "path": "README.md", + "bytes": 6340, + "sha256": "1b5f3dfd236c083c212003d0bafd4b39bec6282f72de6a3f62bdd72f7bf22664" + }, + { + "path": "after-stop-reload.png", + "bytes": 143333, + "sha256": "6b53cc2cb40001cdda9cf34bb276c5369a46ed963d89f534314ae73c094fc30d" + }, + { + "path": "attempts.json", + "bytes": 3844, + "sha256": "fccf05434a4ed4169a95fe00ec126c2273ef850fc15549fa0a82594278e91f60" + }, + { + "path": "before-stop.png", + "bytes": 206637, + "sha256": "5e1c27da8c03aa6167526fe13786aae82cc8be52e67c7a07fe2a0e75e004e006" + }, + { + "path": "cleanup-and-scan.json", + "bytes": 1704, + "sha256": "7f118cd53a12568f08cc936315e84217e518b164b6096a7f50fbd761e9c77f98" + }, + { + "path": "database-proof-all-attempts.json", + "bytes": 3932, + "sha256": "119a4aee83b2677753242ba7263eeef37c1173ae0fec2d4794b2fe99eb9eaef0" + }, + { + "path": "helper-checks.json", + "bytes": 3750, + "sha256": "3af2aaa7073a720b999c48efeee20f367f1fb1d063801f3c53c9a443a43fdc1c" + }, + { + "path": "helpers/cloud-ui.mjs", + "bytes": 314, + "sha256": "c43c112546709fca064842c529d27bc78363ffdd2ca4258830dd17c85affb6cc" + }, + { + "path": "helpers/launch.mjs", + "bytes": 4005, + "sha256": "b6124d665002da53628d3483452cbf30aa8b1acf5b759134c0298d13a5cd4aac" + }, + { + "path": "helpers/provider.mjs", + "bytes": 6159, + "sha256": "b62bd845e63a2736a76bcf834c2a2bd281db46079d5e5c786c7e996f9a9ef269" + }, + { + "path": "helpers/run.mjs", + "bytes": 26810, + "sha256": "026fe92d5452062ddd5840a0bc9aeba9f6b4f7fda57040eaadb76ca80b960409" + }, + { + "path": "helpers/watchdog.mjs", + "bytes": 1849, + "sha256": "cbe0770ef67217a609b13ba5a557ee57dc9440180658314ff38919a81648fd2a" + }, + { + "path": "qualification.json", + "bytes": 10731, + "sha256": "f810543654f373ad4472dd441da3c8d7ea9ad6a8ebfa49a285a0d1ca8cf1b83e" + }, + { + "path": "helpers/verify-failure-cleanup.mjs", + "bytes": 3135, + "sha256": "5b32c076598087cbba2fb51bee16514d3be163bcdc428563385dbd2b46ca9996" + } + ], + "checks": { + "allFiveExactProviderIdsFresh404At": "2026-11-03T23:14:03.153Z", + "retainedAndDecompressedCredentialScanEntries": 3084, + "credentialMatches": 0, + "publicPrivateMetadataCheck": "passed", + "reproductionHelperSyntax": "all6 passed after public guard/cleanup corrections", + "terminalLeaseAndClockCorrections": "passed", + "localControllerDatabaseDataCleanup": "passed", + "offlineFailureCleanupRegression": "passed; old helper failed; current helper ledgered and cleaned both already-admitted owned IDs despite cap1; zero cloud calls" + }, + "excluded": [ + "Private credential bootstrap identifiers", + "Launch configs and credential-bearing application data", + "Raw API records", + "Full optional traces and videos" + ], + "verification": "All current file sizes and hashes independently verified. The public reproduction adaptations are declared in qualification.json; the exact executed historical helper hash remains unchanged." +} diff --git a/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/qualification.json b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/qualification.json new file mode 100644 index 00000000000..037ccb01d18 --- /dev/null +++ b/doc/benchmarks/native-event-ingestion-2026-11-03/cloud-stop/qualification.json @@ -0,0 +1,268 @@ +{ + "schema": "paperclip.real-cloud-stop-qualification.public.v1", + "status": "passed", + "scope": "One functional real-Daytona Stop journey; deterministic Codex provider fixture, real runnerd and real sleep60. Separate live-LLM verification and paired ingestion performance evidence are outside this result.", + "testedRevision": "137501f595fd5ee13d395e26b4147ca8bc4d4eb6", + "productionBuildCommit": "137501f595fd5ee13d395e26b4147ca8bc4d4eb6", + "controllerPlatform": { + "node": "v26.4.0", + "os": "darwin", + "architecture": "arm64" + }, + "dependencies": { + "provider": "deterministic Codex app-server fixture with real sleep60; no model call", + "sandbox": "real Daytona", + "sdk": "0.203.0", + "image": "ghcr.io/paperclipai/paperclip-daytona-runner@sha256:bff4c3f291087a0eeae37e4c20dd51857b92833eaf73ba3aca4157f37de1e109", + "controller": "normal production-built local Paperclip", + "actualLlmJourney": "separate real OpenCode/DeepSeek acceptance" + }, + "hashes": { + "server": "7db3ab75d39e5d6a467f961a267991cc55781439572cc37e45b4599908d1e23b", + "ui": "ba61db172af52228b1a5aacf4972bf7a85bb8a6da9bd359e5ce755b7e772db95", + "linuxRunner": "d4b3b76a510524a96ce23a1173c9559699bc0a0b2c096755d260f21e844ab0f2", + "provider": "b62bd845e63a2736a76bcf834c2a2bd281db46079d5e5c786c7e996f9a9ef269" + }, + "runtime": { + "node": "/opt/paperclip-runner/provider-pack/node_modules/node/bin/node", + "nodeVersion": "v24.21.0", + "nodeSha256": "7fde7b8afa198da66257f42ee2001d874c7355631e6d1579a5fb5ef1f246df4c", + "runnerSha256": "d4b3b76a510524a96ce23a1173c9559699bc0a0b2c096755d260f21e844ab0f2", + "fixtureSha256": "b62bd845e63a2736a76bcf834c2a2bd281db46079d5e5c786c7e996f9a9ef269", + "runnerMetadata": { + "binaryContractVersion": 2, + "binaryName": "paperclip-runnerd", + "durableSessionCapabilities": [ + "unlimited_runtime", + "connection_lease_renewal" + ], + "externalProviderCapabilities": [ + "openai_dot_mcp", + "grok_bot_mcp", + "muse_external_v1" + ], + "harnessDriverVersion": 1, + "nativeExecutionVersion": 1, + "packageName": "@paperclipai/paperclip-runner", + "packageVersion": "0.0.0", + "providerPacks": [], + "prp": { + "maximumVersion": 2, + "minimumVersion": 1, + "name": "paperclip.runner" + }, + "prpTransportModes": [ + "dial_ws_loopback", + "dial_wss", + "listen_ws" + ], + "schema": "paperclip-runner/runnerd-build-metadata/v1" + }, + "fixtureState": "awaiting-observer" + }, + "issueId": "0b686e4a-185f-4a49-a6b7-0672d08bfe65", + "runId": "c42d6ef1-ae58-44e3-8b76-fcf9b682000f", + "leaseId": "1b1aca64-e936-47d4-8509-43b7fe49f75f", + "providerLeaseId": "19e2daa7-e0fa-4427-988b-de136dd1b2b0", + "checks": { + "normalApiBindings": true, + "currentDaemonAndFixture": true, + "visibleStreamingBeforeTerminal": true, + "realSleepObserved": true, + "nativeAcknowledgement": true, + "sharedProviderReceiptAcknowledgement": true, + "independentChildRetirementNoEffect": true, + "exactLeaseProviderReceipt": true, + "independentProviderStopped": true, + "reloadAndNoSuccessorOrResult": true + }, + "visibleSourceMarker": { + "eventType": "item.delta", + "kind": "agentMessage", + "text": "CLOUD-STOP-STREAM: real sandbox process is starting. ", + "sourceEventId": "2f93856a-210e-4e98-ad38-70ee46014d9a:c42d6ef1-ae58-44e3-8b76-fcf9b682000f:12", + "sourceSeq": 12 + }, + "uiStop": { + "method": "POST", + "path": "/api/heartbeat-runs/c42d6ef1-ae58-44e3-8b76-fcf9b682000f/cancel", + "httpStatus": 200, + "clickedAtMs": 1793747222119, + "responseRunId": "c42d6ef1-ae58-44e3-8b76-fcf9b682000f", + "responseStatus": "cancelled", + "nativeCancellation": { + "runId": "c42d6ef1-ae58-44e3-8b76-fcf9b682000f", + "scope": "run", + "schema": "paperclip.native-cancellation.v1", + "effects": [ + "release_run_resources" + ], + "issueId": "0b686e4a-185f-4a49-a6b7-0672d08bfe65", + "intentId": "native-cancellation:5d947d68-4a95-4ed1-b7d7-cf0053d9402b", + "companyId": "f2ad7f34-3348-4be1-a7ef-139e3fd83324", + "decisionId": null, + "dispatched": true, + "reasonCode": "cancellation_run_only", + "recordedAt": "2026-11-03T23:07:02.184Z", + "dispatchState": "acknowledged", + "intentAuditId": "77c08f8e-6efb-40e5-bec8-a5ef78bc6379", + "acknowledgedAt": "2026-11-03T23:07:02.574Z", + "acknowledgementAuditId": "489f0915-f4e9-4571-b4ad-36b4a35acaed", + "priorCoordinatorDecisionId": null + }, + "executionCancellation": { + "proof": "provider_termination_receipt", + "state": "acknowledged", + "requestedAt": "2026-11-03T23:07:02.165Z", + "acknowledgedAt": "2026-11-03T23:07:14.581Z" + }, + "timingScope": "One functional sample, no performance claim" + }, + "processProof": { + "beforeStop": { + "observedAtMs": 1793747222186, + "child": { + "pid": 1226, + "ppid": 1171, + "pgid": 1226, + "startTicks": "2214420701", + "bootId": "89a86dc0-653a-4053-ae37-0701d477ebb5" + }, + "groupProcesses": [ + { + "pid": 1226, + "ppid": 1171, + "pgid": 1226, + "startTicks": "2214420701", + "command": "/bin/sh -c sleep 60; printf forbidden > \"$1\" cloud-stop-owned-child /home/daytona/paperclip-workspace/.cloud-stop-forbidden-effect " + }, + { + "pid": 1227, + "ppid": 1226, + "pgid": 1226, + "startTicks": "2214420702", + "command": "sleep 60 " + } + ], + "effectPresent": false, + "remoteStartedAtMs": 1793747221392, + "remoteProcessAgeAtObservationMs": 794 + }, + "retirement": { + "observedAtMs": 1793747222497, + "phase": "interrupted-child-retired", + "childLive": false, + "groupProcesses": [], + "effectPresent": false + }, + "causalOrdering": "The independent SDK snapshot promise resolved before the ordinary UI Stop click.", + "clockLimitation": "Provider and controller/browser clocks are independent. No remote-observation-to-UI interval or cross-host process age is inferred." + }, + "providerReceipt": { + "runId": "c42d6ef1-ae58-44e3-8b76-fcf9b682000f", + "state": "stopped", + "schema": "paperclip.remote-termination.v1", + "leaseId": "1b1aca64-e936-47d4-8509-43b7fe49f75f", + "provider": "daytona", + "companyId": "f2ad7f34-3348-4be1-a7ef-139e3fd83324", + "confirmedAt": "2026-11-03T23:07:14.570Z", + "providerLeaseId": "19e2daa7-e0fa-4427-988b-de136dd1b2b0" + }, + "independentProviderState": { + "id": "19e2daa7-e0fa-4427-988b-de136dd1b2b0", + "state": "stopped", + "labels": { + "paperclip-provider": "daytona", + "paperclip-company-id": "f2ad7f34-3348-4be1-a7ef-139e3fd83324", + "paperclip-environment-id": "54a2f054-9a17-4678-bdf0-ff11898ef913", + "paperclip-run-id": "c42d6ef1-ae58-44e3-8b76-fcf9b682000f", + "paperclip-reuse-lease": "false" + }, + "observedAt": 1793747234856 + }, + "databaseProof": { + "runId": "c42d6ef1-ae58-44e3-8b76-fcf9b682000f", + "companyId": "f2ad7f34-3348-4be1-a7ef-139e3fd83324", + "status": "cancelled", + "nextEventSeq": 82, + "eventCursorProjectionSeq": 81, + "eventCursorRebuildPending": false, + "globalEvents": 81, + "canonicalPayloadDigestsVerified": 16, + "encodedPayloadsDecoded": 0, + "projectionChecked": true, + "projectionStatus": "matches_retained_complete_fixture_history", + "sources": [ + { + "sourceInstanceId": "2f93856a-210e-4e98-ad38-70ee46014d9a", + "events": 16, + "contiguousSourceSeq": 15, + "pendingRanges": [ + { + "startSeq": 19, + "endSeq": 19 + } + ] + } + ] + }, + "cleanup": { + "normalEnvironmentDelete": true, + "exactOwnedFallbackDelete": true, + "providerAbsence": [ + { + "id": "19e2daa7-e0fa-4427-988b-de136dd1b2b0", + "state": "absent", + "observedAt": 1793747239557 + } + ], + "watchdogDisarmed": true + }, + "localCleanup": { + "status": "passed", + "checkedAt": "2026-11-03T23:10:48.163Z", + "ownedControllerProcessGone": true, + "ownedHttpPortClosed": true, + "ownedDatabasePortClosed": true, + "postmasterPidFileGone": true, + "credentialBearingDataRemoved": true, + "initialRemovalFailure": "EACCES unlink readonly skill bundle; retained as cleanup failure before bounded repair", + "ownedDirectoriesMadeWritable": 143, + "followedSymlinks": false + }, + "sourceDurabilityLimit": "This verifies database-retained events and derived cursors. It does not equate every wire ACK with a durable ordinary streaming event.", + "provenance": { + "executedPrivateHelperSha256": "a1a9b14f3718a448dbd566a92de9e14306df165fbacad5f3b6371eaaae3e005b", + "publicReproductionAdaptation": "Environment-supplied DAYTONA_API_KEY, one-resource cap, OP metadata omitted, same-provider-clock process-age guard, and ledger all observed owned resources before enforcing the cap. Public adaptations passed syntax checks and an offline actual-source failure-cleanup regression; they were not used for a second live acceptance claim. The historical same-provider-clock process age is independently retained in this proof." + }, + "terminalLease": { + "id": "1b1aca64-e936-47d4-8509-43b7fe49f75f", + "companyId": "f2ad7f34-3348-4be1-a7ef-139e3fd83324", + "environmentId": "54a2f054-9a17-4678-bdf0-ff11898ef913", + "heartbeatRunId": "c42d6ef1-ae58-44e3-8b76-fcf9b682000f", + "provider": "daytona", + "providerLeaseId": "19e2daa7-e0fa-4427-988b-de136dd1b2b0", + "status": "released", + "releasedAt": "2026-11-03T23:07:14.570Z", + "cleanupStatus": "success", + "receipt": { + "runId": "c42d6ef1-ae58-44e3-8b76-fcf9b682000f", + "state": "stopped", + "schema": "paperclip.remote-termination.v1", + "leaseId": "1b1aca64-e936-47d4-8509-43b7fe49f75f", + "provider": "daytona", + "companyId": "f2ad7f34-3348-4be1-a7ef-139e3fd83324", + "confirmedAt": "2026-11-03T23:07:14.570Z", + "providerLeaseId": "19e2daa7-e0fa-4427-988b-de136dd1b2b0" + } + }, + "sourceEquivalence": { + "testedAndBuiltRevision": "137501f595fd5ee13d395e26b4147ca8bc4d4eb6", + "laterPrRevision": "fcb5d94848b3c4b0b32096ff5e023618e0f0637b", + "changedPaths": [ + "packages/plugins/paperclip-plugin-fake-sandbox/src/lease-cleanup.test.ts", + "packages/plugins/paperclip-plugin-fake-sandbox/src/lease-cleanup.ts" + ], + "scope": "Only fake sandbox readonly cleanup changed. Daytona, ingestion, native runner and dependency inputs are unchanged. Actual qualification remains the production build at the tested revision." + } +}