From 3b3037aab504a43ab8a71871aaabe79850e7b8df Mon Sep 17 00:00:00 2001 From: Nicky Leach Date: Fri, 6 Nov 2026 10:34:03 -0800 Subject: [PATCH] fix(ci): run the sandbox provider tests in required CI (#19533) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox provider plugins (Daytona, E2B, Cloudflare, Kubernetes, Modal, and others) run agents in remote sandboxes > - The providers are outside the root pnpm workspace, so `pnpm test:run` and the CI test lanes never run their Vitest suites > - paperclipai/paperclip#18839 typechecks those tests, but a test that fails at runtime still fails nothing > - The providers have about 1,040 tests, and they finish in about 15 seconds on Linux > - This pull request adds a required `Sandbox provider tests` job that runs them when a change can affect them > - The benefit is that a broken provider, plugin SDK, or shared contract fails CI on the pull request instead of in a release or a paid end-to-end run ## Linked Issues or Issue Description Refs #18839 **What happened?** No CI job runs the sandbox provider tests. `runner-full-stack-e2e.yml` builds the Daytona plugin for a paid nightly run, but it does not run any provider's `pnpm test`. **Expected behavior** A pull request that can break a provider runs that provider's tests, and a failure blocks `ci / verify`. **Steps to reproduce** Add `expect(1).toBe(2)` to a test in `packages/plugins/sandbox-providers/e2b/src`. Open a pull request. CI stays green. ## What Changed - Added `scripts/test-sandbox-providers.mjs` and its tests. The script installs the providers through `typecheckTestPrograms` with no programs, because that function already owns the provider install and the plugin SDK link. Then it runs each provider's `pnpm test`, four at a time, and prints each provider's output as one block. It fails if a suite fails or if no provider has a test script. - The script skips `cloudflare/bridge-template`, because Cloudflare's Vitest config already runs those tests. - Added the `sandbox_provider_tests` job to `.github/workflows/pr-trusted.yml`, and made `ci / verify` require its result. - Added the `//#ci:sandbox-providers` task to `turbo.json` and its lane to `.github/ci-task-matrix.json`. The CI planner runs the job only when a change touches `packages/plugins/sandbox-providers/**`, `packages/plugins/sdk/**`, or `packages/shared/**`. `packages/shared` is the plugin SDK's only workspace dependency. Global CI inputs, such as `scripts/**`, `.github/**`, `pnpm-lock.yaml`, or any `package.json`, still select every job. - The job turns off core dumps. A Daytona capacity test kills a process at a file-size limit inside the exported directory. If core dumps are on, the core file joins the export, and the test sees 12 files instead of 11. - Updated the job lists in `scripts/__tests__/e2e-shard.test.mjs` and the job count in `.github/scripts/tests/ci-profile.test.mjs`. Added planner tests to `.github/scripts/tests/ci-plan.test.mjs`. One of these tests fails if the plugin SDK gets a new workspace dependency that is not in the job's inputs. - Added the new test file to `test:release-registry`, and documented the script in `doc/DEVELOPING.md`. ## Verification - `node scripts/test-sandbox-providers.mjs` on macOS: 8 of 9 suites pass in 31.6 s, including the install. Kubernetes fails 12 `file-sync` tests, because macOS has no `/proc/self/fd`. Some Daytona tests skip, because they need GNU `tar` and coreutils. - The same script in a Linux container (`node:24`, a non-root user, core dumps off): all 9 suites pass, with 1,012 tests passed and 26 skipped. The script took 15 s after the install. - Kubernetes: 205 passed. - Daytona: 590 passed and 16 skipped. - With core dumps on, Daytona's `preserve literal capacity repair` test fails. The export includes a `core` file. With `--ulimit core=0`, it passes. - To look for timing failures under load, I ran the whole script 12 more times in the container, limited to 2 CPUs (8 runs) and 1 CPU (4 runs). All 12 runs passed. They took 45 to 76 s on 2 CPUs and 73 to 102 s on 1 CPU. - `ubuntu-latest` has `zstd`, and my container did not, so some Daytona tests skipped in those runs. With `zstd` installed, Daytona passed with 598 tests and 8 skipped, and only the live tests skipped. One earlier `zstd` run failed while the stress runs used the same CPUs, and I did not capture its output. The next run passed. - `node --test` on every test that reads `pr-trusted.yml`, the CI planner, or the task matrix: 816 passed. These include `.github/scripts/tests/*.test.mjs`, `scripts/__tests__/e2e-shard.test.mjs`, `scripts/ci/test-selection-workflow.test.mjs`, and the new tests. - `pnpm run test:release-registry` on macOS: 251 of 255 pass, including the new tests. The 4 failures are `wait_for_npm_package_versions` tests in `scripts/release-lib.test.mjs`. They fail the same way on master on the same machine, and CI runs this suite on Linux. - `actionlint` 1.7.7 reports no errors on the changed workflow. - I called `planTasks` from `.github/scripts/ci-plan.mjs` with one changed file at a time. `server/src/index.ts` and `ui/src/App.tsx` leave `sandbox_provider_tests` off. `packages/plugins/sandbox-providers/e2b/src/plugin.ts` and `packages/shared/src/index.ts` turn it on. This pull request cannot run its own new job. `pr.yml` calls `pr-trusted.yml@master`, so the job first runs on pull requests after this one merges. After merge, check the first pull request that changes a provider or `packages/shared`. ## Risks - **First real run is after merge.** I measured the time on Linux in Docker, not on a CI runner. CI runners are about 2 times slower than my machine, so I expect about 2 to 3 minutes for the job, including the dependency restore. The job runs in parallel with the other jobs, and the slowest required job takes about 13 minutes. So the job should not make CI slower. - **Open provider pull requests.** After this merges, open pull requests that change a provider run its tests. If those tests fail, `ci / verify` blocks them until they are fixed. - **`packages/shared` changes often.** Any change there runs this job. A Linux-only provider failure would block those pull requests. All suites pass on Linux today. - **One unexplained Daytona failure.** One Daytona run with `zstd` failed under CPU contention, and I did not capture its output. Some Daytona tests have short deadlines. If the job is flaky after merge, quarantine the failing test or revert the job, and do not leave `ci / verify` red. - **Path rules can be too narrow.** If a provider starts to import another workspace package, a change to that package does not run this job. The plugin SDK dependency test covers the SDK. It does not cover a provider that imports a workspace package directly. Today no provider does. - **Overlap with paperclipai/paperclip#19489 and paperclipai/paperclip#19522.** Both pull requests change the provider install in `scripts/typecheck-test-programs.mjs`. This pull request does not edit that file. It calls the same install function, so it uses their install changes when they merge. All three pull requests edit the same section of `doc/DEVELOPING.md`, so a small text conflict is possible. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use (shell, file edits, GitHub CLI, Docker). Extended reasoning was on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .github/ci-task-matrix.json | 5 ++ .github/scripts/tests/ci-plan.test.mjs | 25 ++++++ .github/scripts/tests/ci-profile.test.mjs | 4 +- .github/workflows/pr-trusted.yml | 70 ++++++++++++++- doc/DEVELOPING.md | 21 +++++ package.json | 2 +- scripts/__tests__/e2e-shard.test.mjs | 4 +- scripts/test-sandbox-providers.mjs | 105 ++++++++++++++++++++++ scripts/test-sandbox-providers.test.mjs | 98 ++++++++++++++++++++ turbo.json | 9 ++ 10 files changed, 336 insertions(+), 7 deletions(-) create mode 100644 scripts/test-sandbox-providers.mjs create mode 100644 scripts/test-sandbox-providers.test.mjs diff --git a/.github/ci-task-matrix.json b/.github/ci-task-matrix.json index e6602d115dd..5e9cc021b39 100644 --- a/.github/ci-task-matrix.json +++ b/.github/ci-task-matrix.json @@ -140,6 +140,11 @@ "task": "ci:docker" } ], + "sandbox_provider_tests": [ + { + "task": "ci:sandbox-providers" + } + ], "verify_paperclip_runner": [ { "lane_label": "static checks", diff --git a/.github/scripts/tests/ci-plan.test.mjs b/.github/scripts/tests/ci-plan.test.mjs index ab4f04c2994..e03b6d3a1e6 100644 --- a/.github/scripts/tests/ci-plan.test.mjs +++ b/.github/scripts/tests/ci-plan.test.mjs @@ -76,6 +76,31 @@ test("shared contracts, runtime prompts, protocol inputs and fixtures reach cons } }); +test("sandbox provider tests run only for the providers, the plugin SDK, and what the SDK imports", () => { + for (const file of ["packages/plugins/sandbox-providers/daytona/src/plugin.ts", + "packages/plugins/sandbox-providers/cloudflare/bridge-template/src/index.ts", + "packages/plugins/sdk/src/index.ts", "packages/shared/src/types/issue.ts"]) { + assert.equal(selected(plan([file]), "sandbox_provider_tests"), true, file); + } + for (const file of ["server/src/index.ts", "ui/src/components/Sidebar.tsx", "cli/src/index.ts", + "packages/paperclip-runner/src/index.ts", "packages/plugins/plugin-workspace-diff/src/worker.ts"]) { + assert.equal(selected(plan([file]), "sandbox_provider_tests"), false, file); + } + // The provider lockfiles, the shared install workspace and the runner script are global inputs. + for (const file of ["scripts/typecheck-sandbox-providers/pnpm-lock.yaml", "scripts/test-sandbox-providers.mjs", + "packages/plugins/sandbox-providers/daytona/package.json"]) { + assert.equal(selected(plan([file]), "sandbox_provider_tests"), true, file); + } + // The SDK depends on shared and nothing else in the workspace. A new dependency must join the inputs. + const sdk = JSON.parse(readFileSync(new URL("packages/plugins/sdk/package.json", root), "utf8")); + const workspace = Object.entries({ ...sdk.dependencies, ...sdk.peerDependencies }) + .filter(([, version]) => String(version).startsWith("workspace:")).map(([name]) => name); + assert.deepEqual(workspace, ["@paperclipai/shared"]); + const shared = JSON.parse(readFileSync(new URL("packages/shared/package.json", root), "utf8")); + assert.deepEqual(Object.values({ ...shared.dependencies, ...shared.peerDependencies }) + .filter((version) => String(version).startsWith("workspace:")), []); +}); + test("renames out of source and deletions cannot masquerade as release notes", () => { const paths = comparisonPaths({ files: [ { filename: "releases/v2026.1029.0.md", previous_filename: "packages/shared/src/constants.ts", status: "renamed" }, diff --git a/.github/scripts/tests/ci-profile.test.mjs b/.github/scripts/tests/ci-profile.test.mjs index 08b560fa20a..4f8242c2863 100644 --- a/.github/scripts/tests/ci-profile.test.mjs +++ b/.github/scripts/tests/ci-profile.test.mjs @@ -330,10 +330,10 @@ test("workflow exporter runs trusted code and keeps keys out of the PR jobs", () assertNoPrWorkflowCredentials(pr); assert.match(pr, /if: always\(\) && env.PAPERCLIP_CI_PROFILE == 'true'/); assert.doesNotMatch(pr, /PAPERCLIP_CI_PROFILE:.*labels/); - assert.equal(pr.match(/PAPERCLIP_CI_PROFILE: \$\{\{ needs.gate.outputs.profile_ci \}\}/g).length, 9); + assert.equal(pr.match(/PAPERCLIP_CI_PROFILE: \$\{\{ needs.gate.outputs.profile_ci \}\}/g).length, 10); // upload-artifact excludes files inside dot-directories by default. Keep // this opt-in scoped to timing NDJSON, never the workspace or other secrets. - assert.equal(pr.match(/path: \.ci-profile\/\*\.ndjson\n {10}include-hidden-files: true/g).length, 9); + assert.equal(pr.match(/path: \.ci-profile\/\*\.ndjson\n {10}include-hidden-files: true/g).length, 10); const routing = readFileSync(new URL("../../workflows/ci-routing.yml", import.meta.url), "utf8"); assert.match(routing, /PROFILE_CI: \$\{\{ steps.profile.outputs.profile_ci \}\}/); assert.match(routing, /\/git\/ref\/heads\/master/); diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index d3305f0d2b4..a8c452ceb96 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -661,10 +661,76 @@ jobs: if-no-files-found: warn retention-days: 14 + # Sandbox providers are outside the root workspace, so no other lane runs their + # tests. The `//#ci:sandbox-providers` inputs in turbo.json select this job: the + # providers, the plugin SDK, and the shared package the SDK depends on. + sandbox_provider_tests: + name: Sandbox provider tests + needs: [gate] + if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.sandbox_provider_tests.run) }} + runs-on: ${{ needs.gate.outputs.runner }} + timeout-minutes: 15 + + env: + PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }} + PAPERCLIP_CI_PROFILE_JOB: Sandbox provider tests + + steps: + - name: Checkout repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + persist-credentials: false + + - name: Initialize CI profile + continue-on-error: true + if: env.PAPERCLIP_CI_PROFILE == 'true' + run: node scripts/ci-profile/initialize.mjs + + - name: Setup Node.js for pnpm bootstrap + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 + with: + node-version: 24 + package-manager-cache: false + + - name: Setup pnpm + uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6 + env: + NPM_CONFIG_AUDIT: "false" + NPM_CONFIG_FUND: "false" + NPM_CONFIG_UPDATE_NOTIFIER: "false" + with: + version: 9.15.4 + + # Share the checked-in lockfile key with master. PR merge refs must not + # save full copies of the store or evict the post-merge build caches. + - name: Restore materialized dependencies and shared builds + uses: ./.github/actions/setup-ci-dependencies + + # The script installs the providers from their own lockfiles, links the + # in-repo plugin SDK, and runs every provider suite. Daytona's capacity + # tests kill a process at a file-size limit inside the exported + # directory. On a runner that allows core dumps, its core file joins the + # export and changes the transferred file count, so turn core dumps off. + - name: Run sandbox provider tests + run: | + ulimit -c 0 + node scripts/test-sandbox-providers.mjs + + - name: Upload CI profile timings + continue-on-error: true + if: always() && env.PAPERCLIP_CI_PROFILE == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} + path: .ci-profile/*.ndjson + include-hidden-files: true + if-no-files-found: warn + retention-days: 14 + verify: name: verify if: ${{ always() }} - needs: [gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, verify_serialized_server, canary_dry_run] + needs: [gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run] runs-on: ${{ needs.gate.outputs.runner }} timeout-minutes: 5 steps: @@ -683,7 +749,7 @@ jobs: const plan = process.env.CI_PLAN ? JSON.parse(process.env.CI_PLAN) : null; if (plan) assert.equal(plan.version, 1, 'Supported task plan required'); if (plan?.testSelection) assert.equal(results.test_selection.result, 'success', 'Published selection must succeed'); - for (const job of ["typecheck_release_registry","general_tests","verify_paperclip_runner","build","docker_context_integrity","verify_serialized_server","canary_dry_run"]) { + for (const job of ["typecheck_release_registry","general_tests","verify_paperclip_runner","build","docker_context_integrity","sandbox_provider_tests","verify_serialized_server","canary_dry_run"]) { if (plan) assert.equal(typeof plan.jobs[job]?.run, 'boolean', 'Missing suite decision: ' + job); const selected = process.env.FULL_CI === 'true' && (!plan || plan.jobs[job].run); assert.equal(results[job].result, selected ? 'success' : 'skipped', job); diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index a7b69dee07e..8eecc8fd780 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -1368,6 +1368,27 @@ Do not add a lockfile inside a provider. exists, which changes what a release resolves. Daytona already has one; the release build resolves it in place when it is stale. +#### Sandbox provider tests + +`pnpm test:run` does not reach the sandbox providers, because they are outside the +root workspace. `scripts/test-sandbox-providers.mjs` installs them the same way as +the typecheck above, then runs every provider's `pnpm test`, four at a time: + +```sh +node scripts/test-sandbox-providers.mjs +``` + +The required `Sandbox provider tests` job runs it when a pull request changes a +provider, the plugin SDK, or `packages/shared`, which the SDK imports. Those paths +are the `//#ci:sandbox-providers` inputs in `turbo.json`. Global CI inputs, such as +`scripts/**` or a `package.json`, select every job. If the SDK gets a new workspace +dependency, add that package to the inputs. A test in +`.github/scripts/tests/ci-plan.test.mjs` fails until you do. + +Some Daytona file-sync tests need GNU `tar` and coreutils. They skip on macOS +unless those tools are on `PATH`, so they run only on Linux. The job turns off +core dumps: a core file in an exported directory changes the transferred file count. + #### Reused Runner build in the server build The server build vendors `packages/paperclip-runner/dist`, so its diff --git a/package.json b/package.json index 8ea83e18e59..3eb15777631 100644 --- a/package.json +++ b/package.json @@ -61,7 +61,7 @@ "smoke:notion-generic-live": "node scripts/smoke/notion-generic-live.mjs", "smoke:posthog-live": "node scripts/smoke/posthog-live.mjs", "smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs", - "test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/public-install-registry.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/typecheck-test-programs.test.mjs scripts/__tests__/local-verify.test.mjs", + "test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/public-install-registry.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/typecheck-test-programs.test.mjs scripts/__tests__/local-verify.test.mjs scripts/test-sandbox-providers.test.mjs", "storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs", "test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts", "test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack", diff --git a/scripts/__tests__/e2e-shard.test.mjs b/scripts/__tests__/e2e-shard.test.mjs index 8a08d5427cd..b77536157b2 100644 --- a/scripts/__tests__/e2e-shard.test.mjs +++ b/scripts/__tests__/e2e-shard.test.mjs @@ -263,7 +263,7 @@ test("the trusted PR workflow limits full CI to merge-relevant stack layers", () const verify = jobs.get("verify"); assert.match( verify, - /^ {4}needs: \[gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, verify_serialized_server, canary_dry_run\]$/m, + /^ {4}needs: \[gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run\]$/m, ); assert.equal(runAggregate(verify, true).status, 0); assert.equal(runAggregate(verify, false).status, 0); @@ -272,7 +272,7 @@ test("the trusted PR workflow limits full CI to merge-relevant stack layers", () "an optional selector cannot block the successful original matrix"); } for (const job of ["gate", "policy", "typecheck_release_registry", "general_tests", "verify_paperclip_runner", - "build", "docker_context_integrity", "verify_serialized_server", "canary_dry_run"]) { + "build", "docker_context_integrity", "sandbox_provider_tests", "verify_serialized_server", "canary_dry_run"]) { for (const result of ["failure", "cancelled", "skipped"]) { assert.notEqual(runAggregate(verify, true, { [job]: result }).status, 0, `${job}: ${result}`); } diff --git a/scripts/test-sandbox-providers.mjs b/scripts/test-sandbox-providers.mjs new file mode 100644 index 00000000000..ce2cfdbf42e --- /dev/null +++ b/scripts/test-sandbox-providers.mjs @@ -0,0 +1,105 @@ +#!/usr/bin/env node + +// Runs the Vitest suite of every sandbox provider plugin. +// +// Sandbox providers are outside the root workspace (see pnpm-workspace.yaml), so +// neither the root install nor `pnpm test:run` reaches them. This script installs +// them the same way scripts/typecheck-test-programs.mjs does, because that script +// owns the provider install: it does that install, links the in-repo plugin SDK +// and, when given no programs, typechecks nothing. The tests then run in parallel, +// and each provider's output prints in one block when its suite finishes. +// +// The `Sandbox provider tests` job in .github/workflows/pr-trusted.yml runs this +// script. The `//#ci:sandbox-providers` inputs in turbo.json decide when it runs. + +import { spawn } from "node:child_process"; +import { existsSync, readFileSync } from "node:fs"; +import { availableParallelism } from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { SANDBOX_PROVIDERS, typecheckTestPrograms } from "./typecheck-test-programs.mjs"; + +// Four suites at once keep a CI runner busy without slowing Daytona, the longest one. +export const MAX_CONCURRENCY = 4; + +// A provider inside another provider runs with its parent: Cloudflare's Vitest config +// includes bridge-template/src, so running bridge-template on its own repeats those tests. +export function providersWithTests({ root, providers = SANDBOX_PROVIDERS }) { + return providers.filter((directory) => { + if (providers.some((other) => other !== directory && directory.startsWith(`${other}/`))) return false; + const manifest = path.join(root, directory, "package.json"); + if (!existsSync(manifest)) return false; + return typeof JSON.parse(readFileSync(manifest, "utf8")).scripts?.test === "string"; + }); +} + +function runPnpmTest({ root, directory, env }) { + return new Promise((resolve) => { + const started = Date.now(); + const chunks = []; + const child = spawn("pnpm", ["test"], { cwd: path.join(root, directory), env, stdio: ["ignore", "pipe", "pipe"] }); + child.stdout.on("data", (chunk) => chunks.push(chunk)); + child.stderr.on("data", (chunk) => chunks.push(chunk)); + child.on("error", (error) => { + chunks.push(Buffer.from(`${error.message}\n`)); + resolve({ status: 1, output: Buffer.concat(chunks).toString(), seconds: (Date.now() - started) / 1000 }); + }); + child.on("close", (status, signal) => { + resolve({ + status: signal ? 1 : status ?? 1, + output: Buffer.concat(chunks).toString(), + seconds: (Date.now() - started) / 1000, + }); + }); + }); +} + +export async function runProviderTests({ + root, + env = process.env, + providers = providersWithTests({ root }), + concurrency = Math.max(1, Math.min(MAX_CONCURRENCY, availableParallelism())), + prepare = () => typecheckTestPrograms({ root, env, programs: [], coverage: false }), + runOne = runPnpmTest, + write = (text) => process.stdout.write(text), + grouped = env.GITHUB_ACTIONS === "true", +}) { + if (providers.length === 0) { + write("test:sandbox-providers: no sandbox provider has a test script\n"); + return 1; + } + const prepared = prepare(); + if (prepared !== 0) { + write("test:sandbox-providers: could not install the sandbox providers\n"); + return prepared; + } + + const results = []; + const queue = [...providers]; + async function worker() { + for (let directory = queue.shift(); directory; directory = queue.shift()) { + const result = await runOne({ root, directory, env }); + const name = path.basename(directory); + const outcome = result.status === 0 ? "passed" : "failed"; + const header = `${name}: ${outcome} in ${result.seconds.toFixed(1)}s`; + const body = result.output.endsWith("\n") || result.output === "" ? result.output : `${result.output}\n`; + write(grouped ? `::group::${header}\n${body}::endgroup::\n` : `--- ${header}\n${body}`); + if (result.status !== 0 && grouped) write(`::error title=Sandbox provider tests::${name} failed\n`); + results.push({ name, ...result }); + } + } + await Promise.all(Array.from({ length: Math.min(concurrency, providers.length) }, worker)); + + const failed = results.filter((result) => result.status !== 0).map((result) => result.name).sort(); + write(`test:sandbox-providers: ${results.length - failed.length} of ${results.length} provider suites passed\n`); + if (failed.length > 0) { + write(`test:sandbox-providers: failed: ${failed.join(", ")}\n`); + return 1; + } + return 0; +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); + process.exitCode = await runProviderTests({ root }); +} diff --git a/scripts/test-sandbox-providers.test.mjs b/scripts/test-sandbox-providers.test.mjs new file mode 100644 index 00000000000..37e8e3afbcd --- /dev/null +++ b/scripts/test-sandbox-providers.test.mjs @@ -0,0 +1,98 @@ +import assert from "node:assert/strict"; +import { readdirSync, readFileSync } from "node:fs"; +import path from "node:path"; +import { test } from "node:test"; +import { fileURLToPath } from "node:url"; +import { providersWithTests, runProviderTests } from "./test-sandbox-providers.mjs"; + +const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), ".."); +const providerRoot = "packages/plugins/sandbox-providers"; + +const harness = (statuses = {}) => { + const calls = { prepared: 0, ran: [], output: "", active: 0, peak: 0 }; + return { + calls, + options: { + root: "/repo", + providers: ["p/a", "p/b", "p/c"], + concurrency: 2, + grouped: false, + prepare: () => { + calls.prepared += 1; + return 0; + }, + runOne: async ({ directory }) => { + calls.ran.push(directory); + calls.active += 1; + calls.peak = Math.max(calls.peak, calls.active); + await new Promise((resolve) => setTimeout(resolve, 5)); + calls.active -= 1; + return { status: statuses[directory] ?? 0, output: `${directory} output`, seconds: 1.25 }; + }, + write: (text) => { + calls.output += text; + }, + }, + }; +}; + +test("every top-level sandbox provider runs its own suite", () => { + const onDisk = readdirSync(path.join(root, providerRoot), { withFileTypes: true }) + .filter((entry) => entry.isDirectory()) + .map((entry) => `${providerRoot}/${entry.name}`) + .filter((directory) => { + try { + return Boolean(JSON.parse(readFileSync(path.join(root, directory, "package.json"), "utf8")).scripts?.test); + } catch { + return false; + } + }); + assert.ok(onDisk.length > 0); + assert.deepEqual([...providersWithTests({ root })].sort(), onDisk.sort()); +}); + +test("a provider nested in another provider runs with its parent", () => { + const providers = providersWithTests({ root }); + assert.ok(providers.includes(`${providerRoot}/cloudflare`)); + assert.ok(!providers.includes(`${providerRoot}/cloudflare/bridge-template`)); + const config = readFileSync(path.join(root, providerRoot, "cloudflare", "vitest.config.ts"), "utf8"); + assert.match(config, /bridge-template\/src\/\*\*\/\*\.test\.ts/, "Cloudflare's config must keep running bridge-template tests"); +}); + +test("all suites run after one install, within the concurrency limit", async () => { + const { calls, options } = harness(); + assert.equal(await runProviderTests(options), 0); + assert.equal(calls.prepared, 1); + assert.deepEqual([...calls.ran].sort(), ["p/a", "p/b", "p/c"]); + assert.equal(calls.peak, 2); + assert.match(calls.output, /^--- a: passed in 1\.3s\np\/a output\n/m); + assert.match(calls.output, /3 of 3 provider suites passed/); +}); + +test("one failing suite fails the run after the others finish", async () => { + const { calls, options } = harness({ "p/b": 1 }); + assert.equal(await runProviderTests(options), 1); + assert.deepEqual([...calls.ran].sort(), ["p/a", "p/b", "p/c"]); + assert.match(calls.output, /--- b: failed in 1\.3s/); + assert.match(calls.output, /2 of 3 provider suites passed/); + assert.match(calls.output, /failed: b\n/); +}); + +test("a failed install runs no suite", async () => { + const { calls, options } = harness(); + assert.equal(await runProviderTests({ ...options, prepare: () => 7 }), 7); + assert.deepEqual(calls.ran, []); +}); + +test("an empty provider list fails instead of passing silently", async () => { + const { calls, options } = harness(); + assert.equal(await runProviderTests({ ...options, providers: [] }), 1); + assert.equal(calls.prepared, 0); +}); + +test("GitHub Actions output folds each suite and annotates failures", async () => { + const { calls, options } = harness({ "p/a": 1 }); + await runProviderTests({ ...options, grouped: true, providers: ["p/a"] }); + assert.match(calls.output, /^::group::a: failed in 1\.3s\np\/a output\n::endgroup::\n/); + assert.match(calls.output, /::error title=Sandbox provider tests::a failed/); +}); diff --git a/turbo.json b/turbo.json index 618ca30fd37..d4d1d1fd9d1 100644 --- a/turbo.json +++ b/turbo.json @@ -196,6 +196,15 @@ "outputs": [], "cache": false }, + "//#ci:sandbox-providers": { + "inputs": [ + "packages/plugins/sandbox-providers/**", + "packages/plugins/sdk/**", + "packages/shared/**" + ], + "outputs": [], + "cache": false + }, "//#ci:docker": { "inputs": [ "server/**",