fix(ci): run the sandbox provider tests in required CI (#19533)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Sandbox provider plugins (Daytona, E2B, Cloudflare, Kubernetes,
Modal, and others) run agents in remote sandboxes
> - The providers are outside the root pnpm workspace, so `pnpm
test:run` and the CI test lanes never run their Vitest suites
> - paperclipai/paperclip#18839 typechecks those tests, but a test that
fails at runtime still fails nothing
> - The providers have about 1,040 tests, and they finish in about 15
seconds on Linux
> - This pull request adds a required `Sandbox provider tests` job that
runs them when a change can affect them
> - The benefit is that a broken provider, plugin SDK, or shared
contract fails CI on the pull request instead of in a release or a paid
end-to-end run

## Linked Issues or Issue Description

Refs #18839

**What happened?**

No CI job runs the sandbox provider tests. `runner-full-stack-e2e.yml`
builds the Daytona plugin for a paid nightly run, but it does not run
any provider's `pnpm test`.

**Expected behavior**

A pull request that can break a provider runs that provider's tests, and
a failure blocks `ci / verify`.

**Steps to reproduce**

Add `expect(1).toBe(2)` to a test in
`packages/plugins/sandbox-providers/e2b/src`. Open a pull request. CI
stays green.

## What Changed

- Added `scripts/test-sandbox-providers.mjs` and its tests. The script
installs the providers through `typecheckTestPrograms` with no programs,
because that function already owns the provider install and the plugin
SDK link. Then it runs each provider's `pnpm test`, four at a time, and
prints each provider's output as one block. It fails if a suite fails or
if no provider has a test script.
- The script skips `cloudflare/bridge-template`, because Cloudflare's
Vitest config already runs those tests.
- Added the `sandbox_provider_tests` job to
`.github/workflows/pr-trusted.yml`, and made `ci / verify` require its
result.
- Added the `//#ci:sandbox-providers` task to `turbo.json` and its lane
to `.github/ci-task-matrix.json`. The CI planner runs the job only when
a change touches `packages/plugins/sandbox-providers/**`,
`packages/plugins/sdk/**`, or `packages/shared/**`. `packages/shared` is
the plugin SDK's only workspace dependency. Global CI inputs, such as
`scripts/**`, `.github/**`, `pnpm-lock.yaml`, or any `package.json`,
still select every job.
- The job turns off core dumps. A Daytona capacity test kills a process
at a file-size limit inside the exported directory. If core dumps are
on, the core file joins the export, and the test sees 12 files instead
of 11.
- Updated the job lists in `scripts/__tests__/e2e-shard.test.mjs` and
the job count in `.github/scripts/tests/ci-profile.test.mjs`. Added
planner tests to `.github/scripts/tests/ci-plan.test.mjs`. One of these
tests fails if the plugin SDK gets a new workspace dependency that is
not in the job's inputs.
- Added the new test file to `test:release-registry`, and documented the
script in `doc/DEVELOPING.md`.

## Verification

- `node scripts/test-sandbox-providers.mjs` on macOS: 8 of 9 suites pass
in 31.6 s, including the install. Kubernetes fails 12 `file-sync` tests,
because macOS has no `/proc/self/fd`. Some Daytona tests skip, because
they need GNU `tar` and coreutils.
- The same script in a Linux container (`node:24`, a non-root user, core
dumps off): all 9 suites pass, with 1,012 tests passed and 26 skipped.
The script took 15 s after the install.
  - Kubernetes: 205 passed.
  - Daytona: 590 passed and 16 skipped.
- With core dumps on, Daytona's `preserve literal capacity repair` test
fails. The export includes a `core` file. With `--ulimit core=0`, it
passes.
- To look for timing failures under load, I ran the whole script 12 more
times in the container, limited to 2 CPUs (8 runs) and 1 CPU (4 runs).
All 12 runs passed. They took 45 to 76 s on 2 CPUs and 73 to 102 s on 1
CPU.
- `ubuntu-latest` has `zstd`, and my container did not, so some Daytona
tests skipped in those runs. With `zstd` installed, Daytona passed with
598 tests and 8 skipped, and only the live tests skipped. One earlier
`zstd` run failed while the stress runs used the same CPUs, and I did
not capture its output. The next run passed.
- `node --test` on every test that reads `pr-trusted.yml`, the CI
planner, or the task matrix: 816 passed. These include
`.github/scripts/tests/*.test.mjs`,
`scripts/__tests__/e2e-shard.test.mjs`,
`scripts/ci/test-selection-workflow.test.mjs`, and the new tests.
- `pnpm run test:release-registry` on macOS: 251 of 255 pass, including
the new tests. The 4 failures are `wait_for_npm_package_versions` tests
in `scripts/release-lib.test.mjs`. They fail the same way on master on
the same machine, and CI runs this suite on Linux.
- `actionlint` 1.7.7 reports no errors on the changed workflow.
- I called `planTasks` from `.github/scripts/ci-plan.mjs` with one
changed file at a time. `server/src/index.ts` and `ui/src/App.tsx` leave
`sandbox_provider_tests` off.
`packages/plugins/sandbox-providers/e2b/src/plugin.ts` and
`packages/shared/src/index.ts` turn it on.

This pull request cannot run its own new job. `pr.yml` calls
`pr-trusted.yml@master`, so the job first runs on pull requests after
this one merges. After merge, check the first pull request that changes
a provider or `packages/shared`.

## Risks

- **First real run is after merge.** I measured the time on Linux in
Docker, not on a CI runner. CI runners are about 2 times slower than my
machine, so I expect about 2 to 3 minutes for the job, including the
dependency restore. The job runs in parallel with the other jobs, and
the slowest required job takes about 13 minutes. So the job should not
make CI slower.
- **Open provider pull requests.** After this merges, open pull requests
that change a provider run its tests. If those tests fail, `ci / verify`
blocks them until they are fixed.
- **`packages/shared` changes often.** Any change there runs this job. A
Linux-only provider failure would block those pull requests. All suites
pass on Linux today.
- **One unexplained Daytona failure.** One Daytona run with `zstd`
failed under CPU contention, and I did not capture its output. Some
Daytona tests have short deadlines. If the job is flaky after merge,
quarantine the failing test or revert the job, and do not leave `ci /
verify` red.
- **Path rules can be too narrow.** If a provider starts to import
another workspace package, a change to that package does not run this
job. The plugin SDK dependency test covers the SDK. It does not cover a
provider that imports a workspace package directly. Today no provider
does.
- **Overlap with paperclipai/paperclip#19489 and
paperclipai/paperclip#19522.** Both pull requests change the provider
install in `scripts/typecheck-test-programs.mjs`. This pull request does
not edit that file. It calls the same install function, so it uses their
install changes when they merge. All three pull requests edit the same
section of `doc/DEVELOPING.md`, so a small text conflict is possible.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with
tool use (shell, file edits, GitHub CLI, Docker). Extended reasoning was
on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

.github/ci-task-matrix.json

@@ -140,6 +140,11 @@
       "task": "ci:docker"
     }
   ],
+  "sandbox_provider_tests": [
+    {
+      "task": "ci:sandbox-providers"
+    }
+  ],
   "verify_paperclip_runner": [
     {
       "lane_label": "static checks",

.github/scripts/tests/ci-plan.test.mjs

@@ -76,6 +76,31 @@ test("shared contracts, runtime prompts, protocol inputs and fixtures reach cons
   }
 });
 
+test("sandbox provider tests run only for the providers, the plugin SDK, and what the SDK imports", () => {
+  for (const file of ["packages/plugins/sandbox-providers/daytona/src/plugin.ts",
+    "packages/plugins/sandbox-providers/cloudflare/bridge-template/src/index.ts",
+    "packages/plugins/sdk/src/index.ts", "packages/shared/src/types/issue.ts"]) {
+    assert.equal(selected(plan([file]), "sandbox_provider_tests"), true, file);
+  }
+  for (const file of ["server/src/index.ts", "ui/src/components/Sidebar.tsx", "cli/src/index.ts",
+    "packages/paperclip-runner/src/index.ts", "packages/plugins/plugin-workspace-diff/src/worker.ts"]) {
+    assert.equal(selected(plan([file]), "sandbox_provider_tests"), false, file);
+  }
+  // The provider lockfiles, the shared install workspace and the runner script are global inputs.
+  for (const file of ["scripts/typecheck-sandbox-providers/pnpm-lock.yaml", "scripts/test-sandbox-providers.mjs",
+    "packages/plugins/sandbox-providers/daytona/package.json"]) {
+    assert.equal(selected(plan([file]), "sandbox_provider_tests"), true, file);
+  }
+  // The SDK depends on shared and nothing else in the workspace. A new dependency must join the inputs.
+  const sdk = JSON.parse(readFileSync(new URL("packages/plugins/sdk/package.json", root), "utf8"));
+  const workspace = Object.entries({ ...sdk.dependencies, ...sdk.peerDependencies })
+    .filter(([, version]) => String(version).startsWith("workspace:")).map(([name]) => name);
+  assert.deepEqual(workspace, ["@paperclipai/shared"]);
+  const shared = JSON.parse(readFileSync(new URL("packages/shared/package.json", root), "utf8"));
+  assert.deepEqual(Object.values({ ...shared.dependencies, ...shared.peerDependencies })
+    .filter((version) => String(version).startsWith("workspace:")), []);
+});
+
 test("renames out of source and deletions cannot masquerade as release notes", () => {
   const paths = comparisonPaths({ files: [
     { filename: "releases/v2026.1029.0.md", previous_filename: "packages/shared/src/constants.ts", status: "renamed" },

.github/scripts/tests/ci-profile.test.mjs

@@ -330,10 +330,10 @@ test("workflow exporter runs trusted code and keeps keys out of the PR jobs", ()
   assertNoPrWorkflowCredentials(pr);
   assert.match(pr, /if: always\(\) && env.PAPERCLIP_CI_PROFILE == 'true'/);
   assert.doesNotMatch(pr, /PAPERCLIP_CI_PROFILE:.*labels/);
-  assert.equal(pr.match(/PAPERCLIP_CI_PROFILE: \$\{\{ needs.gate.outputs.profile_ci \}\}/g).length, 9);
+  assert.equal(pr.match(/PAPERCLIP_CI_PROFILE: \$\{\{ needs.gate.outputs.profile_ci \}\}/g).length, 10);
   // upload-artifact excludes files inside dot-directories by default. Keep
   // this opt-in scoped to timing NDJSON, never the workspace or other secrets.
-  assert.equal(pr.match(/path: \.ci-profile\/\*\.ndjson\n {10}include-hidden-files: true/g).length, 9);
+  assert.equal(pr.match(/path: \.ci-profile\/\*\.ndjson\n {10}include-hidden-files: true/g).length, 10);
   const routing = readFileSync(new URL("../../workflows/ci-routing.yml", import.meta.url), "utf8");
   assert.match(routing, /PROFILE_CI: \$\{\{ steps.profile.outputs.profile_ci \}\}/);
   assert.match(routing, /\/git\/ref\/heads\/master/);

.github/workflows/pr-trusted.yml

@@ -661,10 +661,76 @@ jobs:
           if-no-files-found: warn
           retention-days: 14
 
+  # Sandbox providers are outside the root workspace, so no other lane runs their
+  # tests. The `//#ci:sandbox-providers` inputs in turbo.json select this job: the
+  # providers, the plugin SDK, and the shared package the SDK depends on.
+  sandbox_provider_tests:
+    name: Sandbox provider tests
+    needs: [gate]
+    if: ${{ needs.gate.outputs.full_ci == 'true' && (needs.gate.outputs.plan == '' || fromJSON(needs.gate.outputs.plan).jobs.sandbox_provider_tests.run) }}
+    runs-on: ${{ needs.gate.outputs.runner }}
+    timeout-minutes: 15
+
+    env:
+      PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }}
+      PAPERCLIP_CI_PROFILE_JOB: Sandbox provider tests
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+        with:
+          persist-credentials: false
+
+      - name: Initialize CI profile
+        continue-on-error: true
+        if: env.PAPERCLIP_CI_PROFILE == 'true'
+        run: node scripts/ci-profile/initialize.mjs
+
+      - name: Setup Node.js for pnpm bootstrap
+        uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
+        with:
+          node-version: 24
+          package-manager-cache: false
+
+      - name: Setup pnpm
+        uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
+        env:
+          NPM_CONFIG_AUDIT: "false"
+          NPM_CONFIG_FUND: "false"
+          NPM_CONFIG_UPDATE_NOTIFIER: "false"
+        with:
+          version: 9.15.4
+
+      # Share the checked-in lockfile key with master. PR merge refs must not
+      # save full copies of the store or evict the post-merge build caches.
+      - name: Restore materialized dependencies and shared builds
+        uses: ./.github/actions/setup-ci-dependencies
+
+      # The script installs the providers from their own lockfiles, links the
+      # in-repo plugin SDK, and runs every provider suite. Daytona's capacity
+      # tests kill a process at a file-size limit inside the exported
+      # directory. On a runner that allows core dumps, its core file joins the
+      # export and changes the transferred file count, so turn core dumps off.
+      - name: Run sandbox provider tests
+        run: |
+          ulimit -c 0
+          node scripts/test-sandbox-providers.mjs
+
+      - name: Upload CI profile timings
+        continue-on-error: true
+        if: always() && env.PAPERCLIP_CI_PROFILE == 'true'
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        with:
+          name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }}
+          path: .ci-profile/*.ndjson
+          include-hidden-files: true
+          if-no-files-found: warn
+          retention-days: 14
+
   verify:
     name: verify
     if: ${{ always() }}
-    needs: [gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, verify_serialized_server, canary_dry_run]
+    needs: [gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run]
     runs-on: ${{ needs.gate.outputs.runner }}
     timeout-minutes: 5
     steps:
@@ -683,7 +749,7 @@ jobs:
           const plan = process.env.CI_PLAN ? JSON.parse(process.env.CI_PLAN) : null;
           if (plan) assert.equal(plan.version, 1, 'Supported task plan required');
           if (plan?.testSelection) assert.equal(results.test_selection.result, 'success', 'Published selection must succeed');
-          for (const job of ["typecheck_release_registry","general_tests","verify_paperclip_runner","build","docker_context_integrity","verify_serialized_server","canary_dry_run"]) {
+          for (const job of ["typecheck_release_registry","general_tests","verify_paperclip_runner","build","docker_context_integrity","sandbox_provider_tests","verify_serialized_server","canary_dry_run"]) {
             if (plan) assert.equal(typeof plan.jobs[job]?.run, 'boolean', 'Missing suite decision: ' + job);
             const selected = process.env.FULL_CI === 'true' && (!plan || plan.jobs[job].run);
             assert.equal(results[job].result, selected ? 'success' : 'skipped', job);

doc/DEVELOPING.md

@@ -1368,6 +1368,27 @@ Do not add a lockfile inside a provider.
 exists, which changes what a release resolves. Daytona already has one; the
 release build resolves it in place when it is stale.
 
+#### Sandbox provider tests
+
+`pnpm test:run` does not reach the sandbox providers, because they are outside the
+root workspace. `scripts/test-sandbox-providers.mjs` installs them the same way as
+the typecheck above, then runs every provider's `pnpm test`, four at a time:
+
+```sh
+node scripts/test-sandbox-providers.mjs
+```
+
+The required `Sandbox provider tests` job runs it when a pull request changes a
+provider, the plugin SDK, or `packages/shared`, which the SDK imports. Those paths
+are the `//#ci:sandbox-providers` inputs in `turbo.json`. Global CI inputs, such as
+`scripts/**` or a `package.json`, select every job. If the SDK gets a new workspace
+dependency, add that package to the inputs. A test in
+`.github/scripts/tests/ci-plan.test.mjs` fails until you do.
+
+Some Daytona file-sync tests need GNU `tar` and coreutils. They skip on macOS
+unless those tools are on `PATH`, so they run only on Linux. The job turns off
+core dumps: a core file in an exported directory changes the transferred file count.
+
 #### Reused Runner build in the server build
 
 The server build vendors `packages/paperclip-runner/dist`, so its

package.json

@@ -61,7 +61,7 @@
     "smoke:notion-generic-live": "node scripts/smoke/notion-generic-live.mjs",
     "smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
     "smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
-    "test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/public-install-registry.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/typecheck-test-programs.test.mjs scripts/__tests__/local-verify.test.mjs",
+    "test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/public-install-registry.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/__tests__/runner-debug-crate-cache.test.mjs scripts/__tests__/vitest-worker-isolation.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/typecheck-test-programs.test.mjs scripts/__tests__/local-verify.test.mjs scripts/test-sandbox-providers.test.mjs",
     "storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
     "test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
     "test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",

scripts/__tests__/e2e-shard.test.mjs

@@ -263,7 +263,7 @@ test("the trusted PR workflow limits full CI to merge-relevant stack layers", ()
   const verify = jobs.get("verify");
   assert.match(
     verify,
-    /^ {4}needs: \[gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, verify_serialized_server, canary_dry_run\]$/m,
+    /^ {4}needs: \[gate, test_selection, policy, typecheck_release_registry, general_tests, verify_paperclip_runner, build, docker_context_integrity, sandbox_provider_tests, verify_serialized_server, canary_dry_run\]$/m,
   );
   assert.equal(runAggregate(verify, true).status, 0);
   assert.equal(runAggregate(verify, false).status, 0);
@@ -272,7 +272,7 @@ test("the trusted PR workflow limits full CI to merge-relevant stack layers", ()
       "an optional selector cannot block the successful original matrix");
   }
   for (const job of ["gate", "policy", "typecheck_release_registry", "general_tests", "verify_paperclip_runner",
-    "build", "docker_context_integrity", "verify_serialized_server", "canary_dry_run"]) {
+    "build", "docker_context_integrity", "sandbox_provider_tests", "verify_serialized_server", "canary_dry_run"]) {
     for (const result of ["failure", "cancelled", "skipped"]) {
       assert.notEqual(runAggregate(verify, true, { [job]: result }).status, 0, `${job}: ${result}`);
     }

scripts/test-sandbox-providers.mjs

@@ -0,0 +1,105 @@
+#!/usr/bin/env node
+
+// Runs the Vitest suite of every sandbox provider plugin.
+//
+// Sandbox providers are outside the root workspace (see pnpm-workspace.yaml), so
+// neither the root install nor `pnpm test:run` reaches them. This script installs
+// them the same way scripts/typecheck-test-programs.mjs does, because that script
+// owns the provider install: it does that install, links the in-repo plugin SDK
+// and, when given no programs, typechecks nothing. The tests then run in parallel,
+// and each provider's output prints in one block when its suite finishes.
+//
+// The `Sandbox provider tests` job in .github/workflows/pr-trusted.yml runs this
+// script. The `//#ci:sandbox-providers` inputs in turbo.json decide when it runs.
+
+import { spawn } from "node:child_process";
+import { existsSync, readFileSync } from "node:fs";
+import { availableParallelism } from "node:os";
+import path from "node:path";
+import { fileURLToPath } from "node:url";
+import { SANDBOX_PROVIDERS, typecheckTestPrograms } from "./typecheck-test-programs.mjs";
+
+// Four suites at once keep a CI runner busy without slowing Daytona, the longest one.
+export const MAX_CONCURRENCY = 4;
+
+// A provider inside another provider runs with its parent: Cloudflare's Vitest config
+// includes bridge-template/src, so running bridge-template on its own repeats those tests.
+export function providersWithTests({ root, providers = SANDBOX_PROVIDERS }) {
+  return providers.filter((directory) => {
+    if (providers.some((other) => other !== directory && directory.startsWith(`${other}/`))) return false;
+    const manifest = path.join(root, directory, "package.json");
+    if (!existsSync(manifest)) return false;
+    return typeof JSON.parse(readFileSync(manifest, "utf8")).scripts?.test === "string";
+  });
+}
+
+function runPnpmTest({ root, directory, env }) {
+  return new Promise((resolve) => {
+    const started = Date.now();
+    const chunks = [];
+    const child = spawn("pnpm", ["test"], { cwd: path.join(root, directory), env, stdio: ["ignore", "pipe", "pipe"] });
+    child.stdout.on("data", (chunk) => chunks.push(chunk));
+    child.stderr.on("data", (chunk) => chunks.push(chunk));
+    child.on("error", (error) => {
+      chunks.push(Buffer.from(`${error.message}\n`));
+      resolve({ status: 1, output: Buffer.concat(chunks).toString(), seconds: (Date.now() - started) / 1000 });
+    });
+    child.on("close", (status, signal) => {
+      resolve({
+        status: signal ? 1 : status ?? 1,
+        output: Buffer.concat(chunks).toString(),
+        seconds: (Date.now() - started) / 1000,
+      });
+    });
+  });
+}
+
+export async function runProviderTests({
+  root,
+  env = process.env,
+  providers = providersWithTests({ root }),
+  concurrency = Math.max(1, Math.min(MAX_CONCURRENCY, availableParallelism())),
+  prepare = () => typecheckTestPrograms({ root, env, programs: [], coverage: false }),
+  runOne = runPnpmTest,
+  write = (text) => process.stdout.write(text),
+  grouped = env.GITHUB_ACTIONS === "true",
+}) {
+  if (providers.length === 0) {
+    write("test:sandbox-providers: no sandbox provider has a test script\n");
+    return 1;
+  }
+  const prepared = prepare();
+  if (prepared !== 0) {
+    write("test:sandbox-providers: could not install the sandbox providers\n");
+    return prepared;
+  }
+
+  const results = [];
+  const queue = [...providers];
+  async function worker() {
+    for (let directory = queue.shift(); directory; directory = queue.shift()) {
+      const result = await runOne({ root, directory, env });
+      const name = path.basename(directory);
+      const outcome = result.status === 0 ? "passed" : "failed";
+      const header = `${name}: ${outcome} in ${result.seconds.toFixed(1)}s`;
+      const body = result.output.endsWith("\n") || result.output === "" ? result.output : `${result.output}\n`;
+      write(grouped ? `::group::${header}\n${body}::endgroup::\n` : `--- ${header}\n${body}`);
+      if (result.status !== 0 && grouped) write(`::error title=Sandbox provider tests::${name} failed\n`);
+      results.push({ name, ...result });
+    }
+  }
+  await Promise.all(Array.from({ length: Math.min(concurrency, providers.length) }, worker));
+
+  const failed = results.filter((result) => result.status !== 0).map((result) => result.name).sort();
+  write(`test:sandbox-providers: ${results.length - failed.length} of ${results.length} provider suites passed\n`);
+  if (failed.length > 0) {
+    write(`test:sandbox-providers: failed: ${failed.join(", ")}\n`);
+    return 1;
+  }
+  return 0;
+}
+
+if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
+  const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
+  process.exitCode = await runProviderTests({ root });
+}

scripts/test-sandbox-providers.test.mjs

@@ -0,0 +1,98 @@
+import assert from "node:assert/strict";
+import { readdirSync, readFileSync } from "node:fs";
+import path from "node:path";
+import { test } from "node:test";
+import { fileURLToPath } from "node:url";
+import { providersWithTests, runProviderTests } from "./test-sandbox-providers.mjs";
+
+const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..");
+const providerRoot = "packages/plugins/sandbox-providers";
+
+const harness = (statuses = {}) => {
+  const calls = { prepared: 0, ran: [], output: "", active: 0, peak: 0 };
+  return {
+    calls,
+    options: {
+      root: "/repo",
+      providers: ["p/a", "p/b", "p/c"],
+      concurrency: 2,
+      grouped: false,
+      prepare: () => {
+        calls.prepared += 1;
+        return 0;
+      },
+      runOne: async ({ directory }) => {
+        calls.ran.push(directory);
+        calls.active += 1;
+        calls.peak = Math.max(calls.peak, calls.active);
+        await new Promise((resolve) => setTimeout(resolve, 5));
+        calls.active -= 1;
+        return { status: statuses[directory] ?? 0, output: `${directory} output`, seconds: 1.25 };
+      },
+      write: (text) => {
+        calls.output += text;
+      },
+    },
+  };
+};
+
+test("every top-level sandbox provider runs its own suite", () => {
+  const onDisk = readdirSync(path.join(root, providerRoot), { withFileTypes: true })
+    .filter((entry) => entry.isDirectory())
+    .map((entry) => `${providerRoot}/${entry.name}`)
+    .filter((directory) => {
+      try {
+        return Boolean(JSON.parse(readFileSync(path.join(root, directory, "package.json"), "utf8")).scripts?.test);
+      } catch {
+        return false;
+      }
+    });
+  assert.ok(onDisk.length > 0);
+  assert.deepEqual([...providersWithTests({ root })].sort(), onDisk.sort());
+});
+
+test("a provider nested in another provider runs with its parent", () => {
+  const providers = providersWithTests({ root });
+  assert.ok(providers.includes(`${providerRoot}/cloudflare`));
+  assert.ok(!providers.includes(`${providerRoot}/cloudflare/bridge-template`));
+  const config = readFileSync(path.join(root, providerRoot, "cloudflare", "vitest.config.ts"), "utf8");
+  assert.match(config, /bridge-template\/src\/\*\*\/\*\.test\.ts/, "Cloudflare's config must keep running bridge-template tests");
+});
+
+test("all suites run after one install, within the concurrency limit", async () => {
+  const { calls, options } = harness();
+  assert.equal(await runProviderTests(options), 0);
+  assert.equal(calls.prepared, 1);
+  assert.deepEqual([...calls.ran].sort(), ["p/a", "p/b", "p/c"]);
+  assert.equal(calls.peak, 2);
+  assert.match(calls.output, /^--- a: passed in 1\.3s\np\/a output\n/m);
+  assert.match(calls.output, /3 of 3 provider suites passed/);
+});
+
+test("one failing suite fails the run after the others finish", async () => {
+  const { calls, options } = harness({ "p/b": 1 });
+  assert.equal(await runProviderTests(options), 1);
+  assert.deepEqual([...calls.ran].sort(), ["p/a", "p/b", "p/c"]);
+  assert.match(calls.output, /--- b: failed in 1\.3s/);
+  assert.match(calls.output, /2 of 3 provider suites passed/);
+  assert.match(calls.output, /failed: b\n/);
+});
+
+test("a failed install runs no suite", async () => {
+  const { calls, options } = harness();
+  assert.equal(await runProviderTests({ ...options, prepare: () => 7 }), 7);
+  assert.deepEqual(calls.ran, []);
+});
+
+test("an empty provider list fails instead of passing silently", async () => {
+  const { calls, options } = harness();
+  assert.equal(await runProviderTests({ ...options, providers: [] }), 1);
+  assert.equal(calls.prepared, 0);
+});
+
+test("GitHub Actions output folds each suite and annotates failures", async () => {
+  const { calls, options } = harness({ "p/a": 1 });
+  await runProviderTests({ ...options, grouped: true, providers: ["p/a"] });
+  assert.match(calls.output, /^::group::a: failed in 1\.3s\np\/a output\n::endgroup::\n/);
+  assert.match(calls.output, /::error title=Sandbox provider tests::a failed/);
+});

turbo.json

@@ -196,6 +196,15 @@
       "outputs": [],
       "cache": false
     },
+    "//#ci:sandbox-providers": {
+      "inputs": [
+        "packages/plugins/sandbox-providers/**",
+        "packages/plugins/sdk/**",
+        "packages/shared/**"
+      ],
+      "outputs": [],
+      "cache": false
+    },
     "//#ci:docker": {
       "inputs": [
         "server/**",