Files
PaperClipAI/scripts/run-vitest-stable.mjs
T
DottaandClaude Opus 5.5 03309f1650 fix(native): release an idle warm owner whose sandbox Runner port a later run took over (#21835)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes idle sessions, and closes them again before it
exits.
> - A sandbox has one Runner port. Before a new run launches its Runner
on a reused sandbox, it retires every other Runner on that port.
> - In a zero-downtime release-swap test, a run finished on the outgoing
server and stayed warm. 77 ms later, a follow-up run started on the
incoming server. It reused the same sandbox, started a fresh provider
session, retired the old Runner, and launched its own Runner on the same
port.
> - The outgoing server's idle close could not suspend the old Runner.
Its checkpoint was correctly refused with
`native_runner_checkpoint_owner_lost`. The idle timer retried 3 times
and then kept the owner registered. That phantom owner blocked idle
sleep for 4.8 minutes, and the shutdown close failed again.
> - The existing proofs that a warm owner is gone (#21782) do not count
"a later run launched its own Runner on this sandbox".
> - This pull request adds that proof. When it holds, the server forgets
the owner without a checkpoint. The checkpoint fence does not change.
> - The benefit is that a superseded warm owner no longer blocks idle
sleep or fails a clean shutdown.

## Linked Issues or Issue Description

Refs #21627
Refs #21782

**What happened?**
An idle warm owner finished on the outgoing server. A follow-up run on
the incoming server reused the same sandbox and launched its own Runner
on the sandbox's single Runner port. The owner's idle close failed with
`native_runner_checkpoint_owner_lost` 4 times. The owner stayed
registered, held the idle-work flag, and blocked idle sleep. At SIGTERM
the restart sweep logged "idle warm native sessions did not close" with
"runner did not durably suspend before checkpoint". No task was
affected.

**Expected behavior**
A failed idle close releases its owner when a later run proves that the
owner's Runner is gone. The checkpoint stays refused while the successor
is live.

**Steps to reproduce**
1. Finish a warm run on a sandbox and keep it idle.
2. Start a successor run for the same task on another server. Give it a
lease on the same sandbox, its own `native.process_identity_recorded`
event and its own `session.started` event.
3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result
is `closed: 0, failed: 1`, and `readNativeSessionIdleWork()` stays
`present`.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: new
`warmRunnerPortTakenBySuccessor`, used by
`warmCloseFailureProvesOwnerGone` after the existing sandbox-termination
proof. It holds only when another run has all of these:
- an `environment_leases` row with the same environment and
`providerLeaseId` as the owner;
- its own `native.process_identity_recorded` event (`source_event_id is
null`, so an inherited identity does not count);
- a `session.started` event from its own Runner instance
(`source_instance_id` equals the run's `runner_instance_id`);
  - both events created after the owner run's `finishedAt`.
- When the proof holds, the idle-timer path and the controller-restart
path log "warm native session's Runner or sandbox is gone; released its
owner" and forget the owner without a checkpoint. Both paths already
call `warmCloseFailureProvesOwnerGone`.
- `validateNativeRunnerCheckpointOwner` does not change. A checkpoint
still cannot land under a live successor.
- Cost: the proof runs only after a failed close. It does one
primary-key read of the owner run and one query on
`environment_leases.provider_lease_id` (indexed), joined to
`heartbeat_runs` by primary key, with `heartbeat_run_events` read
through `(company_id, run_id)`.
- New real-database test file
`server/src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`.
- `native-session-executor.test.ts`: one environment-delete test counts
the extra proof read when the first proof does not hold.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`:
6 passed.
- "releases an idle warm owner whose checkpoint is refused because a
successor on another server launched its own Runner on the same sandbox:
own_runner": the close is refused with
`native_runner_checkpoint_owner_lost`, the result is `closed: 1, failed:
0`, the idle-work flag is `none`, the owner has no checkpoint row, and
`assertNativeRunnerCheckpointOwner` still throws
`native_runner_checkpoint_owner_lost`.
- These stay failed and keep the owner registered: the successor is on
another sandbox; the successor has only
`native.process_start_requested`; the successor has only inherited
events (`source_event_id` set); the successor's events come before the
owner finished.
- "releases a superseded idle warm owner from its idle timer without
writing a checkpoint": the idle timer releases the owner after the first
refused close.
- Red on master (test file only, fix reverted): `own_runner` fails with
`closed: 0, failed: 1`; the idle-timer test fails with the idle-work
flag still `present`. The 4 negative cases pass on master and with the
fix.
- `cd server && npx vitest run
src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts`:
9 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: all
passed.
- `cd server && npx tsc --noEmit -p .`: no errors.
- `pnpm verify`: see the comment below for the result.
- Not run: a restart-chaos wave. The release-swap timing is reproduced
in the database test, not on a live sandbox.

## Risks

- A superseded owner is now released without a checkpoint. Its old
provider conversation cannot be resumed. The successor had already
started a different conversation and holds the task, so no one would
resume the old one.
- The proof needs the successor's own Runner identity and own session
start after the owner finished, on the same sandbox lease. A successor
that reused the owner's Runner, only requested a start, or recorded its
events earlier does not count. In those cases the close fails as before.
- The proof adds two small indexed reads after a failed close only. The
normal close path does not change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 17:49:01 -06:00

784 lines
32 KiB
JavaScript

#!/usr/bin/env node
import { spawn } from "node:child_process";
import { mkdirSync, mkdtempSync, readdirSync, readFileSync, realpathSync, statSync, writeFileSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { loadShardDurations, selectGeneralServerShard } from "./general-server-shard.mjs";
import { invocationEnv } from "./ci-profile/record.mjs";
import { vitestRetryArgs } from "./ci/vitest-retry.mjs";
import { assertSelectedTests, partitionTestLines } from "./test-line-shard.mjs";
import { loadSelection, selectedShard } from "./ci/test-selection-manifest.mjs";
const repoRoot = process.cwd();
const scriptsDir = path.dirname(fileURLToPath(import.meta.url));
const generalServerShardDurations = loadShardDurations(
path.join(scriptsDir, "general-server-shard-durations.json"),
);
const serializedShardDurations = loadShardDurations(
path.join(scriptsDir, "serialized-shard-durations.json"),
);
const chatShardDurations = loadShardDurations(path.join(scriptsDir, "chat-shard-durations.json"));
const serverRoot = path.join(repoRoot, "server");
const serverSrcDir = path.join(repoRoot, "server", "src");
const serverTestsDir = path.join(repoRoot, "server", "src", "__tests__");
const serverScriptsDir = path.join(repoRoot, "server", "scripts");
const nonServerProjects = [
"@paperclipai/shared",
"@paperclipai/skills-catalog",
"@paperclipai/db",
"@paperclipai/adapter-utils",
"@paperclipai/adapter-claude-local",
"@paperclipai/adapter-codex-local",
"@paperclipai/adapter-cursor-cloud",
"@paperclipai/adapter-cursor-local",
"@paperclipai/adapter-gemini-local",
"@paperclipai/adapter-grok-local",
"@paperclipai/hermes-paperclip-adapter",
"@paperclipai/adapter-kimi-local",
"@paperclipai/adapter-openclaw-gateway",
"@paperclipai/adapter-opencode-local",
"@paperclipai/adapter-pi-local",
"@paperclipai/plugin-daytona",
"@paperclipai/plugin-sdk",
"@paperclipai/create-paperclip-plugin",
"@paperclipai/ui",
"paperclipai",
];
const routeTestPattern = /[^/]*(?:route|routes|authz)[^/]*\.test\.ts$/;
const additionalSerializedServerTests = new Set([
"server/src/__tests__/approval-routes-idempotency.test.ts",
"server/src/__tests__/assets.test.ts",
"server/src/__tests__/authz-company-access.test.ts",
"server/src/__tests__/companies-route-path-guard.test.ts",
"server/src/__tests__/company-portability.test.ts",
"server/src/__tests__/costs-service.test.ts",
"server/src/__tests__/express5-auth-wildcard.test.ts",
"server/src/__tests__/health-dev-server-token.test.ts",
"server/src/__tests__/health.test.ts",
"server/src/__tests__/heartbeat-dependency-scheduling.test.ts",
"server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts",
"server/src/__tests__/heartbeat-process-recovery.test.ts",
"server/src/__tests__/invite-accept-existing-member.test.ts",
"server/src/__tests__/invite-accept-gateway-defaults.test.ts",
"server/src/__tests__/invite-accept-replay.test.ts",
"server/src/__tests__/invite-expiry.test.ts",
"server/src/__tests__/invite-join-manager.test.ts",
"server/src/__tests__/invite-onboarding-text.test.ts",
"server/src/__tests__/invite-url-public-base-url.test.ts",
"server/src/__tests__/issues-checkout-wakeup.test.ts",
"server/src/__tests__/issues-service.test.ts",
"server/src/__tests__/opencode-local-adapter-environment.test.ts",
"server/src/__tests__/project-routes-env.test.ts",
"server/src/__tests__/redaction.test.ts",
"server/src/__tests__/routines-e2e.test.ts",
]);
// Suite-specific external URLs can also name a shared mutable database.
// Owned database factories use PAPERCLIP_TEST_POSTGRES_ADMIN_URL instead.
const hasSharedTestDatabase = Object.entries(process.env).some(([key, value]) =>
key.startsWith("PAPERCLIP_") && key.endsWith("_TEST_DATABASE_URL") && value?.trim(),
);
let invocationIndex = 0;
const serializedModeName = "serialized";
const generalModeName = "general";
const allModeName = "all";
const generalServerGroupName = "general-server";
const generalServerWithoutChatGroupName = "general-server-without-chat";
const generalServerWithoutChatOrNativeRunnerGroupName = "general-server-without-chat-or-native-runner";
const generalChatGroupName = "general-chat";
const generalServerNativeRunnerGroupName = "general-server-native-runner";
const chatSuite = "server/src/__tests__/chat-channels.integration.test.ts";
// The first suite rebuilds the Runner release binaries with cargo in beforeAll.
// Inside the PR workflow's plain server shards, which carry no Rust cache,
// that build was a ~4m30s cold compile of every third-party crate on each run
// (277s of a 291s shard vitest step, actions run 35246999382, 2026-09-17).
const nativeRunnerSuites = [
"server/src/services/native-runtime/plugin-runner-probe.test.ts",
"server/src/services/native-runtime/native-codex-runner.integration.test.ts",
"server/src/services/native-runtime/native-ingress-bootstrap-retry.integration.test.ts",
"server/src/__tests__/dot-runner.test.ts",
// These process cases otherwise skip in server shards without Runner binaries.
"server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts",
"server/src/services/native-runtime/native-runner-restart-lease-renewal.integration.test.ts",
"server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts",
"server/src/services/native-runtime/native-runner-restart-checkpoint-cost.integration.test.ts",
"server/src/services/native-runtime/native-runner-restart-diskless.integration.test.ts",
"server/src/services/native-runtime/native-cut-turn-start-retry.integration.test.ts",
"server/src/services/native-runtime/native-usage-delivery.integration.test.ts",
"server/src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts",
"server/src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts",
"server/src/services/native-runtime/plugin-runner-task-warm.integration.test.ts",
"server/src/services/native-runtime/plugin-runner-release-resume.integration.test.ts",
"server/src/services/native-runtime/plugin-runner-eviction.integration.test.ts",
];
// In the PR workflow (pr.yml, the caller of pr-trusted.yml — reusable
// workflows inherit the caller's GITHUB_WORKFLOW), the last Verify Paperclip
// Runner vitest shard runs the native-runner group instead, because those
// lanes restore the shared release-runner-v2 Rust cache (see
// packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs). Every other
// caller of the same group keeps the suite in the server shards, so a renamed or
// unknown workflow degrades to today's slower-but-covered behavior rather
// than dropping the suite.
const prWorkflowName = "PR";
const nativeRunnerSuiteRunsInRustCachedLane = process.env.GITHUB_WORKFLOW === prWorkflowName;
const withoutChatExcludedSuites = nativeRunnerSuiteRunsInRustCachedLane
? [chatSuite, ...nativeRunnerSuites]
: [chatSuite];
// Release verification selects this explicit group and runs the omitted suites
// in its cached Runner job. Local callers keep the complete default group.
const generalServerGroups = [
generalServerGroupName,
generalServerWithoutChatGroupName,
generalServerWithoutChatOrNativeRunnerGroupName,
];
function excludedServerSuites(groupName) {
if (groupName === generalServerWithoutChatOrNativeRunnerGroupName) return [chatSuite, ...nativeRunnerSuites];
if (groupName === generalServerWithoutChatGroupName) return withoutChatExcludedSuites;
return [];
}
const generalWorkspacesAGroupName = "general-workspaces-a";
const generalWorkspacesBGroupName = "general-workspaces-b";
const generalWorkspacesCGroupName = "general-workspaces-c";
const generalWorkspacesAProjects = ["@paperclipai/ui", "paperclipai"];
// The db project runs in its own lane. Its vitest invocation took 168-250s
// of a 436-513s workspaces-b job (PR runs 38084171761, 38083084276 and
// 38083020834, 2026-10-10), which made workspaces-b the slowest check once
// the server shards were rebalanced. Vitest's native --shard would not help
// here: the project is already file-parallel and client.test.ts alone is
// 117-189s of its wall time, so a hash-ordered slice can land both of its
// slow suites on one runner. Splitting by project is deterministic and
// leaves the remaining projects at ~265-275s of vitest time, roughly level
// with this lane.
const generalWorkspacesCProjects = ["@paperclipai/db"];
const generalWorkspacesBProjects = nonServerProjects.filter(
(project) => !generalWorkspacesAProjects.includes(project) && !generalWorkspacesCProjects.includes(project),
);
const generalGroupNames = [
generalServerGroupName,
generalWorkspacesAGroupName,
generalWorkspacesBGroupName,
generalWorkspacesCGroupName,
];
const allowedGeneralGroupNames = [
...generalGroupNames,
generalServerWithoutChatGroupName,
generalServerWithoutChatOrNativeRunnerGroupName,
generalChatGroupName,
generalServerNativeRunnerGroupName,
];
const serializedServerVitestArgs = [
"--no-file-parallelism",
"--maxWorkers=1",
];
const sourceOnlyVitestArgs = ["--exclude", "**/dist/**"];
function walk(dir) {
const entries = readdirSync(dir);
const files = [];
for (const entry of entries) {
const absolute = path.join(dir, entry);
const stats = statSync(absolute);
if (stats.isDirectory()) {
files.push(...walk(absolute));
} else if (stats.isFile()) {
files.push(absolute);
}
}
return files;
}
function toRepoPath(file) {
return path.relative(repoRoot, file).split(path.sep).join("/");
}
function toServerPath(file) {
return path.relative(serverRoot, file).split(path.sep).join("/");
}
// For a multi-line `it.each([...])("name", fn)` call, Vitest 5's `vitest
// list --includeTaskLocation` and `vitest run <file>:<line>` disagree about
// which line registers the test: `list` reports (and only matches) the line
// of the wrapped "name" argument, while `run` only matches the line above
// it, where Prettier's formatting puts that call's opening `(`. Separately,
// on this file `list` also occasionally invents an entry whose "location"
// sits on an ordinary body statement that merely ends in `)(` or starts a
// line with a quote, with no `it`/`test` call anywhere nearby — a bogus
// artifact of the same collector, not a real test. Both only showed up on
// server/src/__tests__/chat-channels.integration.test.ts, a 70,000+ line
// fixture; smaller files have not reproduced either issue.
//
// chatSuiteRunLine classifies a `list`-reported line against the actual
// source and returns the line `vitest run` accepts, or null if the line
// does not belong to any real `it`/`test` call (a bogus entry to drop).
const callAnywherePattern = /(?:^|[^\w$])(?:it|test)(?:\.(?:each|skip|only))?\(/;
const inlineCallPattern = /\)\(\s*["'`]/;
function chatSuiteRunLine(sourceLines, line) {
const text = (sourceLines[line - 1] ?? "").trim();
if (callAnywherePattern.test(text) || text.endsWith(")(") || inlineCallPattern.test(text)) {
return line;
}
const previous = (sourceLines[line - 2] ?? "").trim();
const isNameArgument = text.startsWith('"') || text.startsWith("'") || text.startsWith("`");
if (isNameArgument && previous.endsWith(")(")) {
return line - 1;
}
return null;
}
function isRouteOrAuthzTest(file) {
if (routeTestPattern.test(file)) {
return true;
}
return additionalSerializedServerTests.has(file);
}
function fail(message) {
console.error(`[test:run] ${message}`);
process.exit(1);
}
function readOptionValue(argv, index, argName) {
const value = argv[index + 1];
if (value === undefined) {
fail(`Missing value for ${argName}`);
}
return value;
}
function parseNonNegativeInteger(value, argName) {
const parsed = Number(value);
if (value.trim() === "" || !Number.isInteger(parsed) || parsed < 0) {
fail(`${argName} must be a non-negative integer. Received "${value}".`);
}
return parsed;
}
function parsePositiveInteger(value, argName) {
const parsed = Number(value);
if (value.trim() === "" || !Number.isInteger(parsed) || parsed < 1) {
fail(`${argName} must be a positive integer. Received "${value}".`);
}
return parsed;
}
function parseCliOptions(argv) {
let mode = allModeName;
let shardIndex = null;
let shardCount = null;
let group = null;
let dryRun = false;
let serialChat = false;
for (let index = 0; index < argv.length; index += 1) {
const arg = argv[index];
if (arg === "--") {
continue;
}
if (arg === "--mode") {
mode = readOptionValue(argv, index, arg);
index += 1;
continue;
}
if (arg.startsWith("--mode=")) {
mode = arg.slice("--mode=".length);
continue;
}
if (arg === "--shard-index") {
shardIndex = parseNonNegativeInteger(readOptionValue(argv, index, arg), arg);
index += 1;
continue;
}
if (arg.startsWith("--shard-index=")) {
shardIndex = parseNonNegativeInteger(arg.slice("--shard-index=".length), "--shard-index");
continue;
}
if (arg === "--shard-count") {
shardCount = parsePositiveInteger(readOptionValue(argv, index, arg), arg);
index += 1;
continue;
}
if (arg.startsWith("--shard-count=")) {
shardCount = parsePositiveInteger(arg.slice("--shard-count=".length), "--shard-count");
continue;
}
if (arg === "--dry-run") {
dryRun = true;
continue;
}
if (arg === "--serial-chat") {
serialChat = true;
continue;
}
if (arg === "--group") {
group = readOptionValue(argv, index, arg);
index += 1;
continue;
}
if (arg.startsWith("--group=")) {
group = arg.slice("--group=".length);
continue;
}
fail(`Unknown argument "${arg}".`);
}
if (!new Set([allModeName, generalModeName, serializedModeName]).has(mode)) {
fail(`Unknown mode "${mode}". Expected one of: ${allModeName}, ${generalModeName}, ${serializedModeName}.`);
}
if ((shardIndex === null) !== (shardCount === null)) {
fail("--shard-index and --shard-count must be provided together.");
}
const shardAllowed =
mode === serializedModeName ||
(mode === generalModeName &&
([...generalServerGroups, generalChatGroupName, generalWorkspacesAGroupName].includes(group)
|| Boolean(process.env.PAPERCLIP_CI_TEST_SELECTION)
&& [generalWorkspacesBGroupName, generalWorkspacesCGroupName].includes(group)));
if (!shardAllowed && shardIndex !== null) {
fail(
"--shard-index/--shard-count require serialized mode, a shardable general server/chat/workspaces-a group, or a selected workspace group.",
);
}
if (group !== null && mode !== generalModeName) {
fail("--group is only valid with --mode general.");
}
if (group !== null && !allowedGeneralGroupNames.includes(group)) {
fail(`Unknown group "${group}". Expected one of: ${allowedGeneralGroupNames.join(", ")}.`);
}
if (serialChat && (mode !== generalModeName || group !== generalChatGroupName)) {
fail("--serial-chat is only valid with --mode general --group general-chat.");
}
if (shardIndex !== null) {
if (shardIndex >= shardCount) {
fail(`--shard-index must be less than --shard-count. Received ${shardIndex} of ${shardCount}.`);
}
}
if (mode === serializedModeName) {
return {
mode,
shardIndex: shardIndex ?? 0,
shardCount: shardCount ?? 1,
group: null,
dryRun,
};
}
return {
mode,
shardIndex,
shardCount,
group,
dryRun,
serialChat,
};
}
function selectSerializedSuites(routeTests, shardIndex, shardCount) {
// Same duration-aware LPT partition as the general-server lane. Round-robin
// over the alphabetical list clustered the heavy heartbeat/issues suites on
// one shard (291s vs 170-201s test steps across the matrix in actions run
// 32012408876), which made that shard the whole PR run's slowest check.
const byRepoPath = new Map(routeTests.map((routeTest) => [routeTest.repoPath, routeTest]));
const shardFiles = selectGeneralServerShard(
routeTests.map((routeTest) => routeTest.repoPath),
shardIndex,
shardCount,
serializedShardDurations,
);
return shardFiles.map((file) => byRepoPath.get(file));
}
function isolatedTestEnv() {
invocationIndex += 1;
const tempRootParent = process.platform === "win32" ? os.tmpdir() : "/tmp";
// Production workspace/security checks reject symlink aliases. In particular
// /tmp is /private/tmp on macOS, so fixture roots must use the canonical path.
const testRoot = realpathSync(mkdtempSync(path.join(tempRootParent, "pv-")));
// Keep per-run paths compact so Unix socket fixtures stay under macOS path limits.
const env = {
...process.env,
NODE_ENV: "test",
PAPERCLIP_TEST_HOST_HOME: process.env.PAPERCLIP_TEST_HOST_HOME
?? (process.env.PAPERCLIP_HOME?.trim() || path.join(os.homedir(), ".paperclip")),
PAPERCLIP_HOME: path.join(testRoot, "h"),
// Config discovery otherwise prefers the checkout's .paperclip/config.json
// over PAPERCLIP_HOME, importing preview scheduling policy into unit tests.
PAPERCLIP_CONFIG: path.join(testRoot, "h", "config.json"),
PAPERCLIP_INSTANCE_ID: `vt-${process.pid}-${invocationIndex}`,
TMPDIR: path.join(testRoot, "t"),
};
// The wrapper has already validated and consumed this shard's manifest.
// Tests may launch independent runner fixtures with a different root/inventory.
delete env.PAPERCLIP_CI_TEST_SELECTION;
delete env.PAPERCLIP_CI_SELECTION_HASH;
mkdirSync(env.PAPERCLIP_HOME, { recursive: true });
mkdirSync(env.TMPDIR, { recursive: true });
return { testRoot, env };
}
async function collectVitestList(filters, { testRoot, env }, name) {
const output = path.join(testRoot, `${name}.json`);
await new Promise((resolve, reject) => {
const child = spawn("pnpm", ["exec", "vitest", "list", ...sourceOnlyVitestArgs,
...filters, "--allowOnly=false", "--includeTaskLocation", `--json=${output}`], {
cwd: repoRoot, env, stdio: "inherit",
});
child.once("error", (error) => reject(new Error(`Vitest collection failed: ${error.message}`)));
child.once("close", (code, signal) => {
if (code === 0) resolve();
else reject(new Error(`Vitest collection failed: ${signal ?? code}`));
});
});
return JSON.parse(readFileSync(output, "utf8"));
}
async function runVitest(args, label, testShard = null, exactFiles = false) {
console.log(`\n[test:run] ${label}`);
const isolated = isolatedTestEnv();
const { env } = isolated;
if (testShard) {
const file = path.resolve(repoRoot, chatSuite);
const sourceLines = readFileSync(file, "utf8").split("\n");
const allCollected = testShard.collected ?? await collectVitestList(args, isolated, "all");
const collected = [];
const bogusCollected = [];
for (const test of allCollected) {
(chatSuiteRunLine(sourceLines, test.location.line) === null ? bogusCollected : collected).push(test);
}
if (bogusCollected.length > 0) {
console.warn(
`[test:run] dropping ${bogusCollected.length} bogus Vitest collection entr${bogusCollected.length === 1 ? "y" : "ies"} with no matching it()/test() call: ${bogusCollected.map((test) => `${test.location.line}:${JSON.stringify(test.name)}`).join(", ")}`,
);
}
const selected = partitionTestLines(collected, testShard.count, file, chatShardDurations)[testShard.index];
// Validate shard membership with the same `list`-reported lines that
// produced `selected`, then switch to the `run`-compatible lines only for
// the filters handed to the real `vitest run` below (see
// chatSuiteRunLine).
const validationFilters = selected.lines.map((line) => `${chatSuite}:${line}`);
const validationArgs = [...args.filter((arg) => arg !== chatSuite), ...validationFilters];
assertSelectedTests(selected.tests, await collectVitestList(validationArgs, isolated, "selected"), file);
console.log(`[test:run] chat shard ${testShard.index + 1}/${testShard.count}: ${selected.tests.length}/${collected.length} tests, ${selected.lines.length} source lines, ${Math.round(selected.weight / 1000)}s recorded; exact filter coverage verified`);
const runFilters = selected.lines.map((line) => `${chatSuite}:${chatSuiteRunLine(sourceLines, line)}`);
args = [...args.filter((arg) => arg !== chatSuite), ...runFilters, "--allowOnly=false"];
}
await new Promise((resolve, reject) => {
const command = exactFiles ? process.execPath : "pnpm";
const prefix = exactFiles ? [path.join(scriptsDir, "ci/run-selected-vitest.mjs")] : ["exec", "vitest", "run"];
// `vitest list` collection above never retries; only the run does.
const child = spawn(command, [...prefix, ...sourceOnlyVitestArgs, ...vitestRetryArgs(), ...args], {
cwd: repoRoot,
env: invocationEnv("vitest", env),
stdio: "inherit",
});
child.once("error", reject);
child.once("close", (code, signal) => {
if (code === 0) resolve();
else reject(new Error(`${label} failed (${signal ?? code})`));
});
});
}
async function runGeneralSuites(routeTests, chatShards) {
for (const shard of chatShards ?? []) {
await runVitest(["--project", "@paperclipai/server", ...serializedServerVitestArgs, chatSuite],
"chat integration test shard", shard);
}
for (const groupName of generalGroupNames) {
await runGeneralGroup(routeTests, groupName, null, null, chatShards ? [chatSuite] : []);
}
}
async function runProjectGroup(projects, groupName, shardIndex = null, shardCount = null) {
// With shard args, lean on Vitest's native --shard: each matrix job runs the
// same per-project invocations but only its slice of each project's test
// files. Vitest's sharding is deterministic for an identical file list, so
// the matrix jobs form a complete, non-overlapping cover of every project.
const shardArgs =
shardCount !== null && shardCount > 1 ? [`--shard=${shardIndex + 1}/${shardCount}`] : [];
const shardSuffix = shardArgs.length > 0 ? ` shard ${shardIndex + 1}/${shardCount}` : "";
for (const project of projects) {
await runVitest(["--project", project, ...shardArgs], `${groupName} project ${project}${shardSuffix}`);
}
}
function chatShardWorkers(index, count) {
const workers = count > 1 && !hasSharedTestDatabase && !options.serialChat ? 3 : 1;
return Array.from({ length: workers }, (_, offset) => ({
index: (index ?? 0) * workers + offset,
count: (count ?? 1) * workers,
}));
}
async function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = null, additionalExcludedSuites = []) {
if (groupName === generalChatGroupName) {
// Each invocation gets its own temporary home and embedded PostgreSQL.
// Wait for both children even on failure so no sibling is abandoned.
const args = ["--project", "@paperclipai/server", ...serializedServerVitestArgs, chatSuite];
const workers = chatShardWorkers(shardIndex, shardCount);
const collected = await collectVitestList(args, isolatedTestEnv(), "all");
const results = await Promise.allSettled(workers.map((shard) =>
runVitest(args, "chat integration test shard", { ...shard, collected }),
));
const failed = results.find((result) => result.status === "rejected");
if (failed) throw failed.reason;
return;
}
if (groupName === generalServerNativeRunnerGroupName) {
await runVitest(
["--project", "@paperclipai/server",
// Each file owns its database, home, and processes. Keep cases serial,
// but overlap long real lease windows in four isolated fork workers.
...(hasSharedTestDatabase ? serializedServerVitestArgs
: ["--pool=forks", "--isolate", "--fileParallelism", "--maxWorkers=4"]),
...nativeRunnerSuites],
"native runner vertical-slice suite",
);
return;
}
if (generalServerGroups.includes(groupName)) {
const excludedSuites = [...excludedServerSuites(groupName), ...additionalExcludedSuites];
const files = generalServerTestFiles.filter((file) => !excludedSuites.includes(file));
if (shardCount !== null && shardCount > 1) {
const shardFiles = selectGeneralServerShard(
files,
shardIndex,
shardCount,
generalServerShardDurations,
);
console.log(
`\n[test:run] general-server shard ${shardIndex + 1}/${shardCount} running ${shardFiles.length} of ${files.length} suites`,
);
if (shardFiles.length === 0) {
return;
}
await runVitest(
[
"--project",
"@paperclipai/server",
// Files have separate fork processes and isolated homes/configs.
// Keep each file's cases serial; overlap three independent files.
// An explicit shared test database cannot be isolated this way.
...(hasSharedTestDatabase
? serializedServerVitestArgs
: ["--pool=forks", "--isolate", "--fileParallelism", "--maxWorkers=3"]),
...shardFiles,
],
`${groupName} shard ${shardIndex + 1}/${shardCount}`,
);
return;
}
const excludeRouteArgs = routeTests.flatMap((file) => ["--exclude", file.serverPath]);
for (const suite of excludedSuites) {
excludeRouteArgs.push("--exclude", suite.replace(/^server\//, ""));
}
await runVitest(
[
"--project",
"@paperclipai/server",
...serializedServerVitestArgs,
...excludeRouteArgs,
],
`${groupName} server suites excluding ${routeTests.length} serialized suites`,
);
return;
}
if (groupName === generalWorkspacesAGroupName) {
// The ui project dominates this lane (~224s of a 319s job in actions run
// 31371439296, 2026-08-10, where workspaces-a was the slowest PR check).
// Its 439 test files shard cleanly with Vitest's native --shard, so the
// lane splits across runners without a duration manifest.
await runProjectGroup(generalWorkspacesAProjects, groupName, shardIndex, shardCount);
return;
}
if (groupName === generalWorkspacesBGroupName) {
await runProjectGroup(generalWorkspacesBProjects, groupName);
return;
}
if (groupName === generalWorkspacesCGroupName) {
await runProjectGroup(generalWorkspacesCProjects, groupName);
return;
}
fail(`Unknown group "${groupName}".`);
}
async function runSerializedSuites(routeTests, shardIndex, shardCount) {
const shardTests = selectSerializedSuites(routeTests, shardIndex, shardCount);
console.log(
`\n[test:run] serialized shard ${shardIndex + 1}/${shardCount} running ${shardTests.length} of ${routeTests.length} suites`,
);
if (shardTests.length > 0) {
await runVitest(
[
"--project",
"@paperclipai/server",
...shardTests.map((test) => test.repoPath),
"--pool=forks",
"--isolate",
...serializedServerVitestArgs,
],
`serialized shard ${shardIndex + 1}/${shardCount}`,
);
}
}
const routeTests = walk(serverTestsDir)
.filter((file) => isRouteOrAuthzTest(toRepoPath(file)))
.map((file) => ({
repoPath: toRepoPath(file),
serverPath: toServerPath(file),
}))
.sort((a, b) => a.repoPath.localeCompare(b.repoPath));
// Every server test file that the general-server group is responsible for,
// i.e. the whole server project minus the route/authz suites that run in the
// dedicated serialized shards. The default server config keeps files serial;
// the sharded entry point overlaps two isolated files per runner unless an
// explicit shared test database requires serial execution.
// Suites are partitioned by recorded duration (scripts/general-server-shard.mjs)
// rather than round-robin, so one slow suite cluster can't stretch a single shard.
const serializedRepoPaths = new Set(routeTests.map(test => test.repoPath));
const generalServerTestFiles = [
...walk(serverSrcDir).filter(file => file.endsWith(".test.ts")),
...walk(serverScriptsDir).filter(file => file.endsWith(".test.mjs")),
]
.map((file) => toRepoPath(file))
// Only exclude suites actually assigned to the serialized lane. A name
// such as services/openrouter-models.test.ts is not a serialized route.
.filter((repoPath) => !serializedRepoPaths.has(repoPath))
.sort((a, b) => a.localeCompare(b));
const options = parseCliOptions(process.argv.slice(2));
const selectionProjects = options.mode === serializedModeName || generalServerGroups.includes(options.group)
? ["@paperclipai/server"]
: options.group === generalWorkspacesAGroupName ? generalWorkspacesAProjects
: options.group === generalWorkspacesBGroupName ? generalWorkspacesBProjects
: options.group === generalWorkspacesCGroupName ? generalWorkspacesCProjects : [];
const selection = process.env.PAPERCLIP_CI_TEST_SELECTION
? selectedShard(loadSelection(), options, selectionProjects) : null;
if (selection?.["@paperclipai/server"]) {
const owned = new Set(options.mode === serializedModeName ? routeTests.map((test) => test.repoPath)
: generalServerTestFiles.filter((file) => !excludedServerSuites(options.group).includes(file)));
for (const file of selection["@paperclipai/server"]) {
if (!owned.has(file)) fail(`Selected test belongs to another server lane: ${file}`);
}
}
if (selection && options.dryRun) {
writeFileSync(1, JSON.stringify({ selection }));
process.exit(0);
}
const defaultChatShards = !selection && options.group === null && options.mode !== serializedModeName && !hasSharedTestDatabase
? Array.from({ length: 6 }, (_, index) => ({ index, count: 6 }))
: null;
if (options.dryRun) {
const serializedSuites =
options.mode === serializedModeName
? selectSerializedSuites(routeTests, options.shardIndex, options.shardCount)
: routeTests;
// The inventory exceeds pipe buffers; flush it before the immediate exit.
writeFileSync(1,
JSON.stringify(
{
mode: options.mode,
shardIndex: options.shardIndex,
shardCount: options.shardCount,
group: options.group,
availableGeneralGroups: allowedGeneralGroupNames,
serializedSuiteCount: routeTests.length,
selectedSerializedSuites: serializedSuites.map((routeTest) => routeTest.repoPath),
generalServerSuiteCount: generalServerTestFiles.length,
selectedGeneralServerSuites:
defaultChatShards
? generalServerTestFiles.filter(file => file !== chatSuite)
: options.mode === generalModeName && options.group === generalServerNativeRunnerGroupName
? nativeRunnerSuites
: options.mode === generalModeName &&
generalServerGroups.includes(options.group) &&
options.shardCount !== null
? selectGeneralServerShard(
generalServerTestFiles.filter((file) => !excludedServerSuites(options.group).includes(file)),
options.shardIndex,
options.shardCount,
generalServerShardDurations,
)
: null,
chatVitestShards: options.group === generalChatGroupName
? chatShardWorkers(options.shardIndex, options.shardCount) : defaultChatShards,
workspaceProjects:
options.group === generalWorkspacesAGroupName
? generalWorkspacesAProjects
: options.group === generalWorkspacesBGroupName
? generalWorkspacesBProjects
: options.group === generalWorkspacesCGroupName
? generalWorkspacesCProjects
: null,
workspacesVitestShard:
options.group === generalWorkspacesAGroupName &&
options.shardCount !== null &&
options.shardCount > 1
? `${options.shardIndex + 1}/${options.shardCount}`
: null,
},
null,
2,
),
);
process.exit(0);
}
try {
if (selection) {
for (const [project, files] of Object.entries(selection)) {
const serverArgs = project !== "@paperclipai/server" ? []
: options.mode === serializedModeName || hasSharedTestDatabase
? ["--pool=forks", "--isolate", ...serializedServerVitestArgs]
: ["--pool=forks", "--isolate", "--fileParallelism", "--maxWorkers=3"];
await runVitest(["--project", project, ...serverArgs, ...files],
`selected ${project}: ${files.length} files`, null, true);
}
} else if (options.mode === generalModeName || options.mode === allModeName) {
if (options.group) {
await runGeneralGroup(routeTests, options.group, options.shardIndex, options.shardCount);
} else {
await runGeneralSuites(routeTests, defaultChatShards);
}
}
if (!selection && (options.mode === serializedModeName || options.mode === allModeName)) {
await runSerializedSuites(routeTests, options.shardIndex ?? 0, options.shardCount ?? 1);
}
} catch (error) {
console.error(`[test:run] ${error.message}`);
process.exitCode = 1;
}