mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 17:33:51 +02:00
## Thinking Path > - Paperclip manages work for AI agents. > - An environment plugin can own a Runner task. > - The merged launch-preparation contract lets the execution owner build the provider configuration. > - Core still needs a task process owner and a setup test that uses that configuration. > - This PR connects the task API to the Runner process-owner contract. > - Operators can test a plugin environment before Core sends it task work. ## Linked Issues or Issue Description **Subsystem affected** Runner, plugin SDK, and agent environment tests. **Problem or motivation** A task-only environment cannot use the shell-based setup path. Core must verify the Runner hello exchange and process cleanup through dedicated task operations. **Proposed solution** Add a lease-bound task driver and use it for a bounded setup probe. Keep heartbeat admission and task execution in #16978. This PR follows the merged launch-preparation contract in #16997. **Alternatives considered** A provider declaration alone cannot prove that Runner and the provider connect. A shell bridge would give the plugin an unnecessary command interface. ## What Changed - Pass structured Runner launch data to process owners. Support an external connection and explicit detach. - Add typed launch authentication checks and checkpoint operations. Check lease, run, project, and retained-task authority. - Connect launch preparation, PRP, termination proof, and credential handoff to the task driver. Reject local environment overrides during installed-account setup. Pass saved-account identity and credential home directly from Core. - Run a short-lived setup probe through the existing agent environment test API. Require a verified provider hello and cleanup. Keep saved Copilot account checks metadata-only through a dedicated task operation. - Update the Copilot source identity because the shared transport source changed. Keep provider behavior and the earlier identity fixture. ## Verification - CI passes on `f987adec7`: 55 successful checks and 5 expected skips. Standard Greptile and Apex are both 5/5 with no open findings. The branch merges cleanly with current master. - Repository typecheck and build pass after the rebase onto the workspace-provider contract. Server typecheck and build also pass after the final setup fix. - The 110 focused lease, SDK, setup, Copilot, and route cases pass after the rebase. All 14 setup cases pass after the final change. These include a real local Runner with a test provider. The saved-agent model-discovery case also passes. - Apex found that Core's relative workspace marker failed the driver path check, and non-Codex sessions omitted the observed model. Both failures were reproduced and fixed. The probe leaves physical workspace selection to plugin preparation. The transport reads each provider's observed model and exposes it to the driver. All 14 setup cases, 6 transport peer cases, and both existing real Runner/OpenCode cases pass. Runner build and server typecheck pass. - The coverage audit removed two redundant cases, repeated fixture code, and two extra real Runner launches. It kept authorization, credential selection, cancellation, lost replies, termination proof, checkpoint identity, and provider setup checks. - The Copilot source-identity check passes. The added-line secret and PII scan passes. The unmerged v52 attestation was regenerated after the shared transport fix. - CI on `13faeff02` caught one stale Copilot digest assertion after attestation regeneration. The literal now matches the candidate attestation, and all 49 installation cases pass. That CI lane otherwise passed 1,818 cases. - An earlier CI run hit an unchanged UI toast timing test. The separate fix is tracked in #16952. Current CI passes that shard. CI also caught a stale model-discovery assertion in this change; the assertion is fixed. - The local full-suite run was stopped when master changed. The final CI run supplies exact-head full-suite validation. No live Cloud or provider test was run. ## Risks - Native hello tests require a saved Core account or the installed account. Local environment overrides are unsupported for installed-account probes and fail before the provider runs. - A native task provider must declare launch preparation and authentication-check version 1. It must keep the checked configuration fixed through submission. Core rejects providers that cannot make this guarantee. - A lost task reply does not prove process termination. Core retains grants until the provider proves termination and Core saves required state. - This PR does not enable native heartbeat dispatch through plugin environments. Cloud deployment and live provider validation remain separate. ## Model Used OpenAI Codex, GPT-6. The exact model ID and context window are not exposed in this session. Tool use includes code editing, local tests, and the GitHub CLI. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal issue details - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
688 lines
27 KiB
JavaScript
688 lines
27 KiB
JavaScript
#!/usr/bin/env node
|
|
import { spawn, spawnSync } from "node:child_process";
|
|
import { mkdirSync, mkdtempSync, readdirSync, readFileSync, realpathSync, statSync, writeFileSync } from "node:fs";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { fileURLToPath } from "node:url";
|
|
import { loadShardDurations, selectGeneralServerShard } from "./general-server-shard.mjs";
|
|
import { invocationEnv } from "./ci-profile/record.mjs";
|
|
|
|
import { assertSelectedTests, partitionTestLines } from "./test-line-shard.mjs";
|
|
|
|
const repoRoot = process.cwd();
|
|
const scriptsDir = path.dirname(fileURLToPath(import.meta.url));
|
|
const generalServerShardDurations = loadShardDurations(
|
|
path.join(scriptsDir, "general-server-shard-durations.json"),
|
|
);
|
|
const serializedShardDurations = loadShardDurations(
|
|
path.join(scriptsDir, "serialized-shard-durations.json"),
|
|
);
|
|
const serverRoot = path.join(repoRoot, "server");
|
|
const serverSrcDir = path.join(repoRoot, "server", "src");
|
|
const serverTestsDir = path.join(repoRoot, "server", "src", "__tests__");
|
|
const serverScriptsDir = path.join(repoRoot, "server", "scripts");
|
|
const nonServerProjects = [
|
|
"@paperclipai/shared",
|
|
"@paperclipai/skills-catalog",
|
|
"@paperclipai/db",
|
|
"@paperclipai/adapter-utils",
|
|
"@paperclipai/adapter-claude-local",
|
|
"@paperclipai/adapter-codex-local",
|
|
"@paperclipai/adapter-cursor-cloud",
|
|
"@paperclipai/adapter-cursor-local",
|
|
"@paperclipai/adapter-gemini-local",
|
|
"@paperclipai/adapter-grok-local",
|
|
"@paperclipai/hermes-paperclip-adapter",
|
|
"@paperclipai/adapter-kimi-local",
|
|
"@paperclipai/adapter-openclaw-gateway",
|
|
"@paperclipai/adapter-opencode-local",
|
|
"@paperclipai/adapter-pi-local",
|
|
"@paperclipai/plugin-daytona",
|
|
"@paperclipai/plugin-sdk",
|
|
"@paperclipai/create-paperclip-plugin",
|
|
"@paperclipai/ui",
|
|
"paperclipai",
|
|
];
|
|
const routeTestPattern = /[^/]*(?:route|routes|authz)[^/]*\.test\.ts$/;
|
|
const additionalSerializedServerTests = new Set([
|
|
"server/src/__tests__/approval-routes-idempotency.test.ts",
|
|
"server/src/__tests__/assets.test.ts",
|
|
"server/src/__tests__/authz-company-access.test.ts",
|
|
"server/src/__tests__/companies-route-path-guard.test.ts",
|
|
"server/src/__tests__/company-portability.test.ts",
|
|
"server/src/__tests__/costs-service.test.ts",
|
|
"server/src/__tests__/express5-auth-wildcard.test.ts",
|
|
"server/src/__tests__/health-dev-server-token.test.ts",
|
|
"server/src/__tests__/health.test.ts",
|
|
"server/src/__tests__/heartbeat-dependency-scheduling.test.ts",
|
|
"server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts",
|
|
"server/src/__tests__/heartbeat-process-recovery.test.ts",
|
|
"server/src/__tests__/invite-accept-existing-member.test.ts",
|
|
"server/src/__tests__/invite-accept-gateway-defaults.test.ts",
|
|
"server/src/__tests__/invite-accept-replay.test.ts",
|
|
"server/src/__tests__/invite-expiry.test.ts",
|
|
"server/src/__tests__/invite-join-manager.test.ts",
|
|
"server/src/__tests__/invite-onboarding-text.test.ts",
|
|
"server/src/__tests__/invite-url-public-base-url.test.ts",
|
|
"server/src/__tests__/issues-checkout-wakeup.test.ts",
|
|
"server/src/__tests__/issues-service.test.ts",
|
|
"server/src/__tests__/opencode-local-adapter-environment.test.ts",
|
|
"server/src/__tests__/project-routes-env.test.ts",
|
|
"server/src/__tests__/redaction.test.ts",
|
|
"server/src/__tests__/routines-e2e.test.ts",
|
|
]);
|
|
let invocationIndex = 0;
|
|
const serializedModeName = "serialized";
|
|
const generalModeName = "general";
|
|
const allModeName = "all";
|
|
const generalServerGroupName = "general-server";
|
|
const generalServerWithoutChatGroupName = "general-server-without-chat";
|
|
const generalServerWithoutChatOrNativeRunnerGroupName = "general-server-without-chat-or-native-runner";
|
|
const generalChatGroupName = "general-chat";
|
|
const generalServerNativeRunnerGroupName = "general-server-native-runner";
|
|
const chatSuite = "server/src/__tests__/chat-channels.integration.test.ts";
|
|
// The first suite rebuilds the Runner release binaries with cargo in beforeAll.
|
|
// Inside the PR workflow's plain server shards, which carry no Rust cache,
|
|
// that build was a ~4m30s cold compile of every third-party crate on each run
|
|
// (277s of a 291s shard vitest step, actions run 35246999382, 2026-09-17).
|
|
const nativeRunnerSuites = [
|
|
"server/src/services/native-runtime/plugin-runner-probe.test.ts",
|
|
"server/src/services/native-runtime/native-codex-runner.integration.test.ts",
|
|
"server/src/__tests__/dot-runner.test.ts",
|
|
// These process cases otherwise skip in server shards without Runner binaries.
|
|
"server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts",
|
|
];
|
|
// In the PR workflow (pr.yml, the caller of pr-trusted.yml — reusable
|
|
// workflows inherit the caller's GITHUB_WORKFLOW), the last Verify Paperclip
|
|
// Runner vitest shard runs the native-runner group instead, because those
|
|
// lanes restore the shared release-runner-v2 Rust cache (see
|
|
// packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs). Every other
|
|
// caller of the same group keeps the suite in the server shards, so a renamed or
|
|
// unknown workflow degrades to today's slower-but-covered behavior rather
|
|
// than dropping the suite.
|
|
const prWorkflowName = "PR";
|
|
const nativeRunnerSuiteRunsInRustCachedLane = process.env.GITHUB_WORKFLOW === prWorkflowName;
|
|
const withoutChatExcludedSuites = nativeRunnerSuiteRunsInRustCachedLane
|
|
? [chatSuite, ...nativeRunnerSuites]
|
|
: [chatSuite];
|
|
// Release verification selects this explicit group and runs the omitted suites
|
|
// in its cached Runner job. Local callers keep the complete default group.
|
|
const generalServerGroups = [
|
|
generalServerGroupName,
|
|
generalServerWithoutChatGroupName,
|
|
generalServerWithoutChatOrNativeRunnerGroupName,
|
|
];
|
|
function excludedServerSuites(groupName) {
|
|
if (groupName === generalServerWithoutChatOrNativeRunnerGroupName) return [chatSuite, ...nativeRunnerSuites];
|
|
if (groupName === generalServerWithoutChatGroupName) return withoutChatExcludedSuites;
|
|
return [];
|
|
}
|
|
const generalWorkspacesAGroupName = "general-workspaces-a";
|
|
const generalWorkspacesBGroupName = "general-workspaces-b";
|
|
const generalWorkspacesCGroupName = "general-workspaces-c";
|
|
const generalWorkspacesAProjects = ["@paperclipai/ui", "paperclipai"];
|
|
// The db project runs in its own lane. Its vitest invocation took 168-250s
|
|
// of a 436-513s workspaces-b job (PR runs 38084171761, 38083084276 and
|
|
// 38083020834, 2026-10-10), which made workspaces-b the slowest check once
|
|
// the server shards were rebalanced. Vitest's native --shard would not help
|
|
// here: the project is already file-parallel and client.test.ts alone is
|
|
// 117-189s of its wall time, so a hash-ordered slice can land both of its
|
|
// slow suites on one runner. Splitting by project is deterministic and
|
|
// leaves the remaining projects at ~265-275s of vitest time, roughly level
|
|
// with this lane.
|
|
const generalWorkspacesCProjects = ["@paperclipai/db"];
|
|
const generalWorkspacesBProjects = nonServerProjects.filter(
|
|
(project) => !generalWorkspacesAProjects.includes(project) && !generalWorkspacesCProjects.includes(project),
|
|
);
|
|
const generalGroupNames = [
|
|
generalServerGroupName,
|
|
generalWorkspacesAGroupName,
|
|
generalWorkspacesBGroupName,
|
|
generalWorkspacesCGroupName,
|
|
];
|
|
const allowedGeneralGroupNames = [
|
|
...generalGroupNames,
|
|
generalServerWithoutChatGroupName,
|
|
generalServerWithoutChatOrNativeRunnerGroupName,
|
|
generalChatGroupName,
|
|
generalServerNativeRunnerGroupName,
|
|
];
|
|
const serializedServerVitestArgs = [
|
|
"--no-file-parallelism",
|
|
"--maxWorkers=1",
|
|
];
|
|
const sourceOnlyVitestArgs = ["--exclude", "**/dist/**"];
|
|
|
|
function walk(dir) {
|
|
const entries = readdirSync(dir);
|
|
const files = [];
|
|
for (const entry of entries) {
|
|
const absolute = path.join(dir, entry);
|
|
const stats = statSync(absolute);
|
|
if (stats.isDirectory()) {
|
|
files.push(...walk(absolute));
|
|
} else if (stats.isFile()) {
|
|
files.push(absolute);
|
|
}
|
|
}
|
|
return files;
|
|
}
|
|
|
|
function toRepoPath(file) {
|
|
return path.relative(repoRoot, file).split(path.sep).join("/");
|
|
}
|
|
|
|
function toServerPath(file) {
|
|
return path.relative(serverRoot, file).split(path.sep).join("/");
|
|
}
|
|
|
|
// For a multi-line `it.each([...])("name", fn)` call, Vitest 5's `vitest
|
|
// list --includeTaskLocation` and `vitest run <file>:<line>` disagree about
|
|
// which line registers the test: `list` reports (and only matches) the line
|
|
// of the wrapped "name" argument, while `run` only matches the line above
|
|
// it, where Prettier's formatting puts that call's opening `(`. Separately,
|
|
// on this file `list` also occasionally invents an entry whose "location"
|
|
// sits on an ordinary body statement that merely ends in `)(` or starts a
|
|
// line with a quote, with no `it`/`test` call anywhere nearby — a bogus
|
|
// artifact of the same collector, not a real test. Both only showed up on
|
|
// server/src/__tests__/chat-channels.integration.test.ts, a 70,000+ line
|
|
// fixture; smaller files have not reproduced either issue.
|
|
//
|
|
// chatSuiteRunLine classifies a `list`-reported line against the actual
|
|
// source and returns the line `vitest run` accepts, or null if the line
|
|
// does not belong to any real `it`/`test` call (a bogus entry to drop).
|
|
const callAnywherePattern = /(?:^|[^\w$])(?:it|test)(?:\.(?:each|skip|only))?\(/;
|
|
const inlineCallPattern = /\)\(\s*["'`]/;
|
|
|
|
function chatSuiteRunLine(sourceLines, line) {
|
|
const text = (sourceLines[line - 1] ?? "").trim();
|
|
if (callAnywherePattern.test(text) || text.endsWith(")(") || inlineCallPattern.test(text)) {
|
|
return line;
|
|
}
|
|
const previous = (sourceLines[line - 2] ?? "").trim();
|
|
const isNameArgument = text.startsWith('"') || text.startsWith("'") || text.startsWith("`");
|
|
if (isNameArgument && previous.endsWith(")(")) {
|
|
return line - 1;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
function isRouteOrAuthzTest(file) {
|
|
if (routeTestPattern.test(file)) {
|
|
return true;
|
|
}
|
|
|
|
return additionalSerializedServerTests.has(file);
|
|
}
|
|
|
|
function fail(message) {
|
|
console.error(`[test:run] ${message}`);
|
|
process.exit(1);
|
|
}
|
|
|
|
function readOptionValue(argv, index, argName) {
|
|
const value = argv[index + 1];
|
|
if (value === undefined) {
|
|
fail(`Missing value for ${argName}`);
|
|
}
|
|
|
|
return value;
|
|
}
|
|
|
|
function parseNonNegativeInteger(value, argName) {
|
|
const parsed = Number(value);
|
|
if (value.trim() === "" || !Number.isInteger(parsed) || parsed < 0) {
|
|
fail(`${argName} must be a non-negative integer. Received "${value}".`);
|
|
}
|
|
|
|
return parsed;
|
|
}
|
|
|
|
function parsePositiveInteger(value, argName) {
|
|
const parsed = Number(value);
|
|
if (value.trim() === "" || !Number.isInteger(parsed) || parsed < 1) {
|
|
fail(`${argName} must be a positive integer. Received "${value}".`);
|
|
}
|
|
|
|
return parsed;
|
|
}
|
|
|
|
function parseCliOptions(argv) {
|
|
let mode = allModeName;
|
|
let shardIndex = null;
|
|
let shardCount = null;
|
|
let group = null;
|
|
let dryRun = false;
|
|
|
|
for (let index = 0; index < argv.length; index += 1) {
|
|
const arg = argv[index];
|
|
if (arg === "--") {
|
|
continue;
|
|
}
|
|
|
|
if (arg === "--mode") {
|
|
mode = readOptionValue(argv, index, arg);
|
|
index += 1;
|
|
continue;
|
|
}
|
|
|
|
if (arg.startsWith("--mode=")) {
|
|
mode = arg.slice("--mode=".length);
|
|
continue;
|
|
}
|
|
|
|
if (arg === "--shard-index") {
|
|
shardIndex = parseNonNegativeInteger(readOptionValue(argv, index, arg), arg);
|
|
index += 1;
|
|
continue;
|
|
}
|
|
|
|
if (arg.startsWith("--shard-index=")) {
|
|
shardIndex = parseNonNegativeInteger(arg.slice("--shard-index=".length), "--shard-index");
|
|
continue;
|
|
}
|
|
|
|
if (arg === "--shard-count") {
|
|
shardCount = parsePositiveInteger(readOptionValue(argv, index, arg), arg);
|
|
index += 1;
|
|
continue;
|
|
}
|
|
|
|
if (arg.startsWith("--shard-count=")) {
|
|
shardCount = parsePositiveInteger(arg.slice("--shard-count=".length), "--shard-count");
|
|
continue;
|
|
}
|
|
|
|
if (arg === "--dry-run") {
|
|
dryRun = true;
|
|
continue;
|
|
}
|
|
|
|
if (arg === "--group") {
|
|
group = readOptionValue(argv, index, arg);
|
|
index += 1;
|
|
continue;
|
|
}
|
|
|
|
if (arg.startsWith("--group=")) {
|
|
group = arg.slice("--group=".length);
|
|
continue;
|
|
}
|
|
|
|
fail(`Unknown argument "${arg}".`);
|
|
}
|
|
|
|
if (!new Set([allModeName, generalModeName, serializedModeName]).has(mode)) {
|
|
fail(`Unknown mode "${mode}". Expected one of: ${allModeName}, ${generalModeName}, ${serializedModeName}.`);
|
|
}
|
|
|
|
if ((shardIndex === null) !== (shardCount === null)) {
|
|
fail("--shard-index and --shard-count must be provided together.");
|
|
}
|
|
|
|
const shardAllowed =
|
|
mode === serializedModeName ||
|
|
(mode === generalModeName &&
|
|
([...generalServerGroups, generalChatGroupName, generalWorkspacesAGroupName].includes(group)));
|
|
if (!shardAllowed && shardIndex !== null) {
|
|
fail(
|
|
"--shard-index/--shard-count are only valid with serialized mode or a shardable general server/chat/workspaces-a group.",
|
|
);
|
|
}
|
|
|
|
if (group !== null && mode !== generalModeName) {
|
|
fail("--group is only valid with --mode general.");
|
|
}
|
|
|
|
if (group !== null && !allowedGeneralGroupNames.includes(group)) {
|
|
fail(`Unknown group "${group}". Expected one of: ${allowedGeneralGroupNames.join(", ")}.`);
|
|
}
|
|
|
|
if (shardIndex !== null) {
|
|
if (shardIndex >= shardCount) {
|
|
fail(`--shard-index must be less than --shard-count. Received ${shardIndex} of ${shardCount}.`);
|
|
}
|
|
}
|
|
|
|
if (mode === serializedModeName) {
|
|
return {
|
|
mode,
|
|
shardIndex: shardIndex ?? 0,
|
|
shardCount: shardCount ?? 1,
|
|
group: null,
|
|
dryRun,
|
|
};
|
|
}
|
|
|
|
return {
|
|
mode,
|
|
shardIndex,
|
|
shardCount,
|
|
group,
|
|
dryRun,
|
|
};
|
|
}
|
|
|
|
function selectSerializedSuites(routeTests, shardIndex, shardCount) {
|
|
// Same duration-aware LPT partition as the general-server lane. Round-robin
|
|
// over the alphabetical list clustered the heavy heartbeat/issues suites on
|
|
// one shard (291s vs 170-201s test steps across the matrix in actions run
|
|
// 32012408876), which made that shard the whole PR run's slowest check.
|
|
const byRepoPath = new Map(routeTests.map((routeTest) => [routeTest.repoPath, routeTest]));
|
|
const shardFiles = selectGeneralServerShard(
|
|
routeTests.map((routeTest) => routeTest.repoPath),
|
|
shardIndex,
|
|
shardCount,
|
|
serializedShardDurations,
|
|
);
|
|
return shardFiles.map((file) => byRepoPath.get(file));
|
|
}
|
|
|
|
async function runVitest(args, label, testShard = null) {
|
|
console.log(`\n[test:run] ${label}`);
|
|
invocationIndex += 1;
|
|
const tempRootParent = process.platform === "win32" ? os.tmpdir() : "/tmp";
|
|
// Production workspace/security checks reject symlink aliases. In particular
|
|
// /tmp is /private/tmp on macOS, so fixture roots must use the canonical path.
|
|
const testRoot = realpathSync(mkdtempSync(path.join(tempRootParent, "pv-")));
|
|
// Keep per-run paths compact so Unix socket fixtures stay under macOS path limits.
|
|
const env = {
|
|
...process.env,
|
|
NODE_ENV: "test",
|
|
PAPERCLIP_TEST_HOST_HOME: process.env.PAPERCLIP_TEST_HOST_HOME
|
|
?? (process.env.PAPERCLIP_HOME?.trim() || path.join(os.homedir(), ".paperclip")),
|
|
PAPERCLIP_HOME: path.join(testRoot, "h"),
|
|
// Config discovery otherwise prefers the checkout's .paperclip/config.json
|
|
// over PAPERCLIP_HOME, importing preview scheduling policy into unit tests.
|
|
PAPERCLIP_CONFIG: path.join(testRoot, "h", "config.json"),
|
|
PAPERCLIP_INSTANCE_ID: `vt-${process.pid}-${invocationIndex}`,
|
|
TMPDIR: path.join(testRoot, "t"),
|
|
};
|
|
mkdirSync(env.PAPERCLIP_HOME, { recursive: true });
|
|
mkdirSync(env.TMPDIR, { recursive: true });
|
|
if (testShard) {
|
|
const file = path.resolve(repoRoot, chatSuite);
|
|
const sourceLines = readFileSync(file, "utf8").split("\n");
|
|
const collect = (filters, name) => {
|
|
const output = path.join(testRoot, `${name}.json`);
|
|
const result = spawnSync("pnpm", ["exec", "vitest", "list", ...sourceOnlyVitestArgs,
|
|
...filters, "--allowOnly=false", "--includeTaskLocation", `--json=${output}`], {
|
|
cwd: repoRoot, env, stdio: "inherit",
|
|
});
|
|
if (result.error || result.status !== 0) throw new Error(`Vitest collection failed: ${result.error?.message ?? result.status}`);
|
|
return JSON.parse(readFileSync(output, "utf8"));
|
|
};
|
|
const allCollected = collect(args, "all");
|
|
const collected = [];
|
|
const bogusCollected = [];
|
|
for (const test of allCollected) {
|
|
(chatSuiteRunLine(sourceLines, test.location.line) === null ? bogusCollected : collected).push(test);
|
|
}
|
|
if (bogusCollected.length > 0) {
|
|
console.warn(
|
|
`[test:run] dropping ${bogusCollected.length} bogus Vitest collection entr${bogusCollected.length === 1 ? "y" : "ies"} with no matching it()/test() call: ${bogusCollected.map((test) => `${test.location.line}:${JSON.stringify(test.name)}`).join(", ")}`,
|
|
);
|
|
}
|
|
const selected = partitionTestLines(collected, testShard.count, file)[testShard.index];
|
|
// Validate shard membership with the same `list`-reported lines that
|
|
// produced `selected`, then switch to the `run`-compatible lines only for
|
|
// the filters handed to the real `vitest run` below (see
|
|
// chatSuiteRunLine).
|
|
const validationFilters = selected.lines.map((line) => `${chatSuite}:${line}`);
|
|
const validationArgs = [...args.filter((arg) => arg !== chatSuite), ...validationFilters];
|
|
assertSelectedTests(selected.tests, collect(validationArgs, "selected"), file);
|
|
console.log(`[test:run] chat shard ${testShard.index + 1}/${testShard.count}: ${selected.tests.length}/${collected.length} tests, ${selected.lines.length} source lines; exact filter coverage verified`);
|
|
const runFilters = selected.lines.map((line) => `${chatSuite}:${chatSuiteRunLine(sourceLines, line)}`);
|
|
args = [...args.filter((arg) => arg !== chatSuite), ...runFilters, "--allowOnly=false"];
|
|
}
|
|
await new Promise((resolve, reject) => {
|
|
const child = spawn("pnpm", ["exec", "vitest", "run", ...sourceOnlyVitestArgs, ...args], {
|
|
cwd: repoRoot,
|
|
env: invocationEnv("vitest", env),
|
|
stdio: "inherit",
|
|
});
|
|
child.once("error", reject);
|
|
child.once("close", (code, signal) => {
|
|
if (code === 0) resolve();
|
|
else reject(new Error(`${label} failed (${signal ?? code})`));
|
|
});
|
|
});
|
|
}
|
|
|
|
async function runGeneralSuites(routeTests) {
|
|
for (const groupName of generalGroupNames) {
|
|
await runGeneralGroup(routeTests, groupName);
|
|
}
|
|
}
|
|
|
|
async function runProjectGroup(projects, groupName, shardIndex = null, shardCount = null) {
|
|
// With shard args, lean on Vitest's native --shard: each matrix job runs the
|
|
// same per-project invocations but only its slice of each project's test
|
|
// files. Vitest's sharding is deterministic for an identical file list, so
|
|
// the matrix jobs form a complete, non-overlapping cover of every project.
|
|
const shardArgs =
|
|
shardCount !== null && shardCount > 1 ? [`--shard=${shardIndex + 1}/${shardCount}`] : [];
|
|
const shardSuffix = shardArgs.length > 0 ? ` shard ${shardIndex + 1}/${shardCount}` : "";
|
|
for (const project of projects) {
|
|
await runVitest(["--project", project, ...shardArgs], `${groupName} project ${project}${shardSuffix}`);
|
|
}
|
|
}
|
|
|
|
function chatShardWorkers(index, count) {
|
|
const workers = count > 1 && !process.env.PAPERCLIP_TEST_DATABASE_URL ? 2 : 1;
|
|
return Array.from({ length: workers }, (_, offset) => ({
|
|
index: (index ?? 0) * workers + offset,
|
|
count: (count ?? 1) * workers,
|
|
}));
|
|
}
|
|
|
|
async function runGeneralGroup(routeTests, groupName, shardIndex = null, shardCount = null) {
|
|
if (groupName === generalChatGroupName) {
|
|
// Each invocation gets its own temporary home and embedded PostgreSQL.
|
|
// Wait for both children even on failure so no sibling is abandoned.
|
|
const results = await Promise.allSettled(chatShardWorkers(shardIndex, shardCount).map((shard) =>
|
|
runVitest(["--project", "@paperclipai/server", ...serializedServerVitestArgs, chatSuite],
|
|
"chat integration test shard", shard),
|
|
));
|
|
const failed = results.find((result) => result.status === "rejected");
|
|
if (failed) throw failed.reason;
|
|
return;
|
|
}
|
|
if (groupName === generalServerNativeRunnerGroupName) {
|
|
await runVitest(
|
|
["--project", "@paperclipai/server", ...serializedServerVitestArgs, ...nativeRunnerSuites],
|
|
"native runner vertical-slice suite",
|
|
);
|
|
return;
|
|
}
|
|
if (generalServerGroups.includes(groupName)) {
|
|
const excludedSuites = excludedServerSuites(groupName);
|
|
const files = generalServerTestFiles.filter((file) => !excludedSuites.includes(file));
|
|
if (shardCount !== null && shardCount > 1) {
|
|
const shardFiles = selectGeneralServerShard(
|
|
files,
|
|
shardIndex,
|
|
shardCount,
|
|
generalServerShardDurations,
|
|
);
|
|
console.log(
|
|
`\n[test:run] general-server shard ${shardIndex + 1}/${shardCount} running ${shardFiles.length} of ${files.length} suites`,
|
|
);
|
|
if (shardFiles.length === 0) {
|
|
return;
|
|
}
|
|
|
|
await runVitest(
|
|
[
|
|
"--project",
|
|
"@paperclipai/server",
|
|
// Files have separate fork processes and isolated homes/configs.
|
|
// Keep each file's cases serial; overlap only two independent files.
|
|
// An explicit shared test database cannot be isolated this way.
|
|
...(process.env.PAPERCLIP_TEST_DATABASE_URL
|
|
? serializedServerVitestArgs
|
|
: ["--pool=forks", "--isolate", "--fileParallelism", "--maxWorkers=2"]),
|
|
...shardFiles,
|
|
],
|
|
`${groupName} shard ${shardIndex + 1}/${shardCount}`,
|
|
);
|
|
return;
|
|
}
|
|
|
|
const excludeRouteArgs = routeTests.flatMap((file) => ["--exclude", file.serverPath]);
|
|
for (const suite of excludedSuites) {
|
|
excludeRouteArgs.push("--exclude", suite.replace(/^server\//, ""));
|
|
}
|
|
await runVitest(
|
|
[
|
|
"--project",
|
|
"@paperclipai/server",
|
|
...serializedServerVitestArgs,
|
|
...excludeRouteArgs,
|
|
],
|
|
`${groupName} server suites excluding ${routeTests.length} serialized suites`,
|
|
);
|
|
return;
|
|
}
|
|
|
|
if (groupName === generalWorkspacesAGroupName) {
|
|
// The ui project dominates this lane (~224s of a 319s job in actions run
|
|
// 31371439296, 2026-08-10, where workspaces-a was the slowest PR check).
|
|
// Its 439 test files shard cleanly with Vitest's native --shard, so the
|
|
// lane splits across runners without a duration manifest.
|
|
await runProjectGroup(generalWorkspacesAProjects, groupName, shardIndex, shardCount);
|
|
return;
|
|
}
|
|
|
|
if (groupName === generalWorkspacesBGroupName) {
|
|
await runProjectGroup(generalWorkspacesBProjects, groupName);
|
|
return;
|
|
}
|
|
|
|
if (groupName === generalWorkspacesCGroupName) {
|
|
await runProjectGroup(generalWorkspacesCProjects, groupName);
|
|
return;
|
|
}
|
|
|
|
fail(`Unknown group "${groupName}".`);
|
|
}
|
|
|
|
async function runSerializedSuites(routeTests, shardIndex, shardCount) {
|
|
const shardTests = selectSerializedSuites(routeTests, shardIndex, shardCount);
|
|
console.log(
|
|
`\n[test:run] serialized shard ${shardIndex + 1}/${shardCount} running ${shardTests.length} of ${routeTests.length} suites`,
|
|
);
|
|
|
|
if (shardTests.length > 0) {
|
|
await runVitest(
|
|
[
|
|
"--project",
|
|
"@paperclipai/server",
|
|
...shardTests.map((test) => test.repoPath),
|
|
"--pool=forks",
|
|
"--isolate",
|
|
...serializedServerVitestArgs,
|
|
],
|
|
`serialized shard ${shardIndex + 1}/${shardCount}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
const routeTests = walk(serverTestsDir)
|
|
.filter((file) => isRouteOrAuthzTest(toRepoPath(file)))
|
|
.map((file) => ({
|
|
repoPath: toRepoPath(file),
|
|
serverPath: toServerPath(file),
|
|
}))
|
|
.sort((a, b) => a.repoPath.localeCompare(b.repoPath));
|
|
|
|
// Every server test file that the general-server group is responsible for,
|
|
// i.e. the whole server project minus the route/authz suites that run in the
|
|
// dedicated serialized shards. The default server config keeps files serial;
|
|
// the sharded entry point overlaps two isolated files per runner unless an
|
|
// explicit shared test database requires serial execution.
|
|
// Suites are partitioned by recorded duration (scripts/general-server-shard.mjs)
|
|
// rather than round-robin, so one slow suite cluster can't stretch a single shard.
|
|
const serializedRepoPaths = new Set(routeTests.map(test => test.repoPath));
|
|
const generalServerTestFiles = [
|
|
...walk(serverSrcDir).filter(file => file.endsWith(".test.ts")),
|
|
...walk(serverScriptsDir).filter(file => file.endsWith(".test.mjs")),
|
|
]
|
|
.map((file) => toRepoPath(file))
|
|
// Only exclude suites actually assigned to the serialized lane. A name
|
|
// such as services/openrouter-models.test.ts is not a serialized route.
|
|
.filter((repoPath) => !serializedRepoPaths.has(repoPath))
|
|
.sort((a, b) => a.localeCompare(b));
|
|
|
|
const options = parseCliOptions(process.argv.slice(2));
|
|
if (options.dryRun) {
|
|
const serializedSuites =
|
|
options.mode === serializedModeName
|
|
? selectSerializedSuites(routeTests, options.shardIndex, options.shardCount)
|
|
: routeTests;
|
|
// The inventory exceeds pipe buffers; flush it before the immediate exit.
|
|
writeFileSync(1,
|
|
JSON.stringify(
|
|
{
|
|
mode: options.mode,
|
|
shardIndex: options.shardIndex,
|
|
shardCount: options.shardCount,
|
|
group: options.group,
|
|
availableGeneralGroups: allowedGeneralGroupNames,
|
|
serializedSuiteCount: routeTests.length,
|
|
selectedSerializedSuites: serializedSuites.map((routeTest) => routeTest.repoPath),
|
|
generalServerSuiteCount: generalServerTestFiles.length,
|
|
selectedGeneralServerSuites:
|
|
options.mode === generalModeName && options.group === generalServerNativeRunnerGroupName
|
|
? nativeRunnerSuites
|
|
: options.mode === generalModeName &&
|
|
generalServerGroups.includes(options.group) &&
|
|
options.shardCount !== null
|
|
? selectGeneralServerShard(
|
|
generalServerTestFiles.filter((file) => !excludedServerSuites(options.group).includes(file)),
|
|
options.shardIndex,
|
|
options.shardCount,
|
|
generalServerShardDurations,
|
|
)
|
|
: null,
|
|
chatVitestShards: options.group === generalChatGroupName
|
|
? chatShardWorkers(options.shardIndex, options.shardCount) : null,
|
|
workspaceProjects:
|
|
options.group === generalWorkspacesAGroupName
|
|
? generalWorkspacesAProjects
|
|
: options.group === generalWorkspacesBGroupName
|
|
? generalWorkspacesBProjects
|
|
: options.group === generalWorkspacesCGroupName
|
|
? generalWorkspacesCProjects
|
|
: null,
|
|
workspacesVitestShard:
|
|
options.group === generalWorkspacesAGroupName &&
|
|
options.shardCount !== null &&
|
|
options.shardCount > 1
|
|
? `${options.shardIndex + 1}/${options.shardCount}`
|
|
: null,
|
|
},
|
|
null,
|
|
2,
|
|
),
|
|
);
|
|
process.exit(0);
|
|
}
|
|
|
|
try {
|
|
if (options.mode === generalModeName || options.mode === allModeName) {
|
|
if (options.group) {
|
|
await runGeneralGroup(routeTests, options.group, options.shardIndex, options.shardCount);
|
|
} else {
|
|
await runGeneralSuites(routeTests);
|
|
}
|
|
}
|
|
|
|
if (options.mode === serializedModeName || options.mode === allModeName) {
|
|
await runSerializedSuites(routeTests, options.shardIndex ?? 0, options.shardCount ?? 1);
|
|
}
|
|
} catch (error) {
|
|
console.error(`[test:run] ${error.message}`);
|
|
process.exitCode = 1;
|
|
}
|