Files
PaperClipAI/scripts/__tests__/run-vitest-stable-shard.test.mjs
T
DottaandClaude Opus 5.5 03309f1650 fix(native): release an idle warm owner whose sandbox Runner port a later run took over (#21835)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm agent session keeps its Runner alive in a sandbox between
turns. The server closes idle sessions, and closes them again before it
exits.
> - A sandbox has one Runner port. Before a new run launches its Runner
on a reused sandbox, it retires every other Runner on that port.
> - In a zero-downtime release-swap test, a run finished on the outgoing
server and stayed warm. 77 ms later, a follow-up run started on the
incoming server. It reused the same sandbox, started a fresh provider
session, retired the old Runner, and launched its own Runner on the same
port.
> - The outgoing server's idle close could not suspend the old Runner.
Its checkpoint was correctly refused with
`native_runner_checkpoint_owner_lost`. The idle timer retried 3 times
and then kept the owner registered. That phantom owner blocked idle
sleep for 4.8 minutes, and the shutdown close failed again.
> - The existing proofs that a warm owner is gone (#21782) do not count
"a later run launched its own Runner on this sandbox".
> - This pull request adds that proof. When it holds, the server forgets
the owner without a checkpoint. The checkpoint fence does not change.
> - The benefit is that a superseded warm owner no longer blocks idle
sleep or fails a clean shutdown.

## Linked Issues or Issue Description

Refs #21627
Refs #21782

**What happened?**
An idle warm owner finished on the outgoing server. A follow-up run on
the incoming server reused the same sandbox and launched its own Runner
on the sandbox's single Runner port. The owner's idle close failed with
`native_runner_checkpoint_owner_lost` 4 times. The owner stayed
registered, held the idle-work flag, and blocked idle sleep. At SIGTERM
the restart sweep logged "idle warm native sessions did not close" with
"runner did not durably suspend before checkpoint". No task was
affected.

**Expected behavior**
A failed idle close releases its owner when a later run proves that the
owner's Runner is gone. The checkpoint stays refused while the successor
is live.

**Steps to reproduce**
1. Finish a warm run on a sandbox and keep it idle.
2. Start a successor run for the same task on another server. Give it a
lease on the same sandbox, its own `native.process_identity_recorded`
event and its own `session.started` event.
3. Call `closeIdleWarmNativeSessionsForRestart()`. On master the result
is `closed: 0, failed: 1`, and `readNativeSessionIdleWork()` stays
`present`.

## What Changed

- `server/src/services/native-runtime/native-session-executor.ts`: new
`warmRunnerPortTakenBySuccessor`, used by
`warmCloseFailureProvesOwnerGone` after the existing sandbox-termination
proof. It holds only when another run has all of these:
- an `environment_leases` row with the same environment and
`providerLeaseId` as the owner;
- its own `native.process_identity_recorded` event (`source_event_id is
null`, so an inherited identity does not count);
- a `session.started` event from its own Runner instance
(`source_instance_id` equals the run's `runner_instance_id`);
  - both events created after the owner run's `finishedAt`.
- When the proof holds, the idle-timer path and the controller-restart
path log "warm native session's Runner or sandbox is gone; released its
owner" and forget the owner without a checkpoint. Both paths already
call `warmCloseFailureProvesOwnerGone`.
- `validateNativeRunnerCheckpointOwner` does not change. A checkpoint
still cannot land under a live successor.
- Cost: the proof runs only after a failed close. It does one
primary-key read of the owner run and one query on
`environment_leases.provider_lease_id` (indexed), joined to
`heartbeat_runs` by primary key, with `heartbeat_run_events` read
through `(company_id, run_id)`.
- New real-database test file
`server/src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`.
- `native-session-executor.test.ts`: one environment-delete test counts
the extra proof read when the first proof does not hold.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts`:
6 passed.
- "releases an idle warm owner whose checkpoint is refused because a
successor on another server launched its own Runner on the same sandbox:
own_runner": the close is refused with
`native_runner_checkpoint_owner_lost`, the result is `closed: 1, failed:
0`, the idle-work flag is `none`, the owner has no checkpoint row, and
`assertNativeRunnerCheckpointOwner` still throws
`native_runner_checkpoint_owner_lost`.
- These stay failed and keep the owner registered: the successor is on
another sandbox; the successor has only
`native.process_start_requested`; the successor has only inherited
events (`source_event_id` set); the successor's events come before the
owner finished.
- "releases a superseded idle warm owner from its idle timer without
writing a checkpoint": the idle timer releases the owner after the first
refused close.
- Red on master (test file only, fix reverted): `own_runner` fails with
`closed: 0, failed: 1`; the idle-timer test fails with the idle-work
flag still `present`. The 4 negative cases pass on master and with the
fix.
- `cd server && npx vitest run
src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts`:
9 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts`: all
passed.
- `cd server && npx tsc --noEmit -p .`: no errors.
- `pnpm verify`: see the comment below for the result.
- Not run: a restart-chaos wave. The release-swap timing is reproduced
in the database test, not on a live sandbox.

## Risks

- A superseded owner is now released without a checkpoint. Its old
provider conversation cannot be resumed. The successor had already
started a different conversation and holds the task, so no one would
resume the old one.
- The proof needs the successor's own Runner identity and own session
start after the owner finished, on the same sandbox lease. A successor
that reused the owner's Runner, only requested a start, or recorded its
events earlier does not count. In those cases the close fails as before.
- The proof adds two small indexed reads after a failed close only. The
normal close path does not change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used in Claude Code
with tool use (shell, file edit) and extended thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 17:49:01 -06:00

903 lines
49 KiB
JavaScript

import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { chmodSync, existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import test from "node:test";
import {
defaultSuiteWeight,
loadShardDurations,
partitionGeneralServerSuites,
} from "../general-server-shard.mjs";
import { assertManifestSource } from "../ci/pr-run-durations.mjs";
import { assertSelectedTests, partitionTestLines } from "../test-line-shard.mjs";
import { chatManifestData, entryDurations } from "../ci/refresh-chat-shard-durations.mjs";
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
const script = path.join(repoRoot, "scripts", "run-vitest-stable.mjs");
const durationsManifest = path.join(repoRoot, "scripts", "general-server-shard-durations.json");
const serializedDurationsManifest = path.join(
repoRoot,
"scripts",
"serialized-shard-durations.json",
);
// Membership of general-server-without-chat depends on GITHUB_WORKFLOW (see
// prWorkflowName in run-vitest-stable.mjs), so strip the ambient value and
// make every test pin the caller it mirrors explicitly — these tests
// themselves run inside a workflow on CI.
function workflowEnv(envOverrides = {}) {
const env = { ...process.env, ...envOverrides };
if (!("GITHUB_WORKFLOW" in envOverrides)) {
delete env.GITHUB_WORKFLOW;
}
return env;
}
function dryRun(args, envOverrides = {}) {
const result = spawnSync(process.execPath, [script, ...args, "--dry-run"], {
cwd: repoRoot,
encoding: "utf8",
env: workflowEnv(envOverrides),
});
return result;
}
function dryRunJson(args, envOverrides = {}) {
const result = dryRun(args, envOverrides);
assert.equal(result.status, 0, `expected success for ${args.join(" ")}: ${result.stderr}`);
return JSON.parse(result.stdout);
}
const SHARD_COUNT = 12;
const SERIALIZED_SHARD_COUNT = 9;
test("the general-server shards form a complete, non-overlapping partition", () => {
const shards = Array.from({ length: SHARD_COUNT }, (_, index) =>
dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", String(index), "--shard-count", String(SHARD_COUNT)]),
);
const total = shards[0].generalServerSuiteCount;
assert.ok(total > 0, "expected a non-empty general-server suite set");
const seen = new Set();
let selectedTotal = 0;
for (const shard of shards) {
assert.equal(shard.generalServerSuiteCount, total, "suite count must be stable across shards");
for (const file of shard.selectedGeneralServerSuites) {
assert.ok(
!(file.startsWith("server/src/__tests__/") && /[^/]*(?:route|routes|authz)[^/]*\.test\.ts$/.test(file)),
`route/authz suite must stay in the serialized lane, not general-server: ${file}`,
);
assert.ok(!seen.has(file), `suite assigned to more than one shard: ${file}`);
seen.add(file);
selectedTotal += 1;
}
}
// Every suite runs exactly once: union covers the whole set with no overlap.
assert.equal(selectedTotal, total, "every suite must be selected exactly once");
assert.equal(seen.size, total, "union of shards must cover the whole suite set");
});
test("general shards retain scripts and route-named suites outside the serialized directory", () => {
const general = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"]);
for (const file of [
"server/scripts/verify-runner-vendor-dependencies.test.mjs",
"server/src/services/openrouter-models.test.ts",
"server/src/routes/setup-token-route.test.ts",
]) assert.ok(general.selectedGeneralServerSuites.includes(file), `missing configured server suite: ${file}`);
});
for (const group of ["general-workspaces-b", "general-workspaces-c"]) {
test(`shard flags are rejected for the ${group} group`, () => {
const result = dryRun(["--mode", "general", "--group", group, "--shard-index", "0", "--shard-count", "3"]);
assert.notEqual(result.status, 0, `${group} must not accept shard flags`);
});
}
test("workspace lanes cover every non-server project in the root Vitest configuration", () => {
const config = readFileSync(path.join(repoRoot, "vitest.config.ts"), "utf8");
const roots = [...config.matchAll(/^\s+"([^"]+)",?\s*$/gm)].map(match => match[1]);
assert.ok(roots.includes("server"), "expected the explicit root Vitest project list");
const expected = roots.filter(root => root !== "server")
.map(root => JSON.parse(readFileSync(path.join(repoRoot, root, "package.json"), "utf8")).name).sort();
const actual = ["general-workspaces-a", "general-workspaces-b", "general-workspaces-c"]
.flatMap(group => dryRunJson(["--mode", "general", "--group", group]).workspaceProjects).sort();
assert.deepEqual(actual, expected, "no configured project may be silently omitted or run twice");
});
// Collects every `- group:` matrix entry in a workflow with the keys it sets.
// Comment and blank lines are dropped first, and an entry ends only when the
// indentation returns to the item level, so a comment between two keys cannot
// hide a shard setting from the assertions below.
function matrixEntries(text) {
const entries = [];
let current = null;
for (const line of text.split("\n")) {
if (/^\s*(#|$)/.test(line)) continue;
const indent = line.match(/^ */)[0].length;
const item = line.match(/^ *- group: (\S+)/);
if (item) {
current = { group: item[1], indent, keys: ["group"] };
entries.push(current);
continue;
}
if (!current) continue;
if (indent <= current.indent) { current = null; continue; }
const key = line.match(/^ *([A-Za-z_]+):/);
if (key) current.keys.push(key[1]);
}
return entries;
}
test("matrixEntries sees shard keys that sit behind comments or later in the entry", () => {
const entries = matrixEntries([
" include:",
" - group: general-workspaces-b",
" group_label: workspaces-b",
" # a comment between keys must not hide the shard setting",
" shard_index: 0",
"",
" shard_count: 3",
" - group: general-workspaces-c",
" group_label: workspaces-c",
" steps:",
" - name: unrelated",
" shard_index: 9",
].join("\n"));
assert.deepEqual(entries.map(entry => entry.group), ["general-workspaces-b", "general-workspaces-c"]);
assert.deepEqual(entries[0].keys, ["group", "group_label", "shard_index", "shard_count"]);
assert.deepEqual(entries[1].keys, ["group", "group_label"], "keys after the matrix must not leak into the last entry");
});
test("workspaces-c runs the db project alone and workspaces-b no longer carries it", () => {
// The db project was 168-250s of the 436-513s workspaces-b vitest time
// (PR runs 38084171761, 38083084276 and 38083020834, 2026-10-10), so it
// runs in its own lane. The PR and release matrices both carry that lane.
const workspacesC = dryRunJson(["--mode", "general", "--group", "general-workspaces-c"]);
assert.deepEqual(workspacesC.workspaceProjects, ["@paperclipai/db"]);
assert.equal(workspacesC.workspacesVitestShard, null);
const workspacesB = dryRunJson(["--mode", "general", "--group", "general-workspaces-b"]);
assert.ok(!workspacesB.workspaceProjects.includes("@paperclipai/db"), "db must not run twice");
assert.ok(workspacesB.workspaceProjects.length > 0, "workspaces-b must still run its other projects");
for (const workflow of ["pr-trusted.yml", "release-verify.yml"]) {
const entries = workflow === "pr-trusted.yml"
? JSON.parse(readFileSync(path.join(repoRoot, ".github/ci-task-matrix.json"), "utf8")).general_tests
.map((entry) => ({ ...entry, keys: Object.keys(entry) }))
: matrixEntries(readFileSync(path.join(repoRoot, ".github", "workflows", workflow), "utf8"));
// The parser must demonstrably see shard keys in this workflow, otherwise
// the unsharded assertions below could pass for the wrong reason.
const sharded = entries.filter(entry => entry.group === "general-workspaces-a");
assert.ok(sharded.length > 0 && sharded.every(entry => entry.keys.includes("shard_index") && entry.keys.includes("shard_count")),
`${workflow} must carry the sharded workspaces-a entries`);
for (const group of ["general-workspaces-b", "general-workspaces-c"]) {
const matching = entries.filter(entry => entry.group === group);
assert.equal(matching.length, 1, `${workflow} must run ${group} exactly once`);
assert.ok(matching[0].keys.includes("group_label"), `${workflow} must label the ${group} job`);
assert.deepEqual(matching[0].keys.filter(key => key.startsWith("shard_")), [], `${workflow} must run ${group} unsharded`);
}
}
});
test("workspaces-a shards map to Vitest native --shard slices over a stable project list", () => {
const shards = [0, 1].map((index) =>
dryRunJson([
"--mode", "general", "--group", "general-workspaces-a",
"--shard-index", String(index), "--shard-count", "2",
]),
);
assert.deepEqual(
shards.map((shard) => shard.workspacesVitestShard),
["1/2", "2/2"],
"each matrix job must pass its own --shard slice to vitest",
);
// Vitest's --shard partitions each project's file list deterministically, so
// an identical project list across jobs is what guarantees complete,
// non-overlapping coverage of the lane.
assert.deepEqual(shards[0].workspaceProjects, shards[1].workspaceProjects);
assert.ok(shards[0].workspaceProjects.length > 0, "workspaces-a must run at least one project");
const unsharded = dryRunJson(["--mode", "general", "--group", "general-workspaces-a"]);
assert.deepEqual(
unsharded.workspaceProjects,
shards[0].workspaceProjects,
"sharding must not change which projects the lane covers",
);
assert.equal(unsharded.workspacesVitestShard, null);
});
test("duration-aware partition balances skewed weights better than round-robin", () => {
// Round-robin puts all three heavy suites on shard 0 (indexes 0, 3, 6).
const files = ["a", "b", "c", "d", "e", "f", "g", "h", "i"];
const durations = { a: 30000, d: 30000, g: 30000, b: 100, c: 100, e: 100, f: 100, h: 100, i: 100 };
const shards = partitionGeneralServerSuites(files, 3, durations);
const totals = shards.map((shard) => shard.totalWeight);
const maxTotal = Math.max(...totals);
const minTotal = Math.min(...totals);
assert.ok(
maxTotal - minTotal <= 200,
`expected near-even shard weights, got ${totals.join(", ")}`,
);
assert.equal(
shards.flatMap((shard) => shard.files).sort().join(","),
files.join(","),
"partition must cover every file exactly once",
);
});
test("the partition is deterministic for identical inputs", () => {
const files = Array.from({ length: 50 }, (_, index) => `suite-${index}.test.ts`);
const durations = Object.fromEntries(files.map((file, index) => [file, (index * 37) % 5000]));
const first = partitionGeneralServerSuites(files, 3, durations);
const second = partitionGeneralServerSuites(files, 3, durations);
assert.deepEqual(first, second, "same inputs must always produce the same partition");
});
test("suites missing from the manifest get the median weight", () => {
assert.equal(defaultSuiteWeight({ a: 100, b: 300, c: 900 }), 300);
assert.equal(defaultSuiteWeight({ a: 100, b: 300, c: 500, d: 900 }), 400);
assert.equal(defaultSuiteWeight({}), 1000, "empty manifest falls back to a fixed weight");
});
test("a missing or malformed manifest degrades to uniform weights", () => {
assert.deepEqual(loadShardDurations(path.join(repoRoot, "scripts", "no-such-manifest.json")), {});
const files = ["a", "b", "c", "d"];
const shards = partitionGeneralServerSuites(files, 2, {});
assert.equal(shards[0].files.length + shards[1].files.length, files.length);
assert.equal(Math.abs(shards[0].files.length - shards[1].files.length), 0);
});
test("the checked-in manifest loads and covers most of the current suite set", () => {
const durations = loadShardDurations(durationsManifest);
assert.ok(Object.keys(durations).length > 0, "manifest must parse to a non-empty duration map");
const shard = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"]);
const currentFiles = shard.selectedGeneralServerSuites;
const known = currentFiles.filter((file) => durations[file] !== undefined).length;
assert.ok(
known / currentFiles.length >= 0.5,
`manifest is stale: only ${known} of ${currentFiles.length} suites have recorded durations — regenerate it from a recent PR run (see the manifest's $comment)`,
);
});
test("the chat integration suite keeps a measured duration for duration-aware fallbacks", () => {
// The PR and release matrices both run the chat suite in dedicated
// line-sharded lanes, but the plain general-server group (local full runs)
// still weighs it into the LPT partition; a median-fallback weight there
// would silently overload whichever shard receives it.
const chatSuite = "server/src/__tests__/chat-channels.integration.test.ts";
const durations = loadShardDurations(durationsManifest);
assert.ok(
Number.isFinite(durations[chatSuite]),
"the full chat cohort must have a measured duration, not the median fallback",
);
});
test("the checked-in serialized manifest loads and covers most of the current suite set", () => {
const durations = loadShardDurations(serializedDurationsManifest);
assert.ok(Object.keys(durations).length > 0, "manifest must parse to a non-empty duration map");
const shard = dryRunJson(["--mode", "serialized", "--shard-index", "0", "--shard-count", "1"]);
const currentFiles = shard.selectedSerializedSuites;
const known = currentFiles.filter((file) => durations[file] !== undefined).length;
assert.ok(
known / currentFiles.length >= 0.5,
`manifest is stale: only ${known} of ${currentFiles.length} suites have recorded durations — regenerate it from a recent PR run (see the manifest's $comment)`,
);
});
test("the real serialized shard partition is complete, non-overlapping and duration-balanced", () => {
const durations = loadShardDurations(serializedDurationsManifest);
const fallback = defaultSuiteWeight(durations);
const shards = Array.from({ length: SERIALIZED_SHARD_COUNT }, (_, index) =>
dryRunJson(["--mode", "serialized", "--shard-index", String(index), "--shard-count", String(SERIALIZED_SHARD_COUNT)]),
);
const total = shards[0].serializedSuiteCount;
const selected = shards.flatMap((shard) => shard.selectedSerializedSuites);
assert.equal(selected.length, total, "every serialized suite must be selected exactly once");
assert.equal(new Set(selected).size, total, "serialized shards must not overlap");
const totals = shards.map((shard) =>
shard.selectedSerializedSuites.reduce((sum, file) => sum + (durations[file] ?? fallback), 0),
);
const maxTotal = Math.max(...totals);
const minTotal = Math.min(...totals);
// LPT keeps the spread within the heaviest single suite; use that as the bound.
const heaviest = Math.max(...Object.values(durations));
assert.ok(
maxTotal - minTotal <= heaviest,
`serialized shard weight spread ${maxTotal - minTotal}ms exceeds heaviest suite ${heaviest}ms: ${totals.join(", ")}`,
);
});
const chatSuitePath = "server/src/__tests__/chat-channels.integration.test.ts";
const nativeRunnerSuitePath =
"server/src/services/native-runtime/native-codex-runner.integration.test.ts";
const ingressRetrySuitePath = "server/src/services/native-runtime/native-ingress-bootstrap-retry.integration.test.ts";
const usageDeliverySuitePath = "server/src/services/native-runtime/native-usage-delivery.integration.test.ts";
const warmCheckpointOwnershipSuitePath = "server/src/services/native-runtime/native-warm-checkpoint-ownership.integration.test.ts";
const warmSuccessorRunnerReleaseSuitePath = "server/src/services/native-runtime/native-warm-successor-runner-release.integration.test.ts";
const pluginWarmOwnerSuitePath = "server/src/services/native-runtime/plugin-runner-task-warm.integration.test.ts";
const pluginReleaseResumeSuitePath = "server/src/services/native-runtime/plugin-runner-release-resume.integration.test.ts";
const pluginEvictionSuitePath = "server/src/services/native-runtime/plugin-runner-eviction.integration.test.ts";
const dotRunnerSuitePath = "server/src/__tests__/dot-runner.test.ts";
const restartRecoverySuitePath =
"server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts";
const cutTurnStartRetrySuitePath = "server/src/services/native-runtime/native-cut-turn-start-retry.integration.test.ts";
const restartWaitSuites = ["lease-renewal", "preparation", "checkpoint-cost", "diskless"]
.map(name => `server/src/services/native-runtime/native-runner-restart-${name}.integration.test.ts`);
// Mirrors pr-trusted.yml (12 shards, called by pr.yml so GITHUB_WORKFLOW is
// "PR"): the chat suite runs in its dedicated lanes and the cargo-dependent
// native-runner suite in the Rust-cached final Verify Paperclip Runner vitest
// shard, so together the three cover the full server group exactly.
test("12 PR without-chat shards plus the dedicated chat and native-runner lanes cover the original server group exactly", () => {
const durations = loadShardDurations(durationsManifest);
const fallback = defaultSuiteWeight(durations);
const prEnv = { GITHUB_WORKFLOW: "PR" };
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], prEnv);
const shards = Array.from({ length: 12 }, (_, index) => dryRunJson([
"--mode", "general", "--group", "general-server-without-chat",
"--shard-index", String(index), "--shard-count", "12",
], prEnv));
const totals = shards.map((shard) =>
shard.selectedGeneralServerSuites.reduce((sum, file) => sum + (durations[file] ?? fallback), 0),
);
const maxTotal = Math.max(...totals);
const minTotal = Math.min(...totals);
// LPT keeps the spread within the heaviest single suite; use that as the
// bound. The chat and native-runner suites run in their own lanes, so
// exclude them here.
const chat = "server/src/__tests__/chat-channels.integration.test.ts";
const nativeRunner =
"server/src/services/native-runtime/native-codex-runner.integration.test.ts";
const heaviest = Math.max(
...Object.entries(durations)
.filter(([file]) => file !== chat && file !== nativeRunner)
.map(([, ms]) => ms),
);
assert.ok(
maxTotal - minTotal <= heaviest,
`shard weight spread ${maxTotal - minTotal}ms exceeds heaviest suite ${heaviest}ms: ${totals.join(", ")}`,
);
const files = shards.flatMap((shard) => shard.selectedGeneralServerSuites);
assert.ok(!files.includes(chatSuitePath));
assert.ok(!files.includes(nativeRunnerSuitePath));
assert.ok(!files.includes(ingressRetrySuitePath));
assert.ok(!files.includes(dotRunnerSuitePath));
assert.ok(!files.includes(restartRecoverySuitePath));
assert.ok(!files.includes(cutTurnStartRetrySuitePath));
assert.ok(!files.includes(usageDeliverySuitePath));
assert.ok(!files.includes(warmCheckpointOwnershipSuitePath));
assert.ok(!files.includes(warmSuccessorRunnerReleaseSuitePath));
assert.ok(!files.includes(pluginWarmOwnerSuitePath));
assert.ok(!files.includes(pluginReleaseResumeSuitePath));
assert.ok(!files.includes(pluginEvictionSuitePath));
assert.deepEqual([...files, chatSuitePath, nativeRunnerSuitePath, ingressRetrySuitePath, dotRunnerSuitePath, restartRecoverySuitePath, ...restartWaitSuites, cutTurnStartRetrySuitePath, usageDeliverySuitePath, warmCheckpointOwnershipSuitePath, warmSuccessorRunnerReleaseSuitePath, pluginWarmOwnerSuitePath, pluginReleaseResumeSuitePath, pluginEvictionSuitePath, "server/src/services/native-runtime/plugin-runner-probe.test.ts"].sort(), full.selectedGeneralServerSuites.sort());
assert.equal(new Set(files).size, files.length);
const defaultRun = dryRunJson([], prEnv);
assert.ok(defaultRun.generalServerSuiteCount === full.generalServerSuiteCount);
});
// The legacy group and local default must remain complete when a caller has
// not selected a separate native lane.
for (const [caller, envOverrides] of [["Release", { GITHUB_WORKFLOW: "Release" }], ["no ambient workflow", {}]]) {
test(`10 without-chat shards under ${caller} keep the native-runner suite and cover the server group with chat alone`, () => {
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], envOverrides);
const shards = Array.from({ length: 10 }, (_, index) => dryRunJson([
"--mode", "general", "--group", "general-server-without-chat",
"--shard-index", String(index), "--shard-count", "10",
], envOverrides));
const files = shards.flatMap((shard) => shard.selectedGeneralServerSuites);
assert.ok(!files.includes(chatSuitePath));
assert.ok(files.includes(nativeRunnerSuitePath));
assert.ok(files.includes(dotRunnerSuitePath));
assert.ok(files.includes(restartRecoverySuitePath));
assert.ok(files.includes(warmCheckpointOwnershipSuitePath));
assert.ok(files.includes(warmSuccessorRunnerReleaseSuitePath));
assert.ok(files.includes(pluginWarmOwnerSuitePath));
assert.ok(files.includes(pluginReleaseResumeSuitePath));
assert.ok(files.includes(pluginEvictionSuitePath));
assert.deepEqual([...files, chatSuitePath].sort(), full.selectedGeneralServerSuites.sort());
assert.equal(new Set(files).size, files.length);
});
}
// Release verification selects its dedicated lane explicitly. Do not infer it
// from the caller name: previews and future workflow callers need the same cover.
for (const caller of ["Release", "Cloud readiness", "another caller"]) {
test(`release server, chat, and native lanes cover every server suite once under ${caller}`, () => {
const env = { GITHUB_WORKFLOW: caller };
const full = dryRunJson(["--mode", "general", "--group", "general-server", "--shard-index", "0", "--shard-count", "1"], env);
const files = Array.from({ length: 10 }, (_, index) => dryRunJson([
"--mode", "general", "--group", "general-server-without-chat-or-native-runner",
"--shard-index", String(index), "--shard-count", "10",
], env)).flatMap((shard) => shard.selectedGeneralServerSuites);
const native = dryRunJson(["--mode", "general", "--group", "general-server-native-runner"], env);
const combined = [...files, chatSuitePath, ...native.selectedGeneralServerSuites];
assert.equal(new Set(combined).size, combined.length, "lanes must not overlap");
assert.deepEqual(combined.sort(), full.selectedGeneralServerSuites.sort());
});
}
test("the native-runner lane runs exactly the cargo-dependent suites", () => {
const lane = dryRunJson(["--mode", "general", "--group", "general-server-native-runner"]);
assert.deepEqual(lane.selectedGeneralServerSuites, [
"server/src/services/native-runtime/plugin-runner-probe.test.ts",
"server/src/services/native-runtime/native-codex-runner.integration.test.ts",
ingressRetrySuitePath,
dotRunnerSuitePath,
restartRecoverySuitePath,
...restartWaitSuites,
cutTurnStartRetrySuitePath,
usageDeliverySuitePath,
warmCheckpointOwnershipSuitePath,
warmSuccessorRunnerReleaseSuitePath,
pluginWarmOwnerSuitePath,
pluginReleaseResumeSuitePath,
pluginEvictionSuitePath,
]);
});
test("shard flags are rejected for the native-runner group", () => {
const result = dryRun(["--mode", "general", "--group", "general-server-native-runner", "--shard-index", "0", "--shard-count", "2"]);
assert.notEqual(result.status, 0, "the native-runner lane is a single suite and must not accept shard flags");
});
// The PR-side exclusion above is safe only while the wiring it assumes holds:
// pr.yml (the caller whose name reusable pr-trusted.yml jobs see as
// GITHUB_WORKFLOW) is named PR, the package vitest lanes cover every
// partition exactly once, exactly one lane runs the native-runner group, and
// those lanes' package script routes through the wrapper.
test("the PR workflow wiring for the native-runner lane holds", () => {
const prWorkflow = readFileSync(path.join(repoRoot, ".github/workflows/pr.yml"), "utf8");
assert.match(prWorkflow, /^name: PR$/m,
"renaming pr.yml silently moves the native-runner suite back into the uncached server shards");
const matrix = JSON.parse(readFileSync(path.join(repoRoot, ".github/ci-task-matrix.json"), "utf8"));
const vitestLanes = matrix.verify_paperclip_runner.filter((entry) => entry.command.startsWith("test:typescript:vitest"));
for (const entry of vitestLanes) {
assert.match(entry.command, /^test:typescript:vitest (--package-lane=\d+\/\d+|--native-server)$/, entry.lane_label);
assert.equal(entry.task, "ci:runner-vitest", entry.lane_label);
}
const lanes = vitestLanes
.map((entry) => /--package-lane=(\d+)\/(\d+)$/.exec(entry.command))
.filter(Boolean).map((match) => [Number(match[1]), Number(match[2])]);
assert.ok(lanes.length > 0, "expected package-lane test:typescript:vitest lanes in the task matrix");
assert.equal(new Set(lanes.map(([, count]) => count)).size, 1, "package lanes must agree on the lane count");
const laneCount = lanes[0][1];
assert.deepEqual(
lanes.map(([index]) => index).sort((left, right) => left - right),
Array.from({ length: laneCount }, (_, index) => index + 1),
"package lanes must cover every partition exactly once",
);
const native = vitestLanes.filter((entry) => entry.command.endsWith("--native-server"));
assert.equal(native.length, 1, "exactly one vitest lane carries the native-runner group");
assert.equal(native[0].release_crate, true, "the native-runner lane needs the release crate");
const runnerPackage = JSON.parse(
readFileSync(path.join(repoRoot, "packages/paperclip-runner/package.json"), "utf8"),
);
assert.equal(runnerPackage.scripts["test:typescript:vitest"], "node ./scripts/run-pr-vitest-lane.mjs");
});
const laneWrapper = path.join(repoRoot, "packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs");
function wrapperPlan(args, envOverrides = {}) {
const result = spawnSync(process.execPath, [laneWrapper, ...args, "--dry-run"], {
cwd: repoRoot,
encoding: "utf8",
env: workflowEnv(envOverrides),
});
assert.equal(result.status, 0, `expected wrapper dry run to succeed for ${args.join(" ")}: ${result.stderr}`);
return JSON.parse(result.stdout);
}
test("the PR lanes separate complete package and native workloads without changing local shards", () => {
for (const [args, envOverrides, packageArgs, native] of [
[["--shard=1/2"], { GITHUB_WORKFLOW: "PR" }, [], false],
[["--shard=2/2"], { GITHUB_WORKFLOW: "PR" }, null, true],
[[], { GITHUB_WORKFLOW: "PR" }, [], true],
[["--shard=2/3"], { GITHUB_WORKFLOW: "PR" }, ["--shard=2/3"], false],
[["--shard=3/3"], { GITHUB_WORKFLOW: "PR" }, ["--shard=3/3"], true],
[["--shard=2/2"], { GITHUB_WORKFLOW: "Release" }, ["--shard=2/2"], false],
[["--shard=2/2"], {}, ["--shard=2/2"], false],
[[], {}, [], false],
]) {
const plan = wrapperPlan(args, envOverrides);
assert.equal(plan.runNativeRunnerGroup, native);
assert.equal(plan.runPackageTests, packageArgs !== null);
const vitest = plan.plannedCommands.find(command => command.args[0] === "exec");
assert.deepEqual(vitest?.args ?? null, packageArgs === null ? null : ["exec", "vitest", "run", ...packageArgs]);
assert.equal(plan.plannedCommands.some(command => command.args.includes("general-server-native-runner")), native);
assert.ok(plan.plannedCommands.some(command => command.args.includes("build:rust")), "both workloads require debug fixtures");
}
const malformed = spawnSync(process.execPath, [laneWrapper, "--shard=nonsense", "--dry-run"], {
cwd: repoRoot,
encoding: "utf8",
env: workflowEnv({ GITHUB_WORKFLOW: "PR" }),
});
assert.notEqual(malformed.status, 0, "a malformed shard flag must fail rather than guess a lane");
});
const runnerRoot = path.join(repoRoot, "packages/paperclip-runner");
const prepCommands = [
{ command: "pnpm", args: ["run", "ensure:eval-build-deps"], cwd: runnerRoot },
{ command: "pnpm", args: ["run", "build:rust"], cwd: runnerRoot },
];
const nativeCommand = {
command: "pnpm",
args: ["test:run:general", "--", "--group", "general-server-native-runner"],
cwd: repoRoot,
};
// Master's trusted workflow keeps sending the --shard commands until the
// package-lane matrix merges, so their exact plans must not move.
test("the master-pinned --shard lane commands plan exactly what they did before package lanes", () => {
assert.deepEqual(wrapperPlan(["--shard=1/2"], { GITHUB_WORKFLOW: "PR" }).plannedCommands, [
...prepCommands,
{ command: "pnpm", args: ["exec", "vitest", "run"], cwd: runnerRoot },
]);
assert.deepEqual(wrapperPlan(["--shard=2/2"], { GITHUB_WORKFLOW: "PR" }).plannedCommands, [...prepCommands, nativeCommand]);
for (const plan of [wrapperPlan(["--shard=1/2"], { GITHUB_WORKFLOW: "PR" }), wrapperPlan(["--shard=2/2"], { GITHUB_WORKFLOW: "PR" })]) {
assert.equal(plan.packageLane, undefined);
}
});
test("--native-server runs only the native group under any workflow", () => {
for (const envOverrides of [{ GITHUB_WORKFLOW: "PR" }, {}]) {
const plan = wrapperPlan(["--native-server"], envOverrides);
assert.deepEqual(plan.plannedCommands, [...prepCommands, nativeCommand]);
assert.equal(plan.runPackageTests, false);
assert.equal(plan.runNativeRunnerGroup, true);
}
});
test("the package lane manifest uses the general server duration schema", () => {
const manifest = JSON.parse(readFileSync(path.join(runnerRoot, "scripts/vitest-lane-durations.json"), "utf8"));
const general = JSON.parse(readFileSync(durationsManifest, "utf8"));
// The refresh script writes exactly these keys; hand-added annotations such
// as the server manifest's splitEstimates are optional.
assert.deepEqual(Object.keys(manifest), ["$comment", "unit", "sources", "coverage", "durations"]);
assert.ok(Object.keys(manifest).every((key) => key in general));
assert.equal(manifest.unit, "ms");
assert.deepEqual(Object.keys(manifest.coverage), Object.keys(general.coverage));
// The weekly refresh samples merged-PR runs for this lane, so its sources
// may lead with a pull number. test-selection-pr-timings.test.mjs passes the
// collector's real output through the same check.
for (const source of manifest.sources) assertManifestSource(source);
const entries = Object.entries(manifest.durations);
assert.ok(entries.length > 100);
for (const [file, ms] of entries) {
assert.match(file, /^packages\/paperclip-runner\/.+\.test\.[cm]?[jt]sx?$/);
assert.ok(Number.isFinite(ms) && ms >= 0, file);
}
});
test("package lane arguments fail rather than guess a lane", () => {
for (const args of [["--package-lane=0/3"], ["--package-lane=4/3"], ["--package-lane=abc"], ["--package-lane", "1/3"],
["--package-lane=1/3", "--package-lane=2/3"], ["--package-lane=1/3", "--shard=1/2"], ["--native-server", "--shard=2/2"],
["--native-server", "--package-lane=1/3"], ["--native-server", "src/foo.test.ts"]]) {
const result = spawnSync(process.execPath, [laneWrapper, ...args, "--dry-run"], {
cwd: repoRoot, encoding: "utf8", env: workflowEnv({ GITHUB_WORKFLOW: "PR" }),
});
assert.notEqual(result.status, 0, args.join(" "));
}
});
const lineShardFile = path.join(repoRoot, "server/src/__tests__/chat-channels.integration.test.ts");
const caseAt = (line, name) => ({ name, file: lineShardFile, projectName: "@paperclipai/server", location: { line, column: 3 } });
test("test-line shards cover nested and parameterized cases exactly once without splitting a source line", () => {
const cases = [caseAt(10, "suite > nested > first"), caseAt(10, "suite > nested > second"),
caseAt(20, "same name"), caseAt(30, "same name"), caseAt(40, "last"), caseAt(50, "new case")];
const shards = partitionTestLines(cases, 3, lineShardFile);
assert.deepEqual(shards.map((shard) => shard.tests.length), [2, 2, 2]);
assert.equal(shards.filter((shard) => shard.lines.includes(10)).length, 1);
assert.equal(shards.find((shard) => shard.lines.includes(10)).tests.length, 2);
assert.equal(shards.flatMap((shard) => shard.lines).length, 5);
assert.deepEqual(shards.flatMap((shard) => shard.tests).sort((a, b) => a.location.line - b.location.line), cases);
assert.deepEqual(partitionTestLines([...cases].reverse(), 3, lineShardFile).map((shard) => shard.lines), shards.map((shard) => shard.lines));
});
test("recorded durations balance chat line shards by time without splitting a source line", () => {
const cases = [caseAt(10, "slow"), caseAt(20, "each %s"), caseAt(20, "each %s"), caseAt(30, "a"), caseAt(40, "b"),
caseAt(50, "c"), caseAt(60, "unrecorded")];
const durations = { slow: 9000, "each %s": 1000, a: 3000, b: 3000, c: 2000 };
const shards = partitionTestLines(cases, 2, lineShardFile, durations);
assert.deepEqual(shards.map((shard) => shard.weight), [11000, 11000]);
assert.deepEqual(shards.map((shard) => shard.lines), [[10, 20], [30, 40, 50, 60]]);
assert.deepEqual(shards.flatMap((shard) => shard.tests).sort((a, b) => a.location.line - b.location.line), cases);
assert.deepEqual(partitionTestLines([...cases].reverse(), 2, lineShardFile, durations).map((shard) => shard.lines),
shards.map((shard) => shard.lines));
assert.deepEqual(partitionTestLines(cases, 2, lineShardFile).map((shard) => shard.tests.length), [4, 3]);
});
test("chat durations attribute every executed case to the collected entry that registers it", () => {
const entries = ["sends %s", "suite > sends %s twice", "plain", "keyed $provider ($mode)"];
const { totals, unmatched } = entryDurations(entries, [
{ name: "chat channel control-plane integration > sends a", duration: 5 },
{ name: "chat channel control-plane integration > suite > sends b twice", duration: 7 },
{ name: "chat channel control-plane integration > plain", duration: 11 },
{ name: "chat channel control-plane integration > keyed slack (fast)", duration: 13 },
{ name: "chat channel control-plane integration > keyed github (slow)", duration: 17 },
{ name: "chat channel control-plane integration > unknown", duration: 19 },
]);
assert.deepEqual(Object.fromEntries(totals), { "sends %s": 5, "suite > sends %s twice": 7, plain: 11, "keyed $provider ($mode)": 30 });
assert.deepEqual(unmatched, ["chat channel control-plane integration > unknown"]);
const data = chatManifestData(["plain", "other"], [[{ name: "x > plain", duration: 4 }], [{ name: "x > plain", duration: 9 }],
[{ name: "x > plain", duration: 6 }]], []);
assert.deepEqual(data.durations, { plain: 6 });
assert.deepEqual(data.coverage, { measured: 1, entries: 2 });
assert.throws(() => chatManifestData(["plain"], [[{ name: "x > missing", duration: 1 }]], []));
});
test("the checked-in chat duration manifest is well formed", () => {
const manifest = JSON.parse(readFileSync(path.join(repoRoot, "scripts", "chat-shard-durations.json"), "utf8"));
assert.equal(manifest.unit, "ms");
const entries = Object.entries(manifest.durations);
assert.ok(entries.length > 0);
assert.ok(entries.every(([name, ms]) => name.length > 0 && Number.isSafeInteger(ms) && ms > 0));
assert.deepEqual(loadShardDurations(path.join(repoRoot, "scripts", "chat-shard-durations.json")), manifest.durations);
});
test("line-shard collection rejects empty, foreign, or unlocated tests and invalid shard counts", () => {
const good = caseAt(10, "valid");
for (const input of [[], null, [{ ...good, file: "/another.test.ts" }], [{ ...good, projectName: "wrong" }],
[{ ...good, location: undefined }], [{ ...good, location: { line: 0 } }], [{ ...good, name: "" }]]) {
assert.throws(() => partitionTestLines(input, 1, lineShardFile));
}
for (const count of [0, -1, 1.5, Infinity, 2]) assert.throws(() => partitionTestLines([good], count, lineShardFile));
});
test("filtered collection must match the exact assigned case identities, including duplicates", () => {
const expected = [caseAt(10, "same"), caseAt(10, "same"), caseAt(20, "nested > case")];
assertSelectedTests(expected, [...expected].reverse(), lineShardFile);
for (const actual of [expected.slice(1), [...expected, caseAt(30, "extra")],
[expected[0], expected[1], caseAt(20, "renamed")],
[expected[0], expected[1], caseAt(21, "nested > case")]]) {
assert.throws(() => assertSelectedTests(expected, actual, lineShardFile));
}
});
test("parallel chat jobs form a complete disjoint partition", () => {
for (const count of [2, 3, 4]) {
const shards = Array.from({ length: count }, (_, index) =>
dryRunJson(["--mode", "general", "--group", "general-chat", "--shard-index", String(index), "--shard-count", String(count)],
{ PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards,
);
assert.ok(shards.every((workers) => workers.length === 3));
assert.deepEqual(shards.flat().map(({ index }) => index).sort((a, b) => a - b),
Array.from({ length: count * 3 }, (_, index) => index));
assert.ok(shards.flat().every((shard) => shard.count === count * 3));
}
});
test("shared-database and unsharded chat runs retain one serial partition", () => {
for (let index = 0; index < 3; index++) {
const result = dryRunJson(["--mode", "general", "--group", "general-chat", "--shard-index", String(index), "--shard-count", "3"],
{ PAPERCLIP_TEST_DATABASE_URL: "postgresql://shared/test" });
assert.deepEqual(result.chatVitestShards, [{ index, count: 3 }]);
}
assert.deepEqual(dryRunJson(["--mode", "general", "--group", "general-chat"],
{ PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards, [{ index: 0, count: 1 }]);
});
test("default chat isolation covers six partitions without dropping native suites", () => {
for (const args of [[], ["--mode", "general"]]) {
for (const workflow of ["PR", "Release", ""]) {
const env = { GITHUB_WORKFLOW: workflow, PAPERCLIP_TEST_DATABASE_URL: "" };
const plan = dryRunJson(args, env);
const full = dryRunJson(["--mode", "general", "--group", "general-server",
"--shard-index", "0", "--shard-count", "1"], env);
assert.deepEqual(plan.chatVitestShards, Array.from({ length: 6 }, (_, index) => ({ index, count: 6 })));
assert.ok(!plan.selectedGeneralServerSuites.includes(chatSuitePath));
assert.equal(new Set(plan.selectedGeneralServerSuites).size, plan.selectedGeneralServerSuites.length);
assert.deepEqual([...plan.selectedGeneralServerSuites, chatSuitePath].sort(), full.selectedGeneralServerSuites.sort());
}
}
});
test("default chat isolation leaves explicit groups and shared databases unchanged", () => {
for (const args of [[], ["--mode", "general"]]) {
const shared = dryRunJson(args, { PAPERCLIP_TEST_DATABASE_URL: "postgresql://shared/test", GITHUB_WORKFLOW: "PR" });
assert.equal(shared.chatVitestShards, null);
assert.equal(shared.selectedGeneralServerSuites, null);
}
const full = dryRunJson(["--mode", "general", "--group", "general-server",
"--shard-index", "0", "--shard-count", "1"], { PAPERCLIP_TEST_DATABASE_URL: "", GITHUB_WORKFLOW: "PR" });
assert.equal(full.chatVitestShards, null);
assert.ok(full.selectedGeneralServerSuites.includes(chatSuitePath));
assert.ok(full.selectedGeneralServerSuites.includes(nativeRunnerSuitePath));
assert.deepEqual(dryRunJson(["--mode", "general", "--group", "general-chat"],
{ PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards, [{ index: 0, count: 1 }]);
assert.equal(dryRunJson(["--mode", "serialized"], { PAPERCLIP_TEST_DATABASE_URL: "" }).chatVitestShards, null);
});
function defaultLauncher({ shared = false, failureIndex = null } = {}) {
const directory = mkdtempSync(path.join(os.tmpdir(), "default-chat-launch-"));
const file = path.join(repoRoot, chatSuitePath);
const lines = readFileSync(file, "utf8").split("\n")
.flatMap((line, index) => /^\s*it\(/.test(line) ? [index + 1] : []).slice(0, 6);
assert.equal(lines.length, 6);
const cases = lines.map(line => ({ file, projectName: "@paperclipai/server",
name: `default launcher fixture ${line}`, location: { line } }));
const receipt = path.join(directory, "events.ndjson");
const active = path.join(directory, "active");
const fakePnpm = path.join(directory, "pnpm");
try {
writeFileSync(fakePnpm, `#!${process.execPath}
const fs = require('node:fs');
const args = process.argv.slice(2);
const tests = ${JSON.stringify(cases)};
const selected = args.flatMap(arg => {
const match = arg.match(/chat-channels\\.integration\\.test\\.ts:(\\d+)$/);
return match ? [Number(match[1])] : [];
});
if (args.includes('list')) {
const output = args.find(arg => arg.startsWith('--json=')).slice(7);
fs.writeFileSync(output, JSON.stringify(selected.length
? tests.filter(test => selected.includes(test.location.line)) : tests));
} else {
const entry = { args, selected, home: process.env.PAPERCLIP_HOME,
config: process.env.PAPERCLIP_CONFIG, instance: process.env.PAPERCLIP_INSTANCE_ID,
tmpdir: process.env.TMPDIR };
const record = edge => fs.appendFileSync(${JSON.stringify(receipt)}, JSON.stringify({ ...entry, edge }) + '\\n');
record('begin');
if (fs.existsSync(${JSON.stringify(active)})) process.exit(91);
fs.writeFileSync(${JSON.stringify(active)}, 'running');
setTimeout(() => {
record('end');
fs.unlinkSync(${JSON.stringify(active)});
process.exit(selected.includes(${JSON.stringify(failureIndex === null ? null : lines[failureIndex])}) ? 7 : 0);
}, selected.length ? 200 : 0);
}
`);
chmodSync(fakePnpm, 0o755);
const result = spawnSync(process.execPath, [script, "--mode", "general"], {
cwd: repoRoot, encoding: "utf8", timeout: 30_000,
env: workflowEnv({ GITHUB_WORKFLOW: "PR", PATH: `${directory}${path.delimiter}${process.env.PATH}`,
PAPERCLIP_TEST_DATABASE_URL: shared ? "postgresql://shared/test" : "", PAPERCLIP_CI_PROFILE: "false" }),
});
const events = existsSync(receipt) ? readFileSync(receipt, "utf8").trim().split("\n").map(line => JSON.parse(line)) : [];
return { result, events, lines };
} finally {
rmSync(directory, { recursive: true, force: true });
}
}
test("default chat isolation executes every partition sequentially before other groups", { skip: process.platform === "win32" }, () => {
const { result, events, lines } = defaultLauncher();
assert.equal(result.status, 0, result.stdout + result.stderr);
const chat = events.filter(event => event.selected.length > 0);
assert.deepEqual(chat.map(event => event.edge), Array.from({ length: 6 }, () => ["begin", "end"]).flat());
const starts = chat.filter(event => event.edge === "begin");
assert.deepEqual(starts.flatMap(event => event.selected).sort((a, b) => a - b), lines);
for (const event of starts) {
assert.ok(event.args.includes("--no-file-parallelism"));
assert.deepEqual(event.args.filter(arg => arg.startsWith("--maxWorkers")), ["--maxWorkers=1"]);
}
for (const field of ["home", "config", "instance", "tmpdir"]) {
assert.equal(new Set(starts.map(event => event[field])).size, 6, `${field} must be isolated per child`);
}
assert.equal(events.findIndex(event => event.selected.length === 0), 12);
const server = events[12];
assert.ok(server.args.includes("@paperclipai/server"));
const excluded = server.args.flatMap((arg, index) => arg === "--exclude" ? [server.args[index + 1]] : []);
assert.equal(excluded.filter(file => file === chatSuitePath.replace(/^server\//, "")).length, 1);
for (const file of [nativeRunnerSuitePath, ingressRetrySuitePath, usageDeliverySuitePath, dotRunnerSuitePath, restartRecoverySuitePath,
"server/src/services/native-runtime/plugin-runner-probe.test.ts"]) {
assert.ok(!excluded.includes(file.replace(/^server\//, "")), `default must retain ${file}`);
}
});
test("default chat isolation propagates failure without starting another partition or group", { skip: process.platform === "win32" }, () => {
const { result, events } = defaultLauncher({ failureIndex: 2 });
assert.equal(result.status, 1, result.stdout + result.stderr);
assert.match(result.stderr, /failed \(7\)/);
assert.equal(events.length, 6);
assert.ok(events.every(event => event.selected.length === 1));
assert.deepEqual(events.map(event => event.edge), ["begin", "end", "begin", "end", "begin", "end"]);
});
test("default chat isolation keeps shared-database execution monolithic and serial", { skip: process.platform === "win32" }, () => {
const { result, events } = defaultLauncher({ shared: true });
assert.equal(result.status, 0, result.stdout + result.stderr);
assert.ok(events.every(event => event.selected.length === 0));
assert.ok(events[0].args.includes("@paperclipai/server"));
assert.ok(events[0].args.includes("--no-file-parallelism"));
assert.deepEqual(events[0].args.filter(arg => arg.startsWith("--maxWorkers")), ["--maxWorkers=1"]);
const excluded = events[0].args.flatMap((arg, index) => arg === "--exclude" ? [events[0].args[index + 1]] : []);
assert.ok(!excluded.includes(chatSuitePath.replace(/^server\//, "")));
assert.ok(!excluded.includes(nativeRunnerSuitePath.replace(/^server\//, "")));
assert.deepEqual(events.map(event => event.edge), Array.from({ length: events.length / 2 }, () => ["begin", "end"]).flat());
});
test("a failed chat child fails the job after its sibling finishes", { skip: process.platform === "win32" }, () => {
const directory = mkdtempSync(path.join(os.tmpdir(), "chat-launch-proof-"));
const file = path.join(repoRoot, "server/src/__tests__/chat-channels.integration.test.ts");
const lines = readFileSync(file, "utf8").split("\n")
.flatMap((line, index) => /^\s*it\(/.test(line) ? [index + 1] : []).slice(0, 9);
assert.equal(lines.length, 9);
const tests = lines.map((line) => ({ file, projectName: "@paperclipai/server",
name: `launcher fixture ${line}`, location: { line } }));
const receipt = path.join(directory, "sibling-finished");
const fakePnpm = path.join(directory, "pnpm");
try {
writeFileSync(fakePnpm, `#!${process.execPath}
const fs = require('node:fs');
const args = process.argv.slice(2);
const tests = ${JSON.stringify(tests)};
const selected = args.flatMap(arg => {
const match = arg.match(/chat-channels\\.integration\\.test\\.ts:(\\d+)$/);
return match ? [Number(match[1])] : [];
});
if (args.includes('list')) {
const output = args.find(arg => arg.startsWith('--json=')).slice(7);
fs.writeFileSync(output, JSON.stringify(selected.length
? tests.filter(test => selected.includes(test.location.line)) : tests));
} else if (selected.includes(${lines[0]})) process.exit(7);
else setTimeout(() => fs.writeFileSync(${JSON.stringify(receipt)}, 'finished'), 200);
`);
chmodSync(fakePnpm, 0o755);
const result = spawnSync(process.execPath, [script, "--mode", "general", "--group", "general-chat",
"--shard-index", "0", "--shard-count", "3"], {
cwd: repoRoot, encoding: "utf8", timeout: 30_000,
env: { ...process.env, PATH: `${directory}${path.delimiter}${process.env.PATH}`,
PAPERCLIP_TEST_DATABASE_URL: "", PAPERCLIP_CI_PROFILE: "false" },
});
assert.equal(result.status, 1, result.stdout + result.stderr);
assert.match(result.stderr, /failed \(7\)/);
assert.equal(readFileSync(receipt, "utf8"), "finished", "a failed worker must not abandon its sibling");
} finally {
rmSync(directory, { recursive: true, force: true });
}
});
for (const [group, parallelWorkers, shardArgs] of [
["general-server-native-runner", 4, []],
["general-server-without-chat-or-native-runner", 3, ["--shard-index", "0", "--shard-count", "12"]],
]) test(`${group} files overlap only when their database fixtures are isolated`, { skip: process.platform === "win32" }, () => {
const directory = mkdtempSync(path.join(os.tmpdir(), "native-file-workers-"));
const receipt = path.join(directory, "args.json");
const fakePnpm = path.join(directory, "pnpm");
writeFileSync(fakePnpm, `#!/usr/bin/env node\nrequire('node:fs').writeFileSync(${JSON.stringify(receipt)}, JSON.stringify(process.argv.slice(2)));\n`);
chmodSync(fakePnpm, 0o755);
try {
for (const [databaseEnv, workers] of [
[{}, parallelWorkers],
[{ PAPERCLIP_TEST_DATABASE_URL: "postgresql://shared/test" }, 1],
[{ PAPERCLIP_ISSUE_RECOVERY_TEST_DATABASE_URL: "postgresql://shared/recovery_test" }, 1],
[{ PAPERCLIP_TEST_POSTGRES_ADMIN_URL: "postgresql://owned/postgres" }, parallelWorkers],
]) {
const env = workflowEnv({ PATH: `${directory}${path.delimiter}${process.env.PATH}`, PAPERCLIP_CI_PROFILE: "false" });
for (const key of Object.keys(env)) if (key.endsWith("_TEST_DATABASE_URL")) delete env[key];
Object.assign(env, databaseEnv);
const result = spawnSync(process.execPath, [script, "--mode", "general", "--group", group, ...shardArgs], {
cwd: repoRoot, encoding: "utf8", env,
});
assert.equal(result.status, 0, result.stdout + result.stderr);
const args = JSON.parse(readFileSync(receipt, "utf8"));
assert.ok(args.includes(`--maxWorkers=${workers}`));
assert.ok(args.includes(workers > 1 ? "--fileParallelism" : "--no-file-parallelism"));
if (workers > 1) assert.ok(args.includes("--pool=forks") && args.includes("--isolate"));
if (group === "general-server-native-runner") {
for (const file of [restartRecoverySuitePath, ...restartWaitSuites]) assert.ok(args.includes(file));
}
}
} finally { rmSync(directory, { recursive: true, force: true }); }
});