Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-13 07:12:00 +01:00
Code Issues Packages Projects Releases Wiki Activity

perf(ci): rebalance server test shards and repair the weekly weight refresh (#21016)

Browse Source 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Core image is published only after the release verification of the
same commit passes (`cloud-readiness.yml` calls `release-verify.yml`)
> - The image is ready about 3 minutes before that gate opens, so the
slowest verification job sets the staging release time
> - The slowest job is a general-server test shard: 11.8-12.7 min, while
other server shards take 8-9 min
> - The shards are balanced by a duration manifest, but the manifest has
not changed since 2026-10-31. The weekly refresh that updates it failed
on 2026-11-02 and 2026-11-09
> - About 40 newer suites per shard use the 1 s median fallback weight,
but several of them run for 90-200 s, so they pile up on one shard
> - This pull request repairs the weekly refresh and refreshes the
server weights from three current master verifications
> - The benefit is a shorter slowest server shard (predicted 10.3 -> 8.8
min of test time) on every master verification and every PR run

## Linked Issues or Issue Description

**What happened?**
The Core image publication waits about 13.5 min for source verification.
The critical path is one general-server shard. In three master
verifications (runs 43133035381, 43134251278, 43145252393) the slowest
server shard took 12.1, 11.0 and 12.7 min. The fastest took 8.0, 8.9 and
8.4 min.

**Root cause**
`scripts/general-server-shard-durations.json` came from one run on
2026-10-31. The weekly `ci-test-durations.yml` refresh failed twice:
- 2026-11-02 (run 41408033686): `gh api .../jobs/<id>/logs` failed. A
server shard log is 30-65 MB. One download took 59 s locally. Three
parallel downloads overran the 120 s `execFile` timeout.
- 2026-11-09 (run 42580964686): the refresh's own check failed.
Merged-PR sources for the Runner lane have a leading `pull` key, and the
test accepted only `run, attempt, sha, url`.

**Expected behavior**
The weekly refresh finishes and keeps the shards balanced as suites are
added.

## What Changed

- `scripts/ci/gh-api.mjs`: job-log downloads get a 15 min timeout and a
512 MB buffer. JSON calls keep 120 s. A failed call now gives a short
error and does not print the partial 60 MB log.
- `scripts/ci/pr-run-durations.mjs`: new shared `assertManifestSource`.
It accepts the optional leading `pull` key that `collectMergedPrSamples`
writes. The checked-in Runner manifest test and the collector test both
use it, so the collector format stays covered.
- `.github/workflows/ci-test-durations.yml`: refresh job timeout 20 ->
45 min (the server lanes alone fetch about 30 large logs).
- `scripts/general-server-shard-durations.json` and
`scripts/serialized-shard-durations.json`: refreshed with `node
scripts/ci/refresh-test-durations.mjs --lanes=server 43133035381
43134251278 43145252393` (median of three successful protected-master
Cloud readiness runs; 1757/1815 general files and 173/173 serialized
files measured).
- No test is added, removed, skipped or reassigned to a different lane.
No shard count changes. `docker.yml`, the Dockerfile and the PID-1 check
are not touched.

## Verification

Measured job timings (minutes from the first job start; three master
verifications):

| Job | 43133035381 | 43134251278 | 43145252393 |
| --- | --- | --- | --- |
| Slowest server shard (duration) | 7/10: 12.1 | 6/10: 11.0 | 8/10: 12.7
|
| Fastest server shard | 2/10: 8.0 | 2/10: 8.9 | 5/10: 8.4 |
| Runner chaos evals (restart/replay) | 10.2 | 11.9 | 11.0 |
| Verify Paperclip Runner (protocol) | 8.9 | 8.9 | 9.6 |
| Chat (slowest of 3) | 8.8 | 10.5 | 9.4 |
| Serialized (slowest of 5) | 9.3 | 9.5 | 9.2 |
| Build / Typecheck | 4.5 / 2.4 | 4.5 / 3.1 | 4.8 / 2.5 |
| Gate "Cloud source verified" ends | +13.5 | +13.2 | +13.9 |

- Queue to runner start is 0.5-1.1 min for every job. In a server shard,
setup (checkout, pnpm, Node, install) is about 50 s and the preflight
plus plugin-sdk build is about 8 s. The test step is the rest (9-11.8
min). Setup is not worth caching.
- Shard prediction, using the refreshed medians as the cost of each file
and the 3 fork workers per shard:
- general-server, 10 shards: old weights max 10.3 / min 8.1 min; new
weights 8.8 / 8.8 min.
- serialized, 5 shards: old weights max 8.2 / min 4.6 min; new weights
6.6 / 6.6 min.
- The old-weight prediction matches what CI saw: predicted 10.3 and 9.9
min for shards 7/10 and 6/10, measured test steps 11.25 and 10.1 min in
run 43133035381.
- Red/green:
- `node --test --test-name-pattern="gh api"
scripts/ci/test-selection-pr-timings.test.mjs` failed before the fix
(`log timeout 120000`) and passes after it.
- With a `pull` key in the Runner manifest sources, the old "package
lane manifest" test failed with the same assertion as run 42580964686.
The new test passes.
- The refresh script, which died before the fix, completed locally in 8
min after it.
- `node --test scripts/ci/test-selection-timings.test.mjs
scripts/ci/test-selection-pr-timings.test.mjs
scripts/__tests__/run-vitest-stable-shard.test.mjs
scripts/__tests__/e2e-shard.test.mjs`: 76/76 pass. This includes the
complete, non-overlapping shard cover checks.
- `actionlint .github/workflows/ci-test-durations.yml`: clean.
- Not run: the release-verify workflow itself, the scheduled refresh
workflow, and the Runner/e2e lane refresh. The shard times above are
predictions from logged per-file timings, not a measured CI run on this
branch.

## Risks

- Low risk. Weights only move files between shards of the same lane. The
shard tests prove every file still runs exactly once.
- The predicted gain is about 1.5 min on the slowest server shard
(12-12.7 -> about 10.5 min job time). The gate then waits for
"Pre-release Runner chaos evals / Restart, replay, trace, and recovery
faults" (10.2-11.9 min). So the expected gate saving is about 1 min
(13.5 -> about 12.5 min), not the full 1.5 min. Run-to-run noise on a
shard is about ±1 min.
- A larger log timeout can make a stuck download wait up to 15 min. The
45 min job timeout still bounds the run.

Candidates for the owner (not changed here):
- The chaos-eval job is now the bound. Its "server finalization" step
takes 7-9 min. Decide if it must gate image publication, or split it.
- Server shard logs are up to 811k lines, mostly repeated stack traces
from `server/src/services/execution-preparation-timing.ts`. That slows
log upload and this refresh.
- 9 test files run over 180 s (largest:
`chat-channels.integration.test.ts` 429 s,
`heartbeat-process-recovery.test.ts` 344 s). One file cannot be split
across shards.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, GitHub CLI read-only). Standard reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Related: #19650 (earlier repair of the weekly refresh), #21013 (Core
image publish time; separate files).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-12 05:29:27 -06:00
1 parent f8e1fb0278
commit bed5f70278
7 files changed
+1972 -1485
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.github/workflows/ci-test-durations.yml
+3 -1
View File
@@ -18,7 +18,9 @@ jobs:
refresh:
if: github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master'
runs-on: ubuntu-latest
timeout-minutes: 20
# Server shard logs are 30-65 MB each and the master lanes alone fetch
# about 30 of them, three at a time; 20 minutes left no headroom.
timeout-minutes: 45
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
with:
scripts/__tests__/run-vitest-stable-shard.test.mjs
+5 -4
View File
@@ -12,6 +12,7 @@ import {
partitionGeneralServerSuites,
} from "../general-server-shard.mjs";
import { assertManifestSource } from "../ci/pr-run-durations.mjs";
import { assertSelectedTests, partitionTestLines } from "../test-line-shard.mjs";
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..");
@@ -567,10 +568,10 @@ test("the package lane manifest uses the general server duration schema", () =>
assert.ok(Object.keys(manifest).every((key) => key in general));
assert.equal(manifest.unit, "ms");
assert.deepEqual(Object.keys(manifest.coverage), Object.keys(general.coverage));
for (const source of manifest.sources) {
assert.deepEqual(Object.keys(source), ["run", "attempt", "sha", "url"]);
assert.match(source.sha, /^[a-f0-9]{40}$/);
}
// The weekly refresh samples merged-PR runs for this lane, so its sources
// may lead with a pull number. test-selection-pr-timings.test.mjs passes the
// collector's real output through the same check.
for (const source of manifest.sources) assertManifestSource(source);
const entries = Object.entries(manifest.durations);
assert.ok(entries.length > 100);
for (const [file, ms] of entries) {
scripts/ci/gh-api.mjs
+22 -2
View File
@@ -13,10 +13,30 @@ export function isUnknownEscapeFlagError(error) {
return /unknown flag:?\s+--allow-escape-sequences/.test(String(error?.stderr ?? error?.message ?? ""));
}
// A server test shard's job log is 30-65 MB (actions run 43133035381) and one
// download took 59 s on its own. Three in parallel overran the old 120 s limit,
// so the weekly refresh died mid-download and the shard weights stopped moving.
const jsonFetchOptions = { maxBuffer: 16 * 1024 * 1024, timeout: 120_000 };
const logFetchOptions = { maxBuffer: 512 * 1024 * 1024, timeout: 900_000 };
// A failed log download carries the partial log on stdout; never print it.
function conciseError(endpoint, error) {
const reason = error?.killed ? `killed by ${error.signal ?? "timeout"}` : `exit ${error?.code ?? "unknown"}`;
const stderr = String(error?.stderr ?? "").trim().split("\n").slice(-3).join(" | ");
return Object.assign(new Error(`gh api ${endpoint} failed (${reason})${stderr ? `: ${stderr}` : ""}`),
{ stderr: error?.stderr, code: error?.code, killed: error?.killed, signal: error?.signal });
}
// Calls `gh api` with argv only; no shell sees the endpoint. `run` exists for tests.
export async function ghApi(endpoint, { json = true, run = execute } = {}) {
const options = { encoding: "utf8", maxBuffer: json ? 16 * 1024 * 1024 : 128 * 1024 * 1024, timeout: 120_000 };
const call = flags => run("gh", ["api", ...flags, endpoint], options);
const options = { encoding: "utf8", ...(json ? jsonFetchOptions : logFetchOptions) };
const call = async flags => {
try {
return await run("gh", ["api", ...flags, endpoint], options);
} catch (error) {
throw conciseError(endpoint, error);
}
};
let result;
if (escapeFlagSupported) {
try {
scripts/ci/pr-run-durations.mjs
+14
View File
@@ -15,6 +15,20 @@ export const runnerPackageRoot = "packages/paperclip-runner";
export const e2eManifest = "scripts/e2e-shard-durations.json";
export const runnerManifest = `${runnerPackageRoot}/scripts/vitest-lane-durations.json`;
// The source record shape every timing manifest may carry. Protected-master
// sources have run, attempt, sha and url; merged-PR sources (written by
// collectMergedPrSamples below) add a leading pull number. Rejecting that key
// failed the 2026-11-09 weekly refresh and froze the shard weights.
export function assertManifestSource(source) {
const keys = Object.keys(source);
assert.deepEqual(keys[0] === "pull" ? keys.slice(1) : keys, ["run", "attempt", "sha", "url"]);
if (keys[0] === "pull") assert.ok(Number.isSafeInteger(source.pull) && source.pull > 0);
assert.ok(Number.isSafeInteger(source.run) && source.run > 0);
assert.ok(Number.isSafeInteger(source.attempt) && source.attempt > 0);
assert.match(source.sha, /^[a-f0-9]{40}$/);
assert.equal(typeof source.url, "string");
}
export function validatePull(pull) {
assert.equal(pull.base?.ref, "master");
assert.equal(pull.base?.repo?.id, repositoryId);
scripts/ci/test-selection-pr-timings.test.mjs
+37 -1
View File
@@ -4,7 +4,7 @@ import test from "node:test";
import { ghApi, resetGhApiForTests } from "./gh-api.mjs";
import OversizedSpecReporter from "./oversized-test-files-playwright.mjs";
import OversizedTestFileReporter, { oversizedAnnotation, oversizedThresholdMs } from "./oversized-test-files.mjs";
import { attributeSpec, collectMergedPrSamples, completeShardSet, e2eShardJob, parseE2eLog, parseRunnerLog,
import { assertManifestSource, attributeSpec, collectMergedPrSamples, completeShardSet, e2eShardJob, parseE2eLog, parseRunnerLog,
runnerInventory, runnerPackageJob, validatePrRun, validatePull } from "./pr-run-durations.mjs";
import { oversizedFiles, oversizedPlaceholder, parseArgs, parseTimings, renderPrBody, serverLane, updatedDurations } from "./refresh-test-durations.mjs";
import { timingPrefix } from "./test-timing-reporter.mjs";
@@ -44,6 +44,36 @@ test("gh api opts into escape sequences, falls back for older gh, and strips col
resetGhApiForTests();
});
test("gh api allows a large server shard log to finish downloading and never prints a partial log", async () => {
resetGhApiForTests();
const seen = [];
const capture = async (command, args, options) => {
seen.push(options);
return { stdout: "{}" };
};
await ghApi("/repos/x/actions/runs/1/jobs", { run: capture });
await ghApi("/repos/x/actions/jobs/1/logs", { json: false, run: capture });
// A 64 MB shard log took 59 s alone (actions run 43133035381); three at once
// overran the old 120 s limit and failed the weekly refresh.
assert.equal(seen[0].timeout, 120_000);
assert.ok(seen[1].timeout >= 600_000, `log timeout ${seen[1].timeout}`);
assert.ok(seen[1].maxBuffer >= 256 * 1024 * 1024, `log buffer ${seen[1].maxBuffer}`);
resetGhApiForTests();
const partial = "x".repeat(10_000);
const timedOut = async () => {
throw Object.assign(new Error(`Command failed: gh api /logs\n${partial}`),
{ killed: true, signal: "SIGTERM", stdout: partial, stderr: "" });
};
await assert.rejects(ghApi("/repos/x/actions/jobs/2/logs", { json: false, run: timedOut }), error => {
assert.match(error.message, /\/repos\/x\/actions\/jobs\/2\/logs failed \(killed by SIGTERM\)/);
assert.ok(!error.message.includes(partial));
assert.equal(error.stdout, undefined);
return true;
});
resetGhApiForTests();
});
test("server timing parsing tolerates colour codes in fetched logs", () => {
const file = "server/src/slow.test.ts";
const log = `${esc}[2m${stamp(0)}${esc}[22m ${timingPrefix}${JSON.stringify({ version: 1, file, durationMs: 360 })}`;
@@ -193,6 +223,12 @@ test("collection skips unmerged, absent, stale-head and incomplete runs, and sam
const result = await collectMergedPrSamples({ gh, lanes, isMerged: merge => merge !== sha("c"), log: line => warnings.push(line) });
assert.deepEqual(result.e2e.sources.map(source => source.run), [21]);
assert.deepEqual(result.runner.sources, [{ pull: 2, run: 21, attempt: 1, sha: sha("2"), url: "https://example.test/21" }]);
// The checked-in manifest test uses this same check, so the format the
// collector writes stays accepted even while no manifest carries it yet.
for (const source of [...result.e2e.sources, ...result.runner.sources]) assertManifestSource(source);
assertManifestSource({ run: 21, attempt: 1, sha: sha("2"), url: "https://example.test/21" });
for (const bad of [{ run: 21, attempt: 1, sha: sha("2") }, { pull: 0, run: 21, attempt: 1, sha: sha("2"), url: "u" },
{ run: 21, pull: 2, attempt: 1, sha: sha("2"), url: "u" }]) assert.throws(() => assertManifestSource(bad));
assert.deepEqual(Object.fromEntries(result.e2e.samples[0]), { "tests/e2e/alpha.spec.ts": { ms: 15_000, complete: true } });
assert.match(warnings.join("\n"), /e2e: skipped run 50 for PR #5: Parsed 4 of 9/);
// PR 3 is not in this checkout and PR 4 never merged: neither is queried.
scripts/general-server-shard-durations.json
+1703 -1310
View File
File diff suppressed because it is too large. Load diff
scripts/serialized-shard-durations.json
+188 -167
View File
@@ -3,181 +3,202 @@
"unit": "ms",
"sources": [
{
"run": 41300933157,
"run": 43133035381,
"attempt": 1,
"sha": "247552fe0719dcc92f7b42c50b658be6f624986d",
"url": "https://github.com/paperclipai/paperclip/actions/runs/41300933157"
"sha": "42fd790e80953000a7be99eff30fb9a81bf9c7b7",
"url": "https://github.com/paperclipai/paperclip/actions/runs/43133035381"
},
{
"run": 43134251278,
"attempt": 2,
"sha": "9dfdeaad8662d9cd73a93490d09516877ef8a276",
"url": "https://github.com/paperclipai/paperclip/actions/runs/43134251278"
},
{
"run": 43145252393,
"attempt": 2,
"sha": "7d9973405c8726d6aadc7d0e6b4f6076085ebc45",
"url": "https://github.com/paperclipai/paperclip/actions/runs/43145252393"
}
],
"coverage": {
"measured": 164,
"withSetup": 164,
"inventory": 164
"measured": 173,
"withSetup": 173,
"inventory": 173
},
"durations": {
"server/src/__tests__/access-routes-hidden-floor.test.ts": 4135,
"server/src/__tests__/access-routes-permissions-upgrade.test.ts": 5005,
"server/src/__tests__/activity-routes.test.ts": 15673,
"server/src/__tests__/adapter-auth-signal-routes.test.ts": 12341,
"server/src/__tests__/adapter-model-refresh-routes.test.ts": 6883,
"server/src/__tests__/adapter-routes-authz.test.ts": 13707,
"server/src/__tests__/adapter-routes.test.ts": 3323,
"server/src/__tests__/agent-action-audit-routes.test.ts": 6056,
"server/src/__tests__/agent-adapter-validation-routes.test.ts": 84643,
"server/src/__tests__/agent-cross-tenant-authz-routes.test.ts": 4315,
"server/src/__tests__/agent-device-login-routes.test.ts": 6491,
"server/src/__tests__/agent-hire-auth-inheritance-routes.test.ts": 7530,
"server/src/__tests__/agent-hire-idempotency-routes.test.ts": 5637,
"server/src/__tests__/agent-instructions-routes.test.ts": 4157,
"server/src/__tests__/agent-live-run-routes.test.ts": 66708,
"server/src/__tests__/agent-permissions-routes.test.ts": 4758,
"server/src/__tests__/agent-runner-qualification-routes.test.ts": 5232,
"server/src/__tests__/agent-secrets-routes.test.ts": 5078,
"server/src/__tests__/agent-skills-routes.test.ts": 5414,
"server/src/__tests__/agent-test-environment-routes.test.ts": 4719,
"server/src/__tests__/ai-connection-router.test.ts": 34133,
"server/src/__tests__/announcements-routes.test.ts": 4301,
"server/src/__tests__/approval-routes-idempotency.test.ts": 3592,
"server/src/__tests__/artifact-review-document-routes.test.ts": 7252,
"server/src/__tests__/assets.test.ts": 9197,
"server/src/__tests__/auth-routes.test.ts": 1477,
"server/src/__tests__/auth-session-route.test.ts": 3303,
"server/src/__tests__/authz-company-access.test.ts": 1265,
"server/src/__tests__/access-routes-hidden-floor.test.ts": 4459,
"server/src/__tests__/access-routes-permissions-upgrade.test.ts": 5465,
"server/src/__tests__/activity-routes.test.ts": 10563,
"server/src/__tests__/adapter-auth-signal-routes.test.ts": 9084,
"server/src/__tests__/adapter-model-refresh-routes.test.ts": 4075,
"server/src/__tests__/adapter-routes-authz.test.ts": 10176,
"server/src/__tests__/adapter-routes.test.ts": 3958,
"server/src/__tests__/agent-action-audit-routes.test.ts": 5904,
"server/src/__tests__/agent-adapter-validation-routes.test.ts": 50458,
"server/src/__tests__/agent-cross-tenant-authz-routes.test.ts": 4272,
"server/src/__tests__/agent-device-login-routes.test.ts": 6231,
"server/src/__tests__/agent-hire-auth-inheritance-routes.test.ts": 8399,
"server/src/__tests__/agent-hire-idempotency-routes.test.ts": 6320,
"server/src/__tests__/agent-instructions-routes.test.ts": 3851,
"server/src/__tests__/agent-live-run-routes.test.ts": 47196,
"server/src/__tests__/agent-permissions-routes.test.ts": 5042,
"server/src/__tests__/agent-runner-qualification-routes.test.ts": 5808,
"server/src/__tests__/agent-secrets-routes.test.ts": 5616,
"server/src/__tests__/agent-skills-routes.test.ts": 4780,
"server/src/__tests__/agent-task-hierarchy-depth-routes.test.ts": 5914,
"server/src/__tests__/agent-test-environment-routes.test.ts": 4386,
"server/src/__tests__/ai-connection-router.test.ts": 28008,
"server/src/__tests__/announcements-routes.test.ts": 4480,
"server/src/__tests__/approval-routes-idempotency.test.ts": 4100,
"server/src/__tests__/artifact-review-document-routes.test.ts": 6376,
"server/src/__tests__/assets.test.ts": 5477,
"server/src/__tests__/auth-routes.test.ts": 4182,
"server/src/__tests__/auth-session-route.test.ts": 3352,
"server/src/__tests__/authz-company-access.test.ts": 1190,
"server/src/__tests__/authz-existence-oracle-guard.test.ts": 100,
"server/src/__tests__/authz-secret-context.test.ts": 1282,
"server/src/__tests__/board-chat-route-feature-flag.test.ts": 881,
"server/src/__tests__/bootstrap-claim-routes.test.ts": 3293,
"server/src/__tests__/browser-use-route-scope.test.ts": 3223,
"server/src/__tests__/built-in-agent-routes.test.ts": 6590,
"server/src/__tests__/cases-routes.test.ts": 26922,
"server/src/__tests__/cli-auth-routes.test.ts": 17761,
"server/src/__tests__/cloud-routes.test.ts": 1476,
"server/src/__tests__/companies-route-cross-company-authz.test.ts": 11710,
"server/src/__tests__/companies-route-path-guard.test.ts": 1523,
"server/src/__tests__/company-branding-route.test.ts": 4996,
"server/src/__tests__/company-import-transfer-routes.test.ts": 2992,
"server/src/__tests__/company-portability-routes.test.ts": 3065,
"server/src/__tests__/company-portability.test.ts": 9374,
"server/src/__tests__/company-search-extract-routes.test.ts": 4311,
"server/src/__tests__/company-search-rate-limit-routes.test.ts": 4322,
"server/src/__tests__/company-skill-policy-routes.test.ts": 3653,
"server/src/__tests__/company-skills-import-authz-routes.test.ts": 5508,
"server/src/__tests__/company-skills-routes.test.ts": 4067,
"server/src/__tests__/company-user-directory-route.test.ts": 3338,
"server/src/__tests__/cost-data-quality-routes.test.ts": 3245,
"server/src/__tests__/costs-service.test.ts": 4246,
"server/src/__tests__/decision-queues-routes.test.ts": 5259,
"server/src/__tests__/document-annotation-routes.test.ts": 8439,
"server/src/__tests__/email-routes-provider-errors.test.ts": 3201,
"server/src/__tests__/environment-custom-image-routes.test.ts": 4007,
"server/src/__tests__/environment-instance-routes.test.ts": 4041,
"server/src/__tests__/environment-routes.test.ts": 3697,
"server/src/__tests__/environment-selection-route-guards.test.ts": 4164,
"server/src/__tests__/execution-workspace-runtime-lease-route.test.ts": 4063,
"server/src/__tests__/execution-workspaces-routes.test.ts": 3474,
"server/src/__tests__/express5-auth-wildcard.test.ts": 135,
"server/src/__tests__/external-object-routes.test.ts": 3862,
"server/src/__tests__/folders-routes.test.ts": 1474,
"server/src/__tests__/health-dev-server-token.test.ts": 3152,
"server/src/__tests__/health.test.ts": 3543,
"server/src/__tests__/heartbeat-dependency-scheduling.test.ts": 31376,
"server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts": 49291,
"server/src/__tests__/heartbeat-process-recovery.test.ts": 362170,
"server/src/__tests__/heartbeat-run-privacy-routes.test.ts": 9796,
"server/src/__tests__/inbox-agent-policy-routes.test.ts": 5019,
"server/src/__tests__/inbox-archive-routes.test.ts": 7129,
"server/src/__tests__/instance-database-backups-routes.test.ts": 1474,
"server/src/__tests__/instance-settings-routes.test.ts": 3499,
"server/src/__tests__/invite-accept-existing-member.test.ts": 3537,
"server/src/__tests__/invite-accept-gateway-defaults.test.ts": 4795,
"server/src/__tests__/invite-accept-replay.test.ts": 4065,
"server/src/__tests__/invite-create-route.test.ts": 7645,
"server/src/__tests__/invite-expiry.test.ts": 4250,
"server/src/__tests__/invite-join-manager.test.ts": 4317,
"server/src/__tests__/invite-list-route.test.ts": 3634,
"server/src/__tests__/invite-logo-route.test.ts": 4177,
"server/src/__tests__/invite-onboarding-text.test.ts": 4032,
"server/src/__tests__/invite-rate-limit-route.test.ts": 6008,
"server/src/__tests__/invite-summary-route.test.ts": 6090,
"server/src/__tests__/invite-test-resolution-route.test.ts": 4352,
"server/src/__tests__/invite-url-public-base-url.test.ts": 4379,
"server/src/__tests__/issue-access-grants-routes.test.ts": 6138,
"server/src/__tests__/issue-activity-events-routes.test.ts": 10999,
"server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts": 14786,
"server/src/__tests__/issue-assigned-backlog-contract-routes.test.ts": 3922,
"server/src/__tests__/issue-assignee-invokability-routes.test.ts": 3837,
"server/src/__tests__/issue-attachment-routes.test.ts": 4351,
"server/src/__tests__/issue-blocker-diagnostics-routes.test.ts": 5648,
"server/src/__tests__/issue-closed-workspace-routes.test.ts": 4191,
"server/src/__tests__/issue-comment-attribution-audit-routes.test.ts": 5562,
"server/src/__tests__/issue-comment-cancel-routes.test.ts": 4176,
"server/src/__tests__/issue-comment-reopen-routes.test.ts": 4355,
"server/src/__tests__/issue-create-deduplication-routes.test.ts": 5953,
"server/src/__tests__/issue-created-from-routes.test.ts": 4860,
"server/src/__tests__/issue-creation-source-routes.test.ts": 6117,
"server/src/__tests__/issue-dependency-wakeups-routes.test.ts": 8961,
"server/src/__tests__/issue-document-restore-routes.test.ts": 7554,
"server/src/__tests__/issue-execution-policy-routes.test.ts": 4791,
"server/src/__tests__/issue-feedback-routes.test.ts": 3966,
"server/src/__tests__/issue-identifier-routes.test.ts": 4818,
"server/src/__tests__/issue-list-assignee-filter-routes.test.ts": 10945,
"server/src/__tests__/issue-list-updatedsince-filter-routes.test.ts": 5143,
"server/src/__tests__/issue-onboarding-first-task-routes.test.ts": 10593,
"server/src/__tests__/issue-queued-comments-routes.test.ts": 53688,
"server/src/__tests__/issue-scheduled-retry-routes.test.ts": 12618,
"server/src/__tests__/issue-stale-execution-lock-routes.test.ts": 6168,
"server/src/__tests__/issue-stalled-review-decision-routes.test.ts": 6305,
"server/src/__tests__/issue-status-version-route-service.test.ts": 5153,
"server/src/__tests__/issue-subtree-diagnostics-routes.test.ts": 5592,
"server/src/__tests__/issue-summaries-routes.test.ts": 5080,
"server/src/__tests__/issue-telemetry-routes.test.ts": 4631,
"server/src/__tests__/issue-thread-interaction-routes.test.ts": 15508,
"server/src/__tests__/issue-tree-control-routes.test.ts": 1520,
"server/src/__tests__/issue-update-comment-wakeup-routes.test.ts": 4994,
"server/src/__tests__/issue-wake-diagnostics-routes.test.ts": 5428,
"server/src/__tests__/issue-watchdogs-routes.test.ts": 10901,
"server/src/__tests__/issue-workspace-command-authz.test.ts": 4648,
"server/src/__tests__/authz-secret-context.test.ts": 1184,
"server/src/__tests__/board-bootstrap-routes.test.ts": 5136,
"server/src/__tests__/board-chat-route-feature-flag.test.ts": 815,
"server/src/__tests__/bootstrap-claim-routes.test.ts": 3801,
"server/src/__tests__/browser-use-route-scope.test.ts": 3617,
"server/src/__tests__/built-in-agent-routes.test.ts": 9291,
"server/src/__tests__/cases-routes.test.ts": 11134,
"server/src/__tests__/cli-auth-routes.test.ts": 17169,
"server/src/__tests__/cloud-routes.test.ts": 3736,
"server/src/__tests__/companies-route-cross-company-authz.test.ts": 15423,
"server/src/__tests__/companies-route-path-guard.test.ts": 3778,
"server/src/__tests__/company-branding-route.test.ts": 6936,
"server/src/__tests__/company-default-decision-owner-route.test.ts": 6504,
"server/src/__tests__/company-import-transfer-routes.test.ts": 4904,
"server/src/__tests__/company-portability-routes.test.ts": 5250,
"server/src/__tests__/company-portability.test.ts": 9588,
"server/src/__tests__/company-search-extract-routes.test.ts": 4424,
"server/src/__tests__/company-search-rate-limit-routes.test.ts": 4412,
"server/src/__tests__/company-skill-policy-routes.test.ts": 4527,
"server/src/__tests__/company-skills-import-authz-routes.test.ts": 5740,
"server/src/__tests__/company-skills-routes.test.ts": 3796,
"server/src/__tests__/company-user-directory-route.test.ts": 3829,
"server/src/__tests__/cost-data-quality-routes.test.ts": 3649,
"server/src/__tests__/costs-service.test.ts": 5218,
"server/src/__tests__/decision-queues-routes.test.ts": 6413,
"server/src/__tests__/document-annotation-routes.test.ts": 6425,
"server/src/__tests__/email-routes-provider-errors.test.ts": 3667,
"server/src/__tests__/environment-custom-image-routes.test.ts": 3810,
"server/src/__tests__/environment-instance-routes.test.ts": 3784,
"server/src/__tests__/environment-routes.test.ts": 3862,
"server/src/__tests__/environment-selection-route-guards.test.ts": 4439,
"server/src/__tests__/execution-workspace-runtime-lease-route.test.ts": 5683,
"server/src/__tests__/execution-workspaces-routes.test.ts": 3883,
"server/src/__tests__/express5-auth-wildcard.test.ts": 120,
"server/src/__tests__/external-object-routes.test.ts": 4334,
"server/src/__tests__/folders-routes.test.ts": 3921,
"server/src/__tests__/health-dev-server-token.test.ts": 4171,
"server/src/__tests__/health.test.ts": 3773,
"server/src/__tests__/heartbeat-dependency-scheduling.test.ts": 50657,
"server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts": 54768,
"server/src/__tests__/heartbeat-process-recovery.test.ts": 344017,
"server/src/__tests__/heartbeat-run-privacy-routes.test.ts": 9270,
"server/src/__tests__/inbox-agent-policy-routes.test.ts": 4707,
"server/src/__tests__/inbox-archive-routes.test.ts": 7445,
"server/src/__tests__/inbox-mine-routes.test.ts": 7381,
"server/src/__tests__/instance-database-backups-routes.test.ts": 3506,
"server/src/__tests__/instance-settings-routes.test.ts": 4187,
"server/src/__tests__/invite-accept-existing-member.test.ts": 3588,
"server/src/__tests__/invite-accept-gateway-defaults.test.ts": 4603,
"server/src/__tests__/invite-accept-replay.test.ts": 4210,
"server/src/__tests__/invite-create-route.test.ts": 6191,
"server/src/__tests__/invite-expiry.test.ts": 4468,
"server/src/__tests__/invite-join-manager.test.ts": 4095,
"server/src/__tests__/invite-list-route.test.ts": 3806,
"server/src/__tests__/invite-logo-route.test.ts": 4470,
"server/src/__tests__/invite-onboarding-text.test.ts": 4598,
"server/src/__tests__/invite-rate-limit-route.test.ts": 5758,
"server/src/__tests__/invite-summary-route.test.ts": 6219,
"server/src/__tests__/invite-test-resolution-route.test.ts": 4021,
"server/src/__tests__/invite-url-public-base-url.test.ts": 4803,
"server/src/__tests__/issue-access-grants-routes.test.ts": 7311,
"server/src/__tests__/issue-activity-events-routes.test.ts": 8154,
"server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts": 15844,
"server/src/__tests__/issue-assigned-backlog-contract-routes.test.ts": 4282,
"server/src/__tests__/issue-assignee-invokability-routes.test.ts": 3973,
"server/src/__tests__/issue-attachment-routes.test.ts": 4512,
"server/src/__tests__/issue-blocker-diagnostics-routes.test.ts": 5372,
"server/src/__tests__/issue-chat-bootstrap-routes.test.ts": 7382,
"server/src/__tests__/issue-closed-workspace-routes.test.ts": 3977,
"server/src/__tests__/issue-comment-attribution-audit-routes.test.ts": 5540,
"server/src/__tests__/issue-comment-cancel-routes.test.ts": 3798,
"server/src/__tests__/issue-comment-reopen-routes.test.ts": 6776,
"server/src/__tests__/issue-confirm-run-stopped-routes.test.ts": 7677,
"server/src/__tests__/issue-create-deduplication-routes.test.ts": 6041,
"server/src/__tests__/issue-created-from-routes.test.ts": 5645,
"server/src/__tests__/issue-creation-source-routes.test.ts": 6585,
"server/src/__tests__/issue-dependency-wakeups-routes.test.ts": 8660,
"server/src/__tests__/issue-document-restore-routes.test.ts": 6269,
"server/src/__tests__/issue-execution-policy-routes.test.ts": 4346,
"server/src/__tests__/issue-feedback-routes.test.ts": 4382,
"server/src/__tests__/issue-identifier-routes.test.ts": 5515,
"server/src/__tests__/issue-list-assignee-filter-routes.test.ts": 12312,
"server/src/__tests__/issue-list-updatedsince-filter-routes.test.ts": 4929,
"server/src/__tests__/issue-onboarding-first-task-routes.test.ts": 13445,
"server/src/__tests__/issue-queued-comments-routes.test.ts": 118620,
"server/src/__tests__/issue-scheduled-retry-routes.test.ts": 14427,
"server/src/__tests__/issue-stale-execution-lock-routes.test.ts": 7268,
"server/src/__tests__/issue-stalled-review-decision-routes.test.ts": 6458,
"server/src/__tests__/issue-status-version-route-service.test.ts": 5591,
"server/src/__tests__/issue-subtree-diagnostics-routes.test.ts": 6142,
"server/src/__tests__/issue-summaries-routes.test.ts": 5147,
"server/src/__tests__/issue-telemetry-routes.test.ts": 4790,
"server/src/__tests__/issue-thread-interaction-routes.test.ts": 14207,
"server/src/__tests__/issue-tree-control-routes.test.ts": 3888,
"server/src/__tests__/issue-update-comment-wakeup-routes.test.ts": 4533,
"server/src/__tests__/issue-wake-diagnostics-routes.test.ts": 5628,
"server/src/__tests__/issue-watchdogs-routes.test.ts": 11962,
"server/src/__tests__/issue-workspace-command-authz.test.ts": 4185,
"server/src/__tests__/issues-checkout-wakeup.test.ts": 100,
"server/src/__tests__/issues-goal-context-routes.test.ts": 4119,
"server/src/__tests__/issues-service.test.ts": 26999,
"server/src/__tests__/llms-routes.test.ts": 1838,
"server/src/__tests__/low-trust-red-team-routes.test.ts": 25961,
"server/src/__tests__/maintenance-server-routes.test.ts": 511763,
"server/src/__tests__/managed-agent-profile-routes-authz.test.ts": 3182,
"server/src/__tests__/multilingual-issues-routes.test.ts": 5137,
"server/src/__tests__/muse-qualification-routes.test.ts": 3072,
"server/src/__tests__/onboarding-seed-route.test.ts": 5499,
"server/src/__tests__/openapi-routes.test.ts": 2400,
"server/src/__tests__/openclaw-invite-prompt-route.test.ts": 3380,
"server/src/__tests__/opencode-local-adapter-environment.test.ts": 6840,
"server/src/__tests__/openrouter-attribution.test.ts": 778,
"server/src/__tests__/permissions-upgrade-boundary-routes.test.ts": 5309,
"server/src/__tests__/plugin-install-route-security.test.ts": 4283,
"server/src/__tests__/plugin-routes-authz.test.ts": 3539,
"server/src/__tests__/plugin-scoped-api-routes.test.ts": 3119,
"server/src/__tests__/project-goal-telemetry-routes.test.ts": 3626,
"server/src/__tests__/project-routes-env.test.ts": 4082,
"server/src/__tests__/project-workspace-managed-sandbox-routes.test.ts": 6431,
"server/src/__tests__/projects-list-archived-routes.test.ts": 5044,
"server/src/__tests__/redaction.test.ts": 136,
"server/src/__tests__/resource-memberships-routes.test.ts": 5339,
"server/src/__tests__/routine-document-annotation-routes.test.ts": 2206,
"server/src/__tests__/routines-e2e.test.ts": 18070,
"server/src/__tests__/routines-routes.test.ts": 6367,
"server/src/__tests__/secret-proposals-routes.test.ts": 16478,
"server/src/__tests__/secrets-routes-claude-oauth-service.test.ts": 4530,
"server/src/__tests__/secrets-routes.test.ts": 4522,
"server/src/__tests__/sidebar-preferences-routes.test.ts": 3352,
"server/src/__tests__/subscription-routes.test.ts": 3167,
"server/src/__tests__/summary-slot-routes.test.ts": 5656,
"server/src/__tests__/task-connector-read-routes.test.ts": 6058,
"server/src/__tests__/task-title-routes.test.ts": 13080,
"server/src/__tests__/teams-catalog-routes.test.ts": 4167,
"server/src/__tests__/user-profile-routes.test.ts": 2245,
"server/src/__tests__/issues-goal-context-routes.test.ts": 4365,
"server/src/__tests__/issues-service.test.ts": 39512,
"server/src/__tests__/link-preview-routes.test.ts": 3783,
"server/src/__tests__/llms-routes.test.ts": 4025,
"server/src/__tests__/low-trust-red-team-routes.test.ts": 40548,
"server/src/__tests__/maintenance-server-routes.test.ts": 276429,
"server/src/__tests__/managed-agent-profile-routes-authz.test.ts": 3791,
"server/src/__tests__/multilingual-issues-routes.test.ts": 5790,
"server/src/__tests__/muse-qualification-routes.test.ts": 3804,
"server/src/__tests__/onboarding-seed-route.test.ts": 6481,
"server/src/__tests__/openapi-routes.test.ts": 5123,
"server/src/__tests__/openclaw-invite-prompt-route.test.ts": 3811,
"server/src/__tests__/opencode-local-adapter-environment.test.ts": 6723,
"server/src/__tests__/openrouter-attribution.test.ts": 685,
"server/src/__tests__/permissions-upgrade-boundary-routes.test.ts": 6152,
"server/src/__tests__/plugin-install-route-security.test.ts": 5289,
"server/src/__tests__/plugin-routes-authz.test.ts": 3612,
"server/src/__tests__/plugin-scoped-api-routes.test.ts": 3890,
"server/src/__tests__/project-decision-owner-route.test.ts": 6221,
"server/src/__tests__/project-goal-telemetry-routes.test.ts": 4390,
"server/src/__tests__/project-routes-env.test.ts": 4228,
"server/src/__tests__/project-workspace-managed-sandbox-routes.test.ts": 6636,
"server/src/__tests__/projects-list-archived-routes.test.ts": 5176,
"server/src/__tests__/redaction.test.ts": 160,
"server/src/__tests__/resource-memberships-routes.test.ts": 5530,
"server/src/__tests__/routine-document-annotation-routes.test.ts": 4747,
"server/src/__tests__/routines-e2e.test.ts": 15930,
"server/src/__tests__/routines-routes.test.ts": 11937,
"server/src/__tests__/secret-proposals-routes.test.ts": 15931,
"server/src/__tests__/secrets-routes-claude-oauth-service.test.ts": 5401,
"server/src/__tests__/secrets-routes.test.ts": 4414,
"server/src/__tests__/sidebar-preferences-routes.test.ts": 6241,
"server/src/__tests__/skill-update-diff-routes.test.ts": 4160,
"server/src/__tests__/subscription-routes.test.ts": 3452,
"server/src/__tests__/summary-slot-routes.test.ts": 8150,
"server/src/__tests__/task-connector-read-routes.test.ts": 6479,
"server/src/__tests__/task-title-routes.test.ts": 13501,
"server/src/__tests__/teams-catalog-routes.test.ts": 7702,
"server/src/__tests__/user-profile-routes.test.ts": 4153,
"server/src/__tests__/workspace-command-authz.test.ts": 100,
"server/src/__tests__/workspace-preferences-routes.test.ts": 1477,
"server/src/__tests__/workspace-restore-repair-route.test.ts": 6027,
"server/src/__tests__/workspace-runtime-routes-authz.test.ts": 3659,
"server/src/__tests__/workspace-runtime-service-authz.test.ts": 2230
"server/src/__tests__/workspace-preferences-routes.test.ts": 3922,
"server/src/__tests__/workspace-restore-repair-route.test.ts": 7023,
"server/src/__tests__/workspace-runtime-routes-authz.test.ts": 4185,
"server/src/__tests__/workspace-runtime-service-authz.test.ts": 2027
}
}