Evaluating Reliability in High-Dimensional Clustering: A Dimension-Aware and Stability-Based Analysis of Projection-Induced Misinterpretation in Offshore Wind Data

Abstract

Analyses of variability of offshore wind resources using multi-dimensional atmospheric data are performed using clustering methods. Evaluating the quality of the clustering results becomes a challenging task when dealing with large number of features. The reasons behind this are that visual interpretation of the resulting low-dimensional representations of clusters is not clear and stable, because of the poor separation between clusters and the instability in the composition of the clusters. This work studies the effects of increasing the number of features in the data on the clustering performance, stability and interpretability in the context of offshore wind data, by applying a dimension-aware and stability-oriented methodology.

 A multi-dimensional offshore wind dataset from a controlled experimental testbed was derived from reanalysis-based atmospheric variables. The datasets were divided into four tiles that represented the diversity of offshore wind resources and were subjected to high-level clustering analyses. The progression of the clusters in decreasing orders of feature dimensions (from low to high) and their implications were investigated. In addition to widely used validity indices, other stability indices were introduced to analyse the stability of the clusters, the stability of cluster assignments, and the sensitivity of the clusters to new variables.

The research investigates two aspects of cluster result analysis from high-dimensional data through studies of fixed quality evaluation methods and the reliability of low-dimensional projection techniques. The research shows that two- and three-dimensional visualizations produce incorrect cluster overlap and weak separation perceptions although the original feature space shows large distances between cluster centers. The methodological evaluation of high-dimensional clustering faces a major failure mode because of this phenomenon which scientists call projection-induced misinterpretation.

Cluster analysis is not necessarily subject to the curse of dimensionality, as many clustering problems are a result of data visualization and dimensionality reduction limitations. This study addresses dimension-aware scenario design, stability-driven validation, use of the true high-dimensional structure of the data as a reference and the need for explicit projection auditing. The experimental results demonstrate that any reliable cluster analysis solution for high-dimensional data space has to at least include dimension-aware scenario design, stability-driven validation and explicit projection auditing. The purpose of this paper is to establish some rules of thumb for scientists for experimental design and presentation of results to facilitate cluster analysis investigations of offshore wind resources in high-dimensional data space.