Unveiling Structural Distributional Bias in Collaborative Filtering for Ultra-Sparse Datasets

Collaborative Filtering is one of the most widely used rating prediction methods for recommender systems in recent years. It is based on a solid and verified rationale, it supports explainability and usually delivers satisfactory results. However, its performance declines when applied to ultra-sparse datasets. Despite extensive research, the root cause of its failure in such environments still requires further investigation. This work systematically investigates structural distribution biases in CF ultra-sparse datasets, by examining 11 datasets in this category. The findings of this study are threefold. First, this study reveals that the rating distribution within ultra-sparse datasets, exhibits a strong skew toward boundary ratings, thus creating a non-normal rating structure that challenges the traditional Collaborative Filtering algorithms. Secondly, through extensive experiments, employing four different similarity metrics and using three nearest neighbor selection thresholds, it asserts that ratings with values that are underrepresented in the dataset tend to produce significantly higher error margins — up to four times larger than predictions corresponding to the most frequent rating values. The third and most important finding of this study is the phenomenon of mean rating attraction. Specifically, it reveals that the rating predictions are consistently anchored to user and item averages, a process which essentially computationally smooths the rating predictions, instead of recognizing true and unique non-conforming preferences. The above conclusions were found to be independent of both the selection of user similarity metric and the neighbor selection threshold; rather, they are grounded on the inherent structural bias of the input data. This work offers a new perspective for Collaborative Filtering algorithm design and implementation, by identifying structural bias as an important dimension that needs to be handled by rating prediction algorithms, along with the dimension of similarity metrics between users and items.