Optimizing Collaborative Filtering in Very Sparse Datasets: a Confidence-based Approach

Collaborative filtering systems are a key tool for personalized recommendations, predicting user preferences based on historical data. A major challenge in collaborative filtering is the reliability of predictions, particularly in highly sparse datasets, where the limited overlap of common ratings between users impedes the identification of reliable neighbors, reducing both the accuracy and coverage of the predictions. So far, the literature has proposed confidence factors to quantify prediction reliability, typically using the Pearson and Cosine similarities with threshold-based neighbor selection. However, a recent study demonstrated that using the Sigmoid Cosine similarity metric in combination with the KNN neighborhood selection method and adapting the value of K to each level of dataset sparsity, significantly improves prediction performance. In this paper, we revisit the calculation of confidence factors under these improved settings, assessing their validity and effectiveness across datasets with varying sparsity levels. Continuous assessment of collaborative filtering methods and adjustment of their parameters according to dataset characteristics is particularly crucial in highly sparse datasets, where the reliability of neighbors strongly influences prediction quality.