A Cross-Validated Benchmark of Classical, Deep Learning, and Transformer Approaches for Arabic Hotel Review Sentiment Classification

Sentiment analysis of Arabic hotel reviews remains challenging due to the morphological richness and dialectal diversity of the Arabic language, a complexity compounded by methodological inconsistencies in prior literature. Specifically, previous studies frequently exhibit discrepancies in rating scale descriptions, rely on unvalidated single-fold train/test splits, and report point estimates devoid of variance metrics. To address these gaps, this study introduces a fully reproducible benchmark evaluating four classical machine learning algorithms (Logistic Regression, Support Vector Machine, Multinomial Naive Bayes, and Random Forest), two deep learning architectures (Long Short-Term Memory networks and Convolutional Neural Networks), and the optimized Arabic Transformer model, AraBERTv02, utilizing the official Hotel Arabic Reviews Dataset (HARD). Model evaluation employed stratified k-fold cross-validation, utilizing five folds for the classical and deep learning models and three folds for the computationally intensive Transformer—a methodological deviation that is explicitly justified. Furthermore, out-of-fold predictions were leveraged to conduct McNemar’s tests, enabling comprehensive pairwise statistical comparisons across all models irrespective of fold count. Evaluated on a balanced, binary-classified subset of 49,973 reviews, AraBERTv02 achieved the highest mean F1 score of 95.84% ± 0.21. This performance significantly surpasses the strongest classical baseline, Logistic Regression (McNemar’s χ² = 506.99, p < 0.001).