James Scott

Machine learning competitions are often a good indicator of what techniques actually work well in practice on new datasets. The very comprehensive State of Competitive Machine Learning 2022 report just came out and contained many interesting and surprising insights! 1) As expected, transformers dominate natural language processing (NLP). ALL NLP-related winning solutions used transformers. 2) Convolutional neural networks still dominate computer vision. And EfficientNet is the most popular pretrained architecture for computer vision -- most people finetune pretrained models rather than training from scratch. 3) Almost twice as many winning solutions used k-fold CV instead of a fixed validation set. 4) Kaggle (barely) remains the most popular competition platform. 5) Almost everyone uses Python. 6) Out of 46 winning solutions using deep learning, 44 used PyTorch, and only 2 used TensorFlow. 7) A big surprise for tabular competitions: the reign of XGBoost seems over. While gradient boosting still wins most tabular competitions, LightGBM is now the preferred approach, with CatBoost coming in second. XGBoost is third. 8) Winning solutions of 7 out of the 10 tabular competitions used gradient boosting, 5 out of 10 used deep neural networks (implemented in PyTorch), and most winning solutions were ensemble methods. Here's a link to the full report: mlcontests.com/state-of-comp…