High Per Parameter: A Large-Scale Study of Hyperparameter Tuning for Machine Learning Algorithms

Research output: Contribution to journalArticlepeer-review

8 Scopus citations

Abstract

Hyperparameters in machine learning (ML) have received a fair amount of attention, and hyperparameter tuning has come to be regarded as an important step in the ML pipeline. However, just how useful is said tuning? While smaller-scale experiments have been previously conducted, herein we carry out a large-scale investigation, specifically one involving 26 ML algorithms, 250 datasets (regression and both binary and multinomial classification), 6 score metrics, and 28,857,600 algorithm runs. Analyzing the results we conclude that for many ML algorithms, we should not expect considerable gains from hyperparameter tuning on average; however, there may be some datasets for which default hyperparameters perform poorly, especially for some algorithms. By defining a single hp_score value, which combines an algorithm’s accumulated statistics, we are able to rank the 26 ML algorithms from those expected to gain the most from hyperparameter tuning to those expected to gain the least. We believe such a study shall serve ML practitioners at large.

Original languageEnglish
Article number315
JournalAlgorithms
Volume15
Issue number9
DOIs
StatePublished - 2 Sep 2022

Keywords

  • hyperparemeters
  • machine learning

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Numerical Analysis
  • Computational Theory and Mathematics
  • Computational Mathematics

Fingerprint

Dive into the research topics of 'High Per Parameter: A Large-Scale Study of Hyperparameter Tuning for Machine Learning Algorithms'. Together they form a unique fingerprint.

Cite this