Neural Networks with À La Carte Selection of Activation Functions.

Research output: Contribution to journalArticlepeer-review


Activation functions (AFs), which are pivotal to the success (or failure) of a neural network, have received increased attention in recent years, with researchers seeking to design novel AFs that improve some aspect of network performance. In this paper we take another direction, wherein we combine a slew of known AFs into successful architectures, proposing three methods to do so beneficially: (1) generate AF architectures at random, (2) use Optuna, an automatic hyper-parameter optimization software framework, with a Tree-structured Parzen Estimator (TPE) sampler, and (3) use Optuna with a Covariance Matrix Adaptation Evolution Strategy (CMA-ES) sampler. We show that all methods often produce significantly better results for 25 classification problems when compared with a standard network composed of ReLU hidden units and a softmax output unit. Optuna with the TPE sampler emerged as the best AF architecture-producing method.
Original languageEnglish
Article number470
Pages (from-to)1-9
JournalSN Comput. Sci.
StatePublished - 2021


Dive into the research topics of 'Neural Networks with À La Carte Selection of Activation Functions.'. Together they form a unique fingerprint.

Cite this