Evolution of activation functions for deep learning-based image classification.

Raz Lapid, Moshe Sipper

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, co-evolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets---MNIST, FashionMNIST, KMNIST, and USPS---coevolution proves to be a performant algorithm for finding good AFs and AF architectures.
Original languageEnglish
Title of host publicationProceedings of the Genetic and Evolutionary Computation Conference Companion GECCO 2022
PublisherAssociation for Computing Machinery, Inc
Pages2113-2121
Number of pages9
ISBN (Electronic)9781450392686
DOIs
StatePublished - 2022
Event2022 Genetic and Evolutionary Computation Conference, GECCO 2022 - Virtual, Online, United States
Duration: 9 Jul 202213 Jul 2022

Publication series

NameGECCO 2022 Companion - Proceedings of the 2022 Genetic and Evolutionary Computation Conference

Conference

Conference2022 Genetic and Evolutionary Computation Conference, GECCO 2022
Country/TerritoryUnited States
CityVirtual, Online
Period9/07/2213/07/22

Keywords

  • Computing methodologies
  • Computer graphics
  • Image manipulation
  • Image processing
  • Machine learning
  • Discrete optimization
  • Mathematical optimization
  • Design and analysis of algorithms
  • Theory of computation
  • Optimization with randomized search heuristics

Fingerprint

Dive into the research topics of 'Evolution of activation functions for deep learning-based image classification.'. Together they form a unique fingerprint.

Cite this