Evolution of activation functions for deep learning-based image classification.

Raz Lapid; Moshe Sipper

doi:10.1145/3520304.3533949

Evolution of activation functions for deep learning-based image classification.

Raz Lapid, Moshe Sipper

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, co-evolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets---MNIST, FashionMNIST, KMNIST, and USPS---coevolution proves to be a performant algorithm for finding good AFs and AF architectures.

Original language	English
Title of host publication	Proceedings of the Genetic and Evolutionary Computation Conference Companion GECCO 2022
Place of Publication	New York, NY, USA
Publisher	Association for Computing Machinery, Inc
Pages	2113-2121
Number of pages	9
ISBN (Electronic)	9781450392686
DOIs	https://doi.org/10.1145/3520304.3533949
State	Published - 19 Jul 2022
Event	2022 Genetic and Evolutionary Computation Conference, GECCO 2022 - Virtual, Online, United States Duration: 9 Jul 2022 → 13 Jul 2022

Conference

Conference	2022 Genetic and Evolutionary Computation Conference, GECCO 2022
Country/Territory	United States
City	Virtual, Online
Period	9/07/22 → 13/07/22

Keywords

Computing methodologies
Computer graphics
Image manipulation
Image processing
Machine learning
Discrete optimization
Mathematical optimization
Design and analysis of algorithms
Theory of computation
Optimization with randomized search heuristics

Access to Document

10.1145/3520304.3533949

https://dblp.org/db/conf/gecco/gecco2022c.html#LapidS22

Cite this

@inproceedings{d46d539aac46489ab976d84c0d719424,

title = "Evolution of activation functions for deep learning-based image classification.",

abstract = "Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, co-evolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets---MNIST, FashionMNIST, KMNIST, and USPS---coevolution proves to be a performant algorithm for finding good AFs and AF architectures.",

keywords = "Computing methodologies, Computer graphics, Image manipulation, Image processing, Machine learning, Discrete optimization, Mathematical optimization, Design and analysis of algorithms, Theory of computation, Optimization with randomized search heuristics",

author = "Raz Lapid and Moshe Sipper",

note = "DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.; 2022 Genetic and Evolutionary Computation Conference, GECCO 2022 ; Conference date: 09-07-2022 Through 13-07-2022",

year = "2022",

month = jul,

day = "19",

doi = "10.1145/3520304.3533949",

language = "English",

pages = "2113--2121",

booktitle = "Proceedings of the Genetic and Evolutionary Computation Conference Companion GECCO 2022",

publisher = "Association for Computing Machinery, Inc",

}

Lapid, R & Sipper, M 2022, Evolution of activation functions for deep learning-based image classification. in Proceedings of the Genetic and Evolutionary Computation Conference Companion GECCO 2022. Association for Computing Machinery, Inc, New York, NY, USA, pp. 2113-2121, 2022 Genetic and Evolutionary Computation Conference, GECCO 2022, Virtual, Online, United States, 9/07/22. https://doi.org/10.1145/3520304.3533949

Evolution of activation functions for deep learning-based image classification. / Lapid, Raz; Sipper, Moshe.
Proceedings of the Genetic and Evolutionary Computation Conference Companion GECCO 2022. New York, NY, USA: Association for Computing Machinery, Inc, 2022. p. 2113-2121.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Evolution of activation functions for deep learning-based image classification.

AU - Lapid, Raz

AU - Sipper, Moshe

N1 - DBLP License: DBLP's bibliographic metadata records provided through http://dblp.org/ are distributed under a Creative Commons CC0 1.0 Universal Public Domain Dedication. Although the bibliographic metadata records are provided consistent with CC0 1.0 Dedication, the content described by the metadata records is not. Content may be subject to copyright, rights of privacy, rights of publicity and other restrictions.

PY - 2022/7/19

Y1 - 2022/7/19

N2 - Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, co-evolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets---MNIST, FashionMNIST, KMNIST, and USPS---coevolution proves to be a performant algorithm for finding good AFs and AF architectures.

AB - Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, co-evolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets---MNIST, FashionMNIST, KMNIST, and USPS---coevolution proves to be a performant algorithm for finding good AFs and AF architectures.

KW - Computing methodologies

KW - Computer graphics

KW - Image manipulation

KW - Image processing

KW - Machine learning

KW - Discrete optimization

KW - Mathematical optimization

KW - Design and analysis of algorithms

KW - Theory of computation

KW - Optimization with randomized search heuristics

UR - http://www.scopus.com/inward/record.url?scp=85134921867&partnerID=8YFLogxK

U2 - 10.1145/3520304.3533949

DO - 10.1145/3520304.3533949

M3 - Conference contribution

SP - 2113

EP - 2121

BT - Proceedings of the Genetic and Evolutionary Computation Conference Companion GECCO 2022

PB - Association for Computing Machinery, Inc

CY - New York, NY, USA

T2 - 2022 Genetic and Evolutionary Computation Conference, GECCO 2022

Y2 - 9 July 2022 through 13 July 2022

ER -

Evolution of activation functions for deep learning-based image classification.

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this