An Entropy Maximization Approach to Optimal Dimensionality Reduction

Aviv Dotan, Oren Shriki

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

The maximum entropy principle is a well established approach to unsupervised optimization. Entropy maximization learning algorithms for single-layered neural networks already exist for the cases in which the number of output neurons is greater or equal to the number of input neurons. These models were successfully employed in various applications, most notably for independent component analysis. In this work, we generalize the maximum entropy principle to a single-layered neural network with fewer output than input neurons. The proposed learning algorithm finds a low-dimensional representation of the data and identifies the independent components within it. In general, such a model must incorporate some prior knowledge of the input distribution; however, we overcome this difficulty using a variational approach. We illustrate the performance of the model through several examples and compare it to other algorithms. While our model achieves similar results to the state-of-the-art algorithm for overdetermined independent component analysis within a similar convergence time, its main advantage lies in its ability to be learned efficiently on-line.

Original languageEnglish
Title of host publication2018 International Joint Conference on Neural Networks, IJCNN 2018 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781509060146
DOIs
StatePublished - 10 Oct 2018
Event2018 International Joint Conference on Neural Networks, IJCNN 2018 - Rio de Janeiro, Brazil
Duration: 8 Jul 201813 Jul 2018

Publication series

NameProceedings of the International Joint Conference on Neural Networks
Volume2018-July

Conference

Conference2018 International Joint Conference on Neural Networks, IJCNN 2018
Country/TerritoryBrazil
CityRio de Janeiro
Period8/07/1813/07/18

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'An Entropy Maximization Approach to Optimal Dimensionality Reduction'. Together they form a unique fingerprint.

Cite this