SIGN: Statistical Inference Graphs Based on Probabilistic Network Activity Interpretation

Yael Konforti, Alon Shpigler, Boaz Lerner, Aharon Bar-Hillel

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


Convolutional neural networks (CNNs) have achieved superior accuracy in many visual-related tasks. However, the inference process through a CNN's intermediate layers is opaque, making it difficult to interpret such networks or develop trust in their operation. In this article, we introduce SIGN method for modeling the network's hidden layer activity using probabilistic models. The activity patterns in layers of interest are modeled as Gaussian mixture models, and transition probabilities between clusters in consecutive modeled layers are estimated to identify paths of inference. For fully connected networks, the entire layer activity is clustered, and the resulting model is a hidden Markov model. For convolutional layers, spatial columns of activity are clustered, and a maximum likelihood model is developed for mining an explanatory inference graph. The graph describes the hierarchy of activity clusters most relevant for network prediction. We show that such inference graphs are useful for understanding the general inference process of a class, as well as explaining the (correct or incorrect) decisions the network makes about specific images. In addition, SIGN provide interesting observations regarding hidden layer activity in general, including the concentration of memorization in a single middle layer in fully connected networks, and a highly local nature of column activities in the top CNN layers.

Original languageEnglish
Pages (from-to)3783-3797
Number of pages15
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Issue number3
StatePublished - 1 Mar 2023


  • Deep neural networks
  • XAI
  • convolutional neural networks
  • explainable AI
  • interpretable AI
  • statistical inference
  • visualization

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence
  • Applied Mathematics
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics


Dive into the research topics of 'SIGN: Statistical Inference Graphs Based on Probabilistic Network Activity Interpretation'. Together they form a unique fingerprint.

Cite this