ICAN: Information Capacity Approximate Network for Estimating Regression Model Confidence

While machine learning models demonstrate high accuracy, prediction accuracy for specific segments of data may be poor. Given this, model confidence estimates are needed and should be part of the decision-making process. Reliable confidence estimates help building users’ trust in a model’s predictions. Recently proposed confidence estimation methods are based on noise level estimation; such methods provide insights regarding the prediction error but do not consider the signal-to-noise ratio (SNR), which has been proven useful in estimating the quality of signal transfer. We present a generic uncertainty framework for regression models, referred to as ICAN: Information Capacity Approximate Network, which is implemented as an auxiliary wrapper. We evaluated our method on five LSTM neural network (NN) models which were trained to approximate SQL queries which is also known as approximate query processing (AQP), and three fully connected NN models. Our results demonstrate our method’s superiority in estimating the confidence of predictions; on most of the datasets used in this study, our method outperformed the other methods while producing tighter prediction intervals.