Kernel matrix regularization via shrinkage estimation

The “kernel trick” is a fundamental approach that allows expanding many machine learning algorithms. The kernel matrix obtained from the data requires inner products in the feature space, while the sample covariance matrix of the same data requires outer products. Consequently, both matrices share corresponding eigenvalues up to a constant. The use of kernels often involves a large number of features, compared to the number of observations which reflects a situation of an ill-conditioned or non-invertible sample covariance matrix. To improve the situation mentioned above, we propose to regularize the kernel matrix in a way that reflects a better alternative to the sample covariance matrix, i.e., by shrinking the latter matrix to a well-conditioned matrix with the aim of minimizing the mean-squared error. We demonstrate through numerical simulations that the proposed regularization is useful in classification tasks.