Universal Rates for Regression: Separations between Cut-Off and Absolute Loss

In this work we initiate the study of regression in the universal rates framework of (Bousquet et al., 2021). Unlike the traditional uniform learning setting, we are interested in obtaining learning guarantees that hold for all fixed data-generating distributions, but do not hold uniformly across them. We focus on the realizable setting and we consider two different well-studied loss functions: the cut-off loss at scale γ > 0, which asks for predictions that are γ-close to the correct one, and the absolute loss, which measures how far away the prediction is from the correct one. Our results show that the landscape of the achievable rates in the two cases is completely different. First we give a trichotomic characterization of the optimal learning rates under the cut-off loss: each class is learnable either at an exponential rate, a (nearly) linear rate or requires arbitrarily slow rates. Moving to the absolute loss, we show that the achievable learning rates are significantly more involved by illustrating that an infinite number of different learning rates is achievable. This is the first time that such a rich landscape of rates is obtained in the universal rates literature.