Loss minimization and parameter estimation with heavy tails

Daniel Hsu, Sivan Sabato

Research output: Contribution to journalArticlepeer-review

63 Scopus citations

Abstract

This work studies applications and generalizations of a simple estimation technique that provides exponential concentration under heavy-tailed distributions, assuming only bounded low-order moments. We show that the technique can be used for approximate minimization of smooth and strongly convex losses, and specifically for least squares linear regression. For instance, our d-dimensional estimator requires just Õ(d log(1/δ)) random samples to obtain a constant factor approximation to the optimal least squares loss with probability 1 - δ, without requiring the covariates or noise to be bounded or subgaussian. We provide further applications to sparse linear regression and low-rank covariance matrix estimation with similar allowances on the noise and covariate distributions. The core technique is a generalization of the median-of-means estimator to arbitrary metric spaces.

Original languageEnglish
JournalJournal of Machine Learning Research
Volume17
StatePublished - 1 Apr 2016

Keywords

  • Heavy-tailed distributions
  • Least squares
  • Linear regression
  • Unbounded losses

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Statistics and Probability
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'Loss minimization and parameter estimation with heavy tails'. Together they form a unique fingerprint.

Cite this