Minimum KL-divergence on complements of L1 balls

Daniel Berend, Peter Harremoës, Aryeh Kontorovich

Research output: Contribution to journalArticlepeer-review

8 Scopus citations

Abstract

Pinsker's widely used inequality upper-bounds the total variation distance ||P - Q||1 in terms of the Kullback-Leibler divergence D(P||Q). Although, in general, a bound in the reverse direction is impossible, in many applications the quantity of interest is actually D*(v, Q) - defined, for an arbitrary fixed Q, as the infimum of D(P||Q) over all distributions P that are at least v-far away from Q in total variation. We show that D*(v, Q) ≤ Cv2 + O(v3), where C = C(Q) = 1/2 for balanced distributions, thereby providing a kind of reverse Pinsker inequality. Some of the structural results obtained in the course of the proof may be of independent interest. An application to large deviations is given.

Original languageEnglish
Article number6746175
Pages (from-to)3172-3177
Number of pages6
JournalIEEE Transactions on Information Theory
Volume60
Issue number6
DOIs
StatePublished - 1 Jan 2014

Keywords

  • McDiarmid's inequality
  • Pinsker's inequality
  • Sanov's theorem

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications
  • Library and Information Sciences

Fingerprint

Dive into the research topics of 'Minimum KL-divergence on complements of L1 balls'. Together they form a unique fingerprint.

Cite this