Universal Bayes Consistency in Metric Spaces

Steve Hanneke; Aryeh Kontorovich; Sivan Sabato; Roi Weiss

doi:10.1109/ITA50056.2020.9244988

Universal Bayes Consistency in Metric Spaces

Steve Hanneke, Aryeh Kontorovich, Sivan Sabato, Roi Weiss

Department of Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

7 Scopus citations

Abstract

We show that a recently proposed 1-nearest-neighbor-based multiclass learning algorithm is universally strongly Bayes consistent in all metric spaces where such Bayes consistency is possible, making it an "optimistically universal"Bayes-consistent learner. This is the first learning algorithm known to enjoy this property; by comparison, k-NN and its variants are not generally universally Bayes consistent, except under additional structural assumptions, such as an inner product, a norm, finite doubling dimension, or a Besicovitch-type property.The metric spaces in which universal Bayes consistency is possible are the "essentially separable"ones-a new notion that we define, which is more general than standard separability. The existence of metric spaces that are not essentially separable is independent of the ZFC axioms of set theory. We prove that essential separability exactly characterizes the existence of a universal Bayes-consistent learner for the given metric space. In particular, this yields the first impossibility result for universal Bayes consistency.Taken together, these positive and negative results resolve the open problems posed in Kontorovich, Sabato, Weiss (2017).

Original language	English
Title of host publication	2020 Information Theory and Applications Workshop, ITA 2020
Publisher	Institute of Electrical and Electronics Engineers
ISBN (Electronic)	9781728141909
DOIs	https://doi.org/10.1109/ITA50056.2020.9244988
State	Published - 2 Feb 2020
Event	2020 Information Theory and Applications Workshop, ITA 2020 - San Diego, United States Duration: 2 Feb 2020 → 7 Feb 2020

Conference

Conference	2020 Information Theory and Applications Workshop, ITA 2020
Country/Territory	United States
City	San Diego
Period	2/02/20 → 7/02/20

Keywords

Bayes consistency
classification
metric space
nearest neighbor

ASJC Scopus subject areas

Artificial Intelligence
Computational Theory and Mathematics
Computer Science Applications
Information Systems and Management
Control and Optimization

Access to Document

10.1109/ITA50056.2020.9244988

Cite this

@inproceedings{87746aaba8204d65b40af6b9990d0f73,

title = "Universal Bayes Consistency in Metric Spaces",

abstract = "We show that a recently proposed 1-nearest-neighbor-based multiclass learning algorithm is universally strongly Bayes consistent in all metric spaces where such Bayes consistency is possible, making it an {"}optimistically universal{"}Bayes-consistent learner. This is the first learning algorithm known to enjoy this property; by comparison, k-NN and its variants are not generally universally Bayes consistent, except under additional structural assumptions, such as an inner product, a norm, finite doubling dimension, or a Besicovitch-type property.The metric spaces in which universal Bayes consistency is possible are the {"}essentially separable{"}ones-a new notion that we define, which is more general than standard separability. The existence of metric spaces that are not essentially separable is independent of the ZFC axioms of set theory. We prove that essential separability exactly characterizes the existence of a universal Bayes-consistent learner for the given metric space. In particular, this yields the first impossibility result for universal Bayes consistency.Taken together, these positive and negative results resolve the open problems posed in Kontorovich, Sabato, Weiss (2017).",

keywords = "Bayes consistency, classification, metric space, nearest neighbor",

author = "Steve Hanneke and Aryeh Kontorovich and Sivan Sabato and Roi Weiss",

note = "Funding Information: Acknowledgments. We thank Robert Furber, Iosif Pinelis, and Menachem Kojman for providing some set-theoretical background. Aryeh Kontorovich was supported in part by the Israel Science Foundation (grant No. 755/15), Paypal and IBM. Sivan Sabato was supported in part by the Israel Science Foundation (grant No. 555/15). Publisher Copyright: {\textcopyright} 2020 IEEE.; 2020 Information Theory and Applications Workshop, ITA 2020 ; Conference date: 02-02-2020 Through 07-02-2020",

year = "2020",

month = feb,

day = "2",

doi = "10.1109/ITA50056.2020.9244988",

language = "English",

booktitle = "2020 Information Theory and Applications Workshop, ITA 2020",

publisher = "Institute of Electrical and Electronics Engineers",

address = "United States",

}

TY - GEN

T1 - Universal Bayes Consistency in Metric Spaces

AU - Hanneke, Steve

AU - Kontorovich, Aryeh

AU - Sabato, Sivan

AU - Weiss, Roi

N1 - Funding Information: Acknowledgments. We thank Robert Furber, Iosif Pinelis, and Menachem Kojman for providing some set-theoretical background. Aryeh Kontorovich was supported in part by the Israel Science Foundation (grant No. 755/15), Paypal and IBM. Sivan Sabato was supported in part by the Israel Science Foundation (grant No. 555/15). Publisher Copyright: © 2020 IEEE.

PY - 2020/2/2

Y1 - 2020/2/2

N2 - We show that a recently proposed 1-nearest-neighbor-based multiclass learning algorithm is universally strongly Bayes consistent in all metric spaces where such Bayes consistency is possible, making it an "optimistically universal"Bayes-consistent learner. This is the first learning algorithm known to enjoy this property; by comparison, k-NN and its variants are not generally universally Bayes consistent, except under additional structural assumptions, such as an inner product, a norm, finite doubling dimension, or a Besicovitch-type property.The metric spaces in which universal Bayes consistency is possible are the "essentially separable"ones-a new notion that we define, which is more general than standard separability. The existence of metric spaces that are not essentially separable is independent of the ZFC axioms of set theory. We prove that essential separability exactly characterizes the existence of a universal Bayes-consistent learner for the given metric space. In particular, this yields the first impossibility result for universal Bayes consistency.Taken together, these positive and negative results resolve the open problems posed in Kontorovich, Sabato, Weiss (2017).

AB - We show that a recently proposed 1-nearest-neighbor-based multiclass learning algorithm is universally strongly Bayes consistent in all metric spaces where such Bayes consistency is possible, making it an "optimistically universal"Bayes-consistent learner. This is the first learning algorithm known to enjoy this property; by comparison, k-NN and its variants are not generally universally Bayes consistent, except under additional structural assumptions, such as an inner product, a norm, finite doubling dimension, or a Besicovitch-type property.The metric spaces in which universal Bayes consistency is possible are the "essentially separable"ones-a new notion that we define, which is more general than standard separability. The existence of metric spaces that are not essentially separable is independent of the ZFC axioms of set theory. We prove that essential separability exactly characterizes the existence of a universal Bayes-consistent learner for the given metric space. In particular, this yields the first impossibility result for universal Bayes consistency.Taken together, these positive and negative results resolve the open problems posed in Kontorovich, Sabato, Weiss (2017).

KW - Bayes consistency

KW - classification

KW - metric space

KW - nearest neighbor

UR - http://www.scopus.com/inward/record.url?scp=85097351635&partnerID=8YFLogxK

U2 - 10.1109/ITA50056.2020.9244988

DO - 10.1109/ITA50056.2020.9244988

M3 - Conference contribution

AN - SCOPUS:85097351635

BT - 2020 Information Theory and Applications Workshop, ITA 2020

PB - Institute of Electrical and Electronics Engineers

T2 - 2020 Information Theory and Applications Workshop, ITA 2020

Y2 - 2 February 2020 through 7 February 2020

ER -

Universal Bayes Consistency in Metric Spaces

Abstract

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this